Title: ADVERSARIALLY LEARNED MIXTURE MODEL

Abstract: The Adversarially Learned Mixture Model (AMM) is a generative model for unsupervised or semi-supervised data clustering. The AMM is the first adversarially optimized method to model the conditional dependence between inferred continuous and categorical latent variables. Experiments on the MNIST and SVHN datasets show that the AMM allows for semantic separation of complex data when little or no labeled data is available. The AMM achieves unsupervised clustering error rates of 3.32% and 20.4% on the MNIST and SVHN datasets, respectively. A semi-supervised extension of the AMM achieves a classification error rate of 5.60% on the SVHN dataset.

1. INTRODUCTION:

Semi-supervised or unsupervised representation learning enables the utilization of all available data when tackling problems where there are little or no labeled examples. This is a common scenario in many applications of machine learning, such as medical image analysis, where it is reinforced by the expense of obtaining expert labeled examples. Moreover, machine-learned representations are more likely to be used for subsequent tasks if they are interpretable and meaningful. Deep generative modelling is a suitable approach to this problem, as derived models have been shown capable of learning from both labeled and unlabeled examples, embedding data according to desired latent variable distributions, and producing realistic data examples generated from samples of those latent variables.

The Generative Adversarial Network (GAN) has recently emerged as a powerful framework for modeling complex data distributions without having to approximate intractable likelihoods. In the formulation by Goodfellow et al. (2014) , a GAN consists of two networks: a generator G that is trained to yield unique samples from the data distribution, and a discriminator D that is trained to distinguish between generated and true data samples. Dumoulin et al. (2016) and Donahue et al. (2017) have proposed the ALI and BiGAN models that add an inference process, i.e., the ability to map data samples to points in the latent space, to the GAN framework. A second generator for inference, or encoder, is added to the original GAN generator and the discriminator is adapted for the two-dimensional space of data inputs and latent representations. A variant of the resulting model is also introduced by Dumoulin et al. (2016) for conditional data generation, but still assumes that the class of the data is always observed, as inference of categorical variables is not included.

Adversarial approaches for the inference of both continuous and categorical variables are actively researched. Chen et al. (2016) introduce a hybrid adversarial method that is capable of modelling both continuous and categorical latent variables for unsupervised clustering and feature disentanglement. Another hybrid adversarial method is introduced by Makhzani et al. (2016) where adversarial objectives on continuous and categorical latent variables are optimized for unlabeled examples and categorical cross entropy on categorical variables is optimized for labeled examples. Li et al. (2017) and Deng et al. (2017) point toward fully adversarial semi-supervised classification using inferred categorical variables by introducing a "three player" adversarial game, but stop short by adding auxiliary "collaborative" objectives. In each of these methods, it is assumed that categorical and continuous latent variables are independently distributed. This independence assumption results in discontinuities in the latent space between categories, which removes the notion of inter-categorical proximity.

Another notable family of generative models, Variational Autoencoders (VAEs), maximize the posterior distribution of latent representations given the data instead of using an adversarial approach. As VAEs integrate inference, semi-supervised classification can be performed by conditioning the continuous latent variable of the VAE on the class label (Kingma et al., 2014; Dilokthanakul et al., 2016; Maaløe et al., 2017) . However, the quality of VAE results depend on the expressiveness of the inference distribution and every time the assumptions about the inference or data distributions are changed a new objective function needs to be derived. In this way, variational optimization is not as versatile as adversarial training.

We present the Adversarially Learned Mixture Model (AMM). The AMM is, to our knowledge, the first generative model inferring both continuous and categorical latent variables to perform either unsupervised or semi-supervised clustering of data using a single adversarial objective. This is enabled, in part, by explicitly modelling the dependence between continuous and categorical latent variables, which eliminates discontinuities between categories in the latent space. Semi-supervised clustering and classification is enabled by a simplified formulation of the "three player game", presented by Li et al. (2017) . In this paper we show that the AMM achieves an unsupervised clustering error rate of 3.32% and 20.4% on the MNIST (LeCun & Cortes, 2010) and SVHN datasets (Netzer et al., 2011) respectively, and that a semi-supervised extension, SAMM, achieves a classification error rate of 5.60% on the SVHN dataset . To support the reproducibility of the methodology and experiments presented, a public version of the code will be made available.

2. METHOD:



2.1. PRELIMINARIES:

The ALI and BiGAN models are trained by matching two joint distributions of images x ∈ R D and their latent code z ∈ R L . The two distributions to be matched are the inference distribution q(x,z) and the synthesis distribution p(x,z), where,

EQUATION

(x,z) = p(z)p(x | z).

(2) Samples of q(x) are drawn from the training data and samples of p(z) are drawn from a prior distribution, usually N (0,1). Samples from q(z | x) and p(x | z) are drawn from neural networks that are optimized during training. Dumoulin et al. (2016) show that sampling from q(z | x) = N (µ(x),σ 2 (x)I) is possible by employing the reparametrization trick (Kingma & Welling, 2013) , i.e. computing

EQUATION

where is element wise vector multiplication.

A conditional variant of ALI has also been explored by Dumoulin et al. (2016) where an observed class-conditional categorical variable y has been introduced. The joint factorization of each distribution to be matched are:

q(x,y,z) = q(x,y)q(z | y,x), (4) p(x,y,z) = p(y)p(z)p(x | y,z).

(5) Samples of q(x,y) are drawn from the data. Samples of p(z) are drawn from a continuous prior on z, and samples of p(y) are drawn from a categorical prior on y, both of which are marginally independent. Samples from q(z | y,x) and p(x | y,z) are drawn from neural networks that are optimized during training.

In the following sections we present graphical models for q(x,y,z) and p(x,y,z) that build off of conditional ALI. Where conditional ALI requires the full observation of categorical variables, the models we present will account for both unobserved and partially observed categorical variables. We finally show how they can be optimized using a single adversarial objective.

2.2. ADVERSARIALLY LEARNED MIXTURE MODEL:

The AMM is an adversarial generative model for deep unsupervised clustering of data. Figure 1 presents an overview of the model.

Like conditional ALI, a categorical variable is introduced to model the labels. However, the unsupervised setting now requires a different factorization of the inference distribution in order to 6)) for the inference distribution. AMM consists of two generators, encoder G z (x,G y (x)) and decoder G x (y,z), and a discriminator D(x,y,z). SAMM includes an additional generator for labeled data, G z (x,y).

x ∼ q(x) y ∼ q(y | x) z ∼ q(z | x, y) x, y ∼ q(x, y) z ∼ q(z | x, y) y ∼ p(y) z ∼ p(z | y) x ∼ p(x | y, z) D(x, y, z) Gz(x, Gy(x)) Gz(x,

enable inference of the categorical variable y, namely:

EQUATION

or q 2 (x,y,z) = q(x)q(z | x)q(y | x,z).

(7) Samples of q(x) are drawn from the training data, and samples from q(y | x), q(z | x,y) or q(z | x), q(y | x,z) are generated by neural networks. The reparametrization trick is not directly applicable to discrete variables and multiple methodologies have been introduced to approximate categorical samples (Jang et al., 2016; Maddison et al., 2017) . We follow Kendall & Gal (2017) and sample from q(y | x) by computing

EQUATION

Then, we can sample from q(z | x,y) by computing

EQUATION

A similar sampling strategy can be used to sample from q(y | x,z) in ( 7).

The factorization of the synthesis distribution p(x,y,z) also differs from conditional ALI:

EQUATION

The product p(y)p(z | y) can be conveniently given by a mixture model. Samples from p(y) are drawn from a multinomial prior, and samples from p(z | y) are drawn from a continuous prior, for example, N (µ y=k ,1). Samples from p(z | y) can alternatively be generated by a neural network by again employing the reparameterization trick. Namely,

EQUATION

)

This approach effectively learns the parameters of N (µ y=k ,σ y=k ).

2.2.1. ADVERSARIAL VALUE FUNCTION:

We follow Dumoulin et al. ( 2016) and define the value function that describes the unsupervised game between the discriminator D and the generator G as: min

EQUATION

)

There are four generators in total: two for the encoder G y (x) and G z (x,G y (x)), which map the data samples to the latent space; and two for the decoder G z (y) and G x (y,G z (y)), which map samples from the prior to the input space. G z (y) can either be a learned function, or be specified by a known prior (see Algorithm 1 for a detailed description of the optimization procedure).

Algorithm 1 AMM training procedure using distributions ( 6) and ( 11). θ Gy(x) ,θ Gz(x,Gy(x)) ,θ Gz(y) ,θ Gx(y,Gz(y)) ,θ D Initialize AMM parameters while not done do

x (1) ,...,x (M ) ∼ q(x) Sample from data and priors y (1) ,...,y (M ) ∼ p(y)

z (j) ∼ p(z | y = y (j) ), j = 1,...,M x(j) ∼ p(x | y = y (j) ,z = z (j) ), j = 1,...,M Sample from conditionals ỹ(i) ∼ q(y | x = x (i) ), i = 1,...,M z(i) ∼ q(z | x = x (i) ,y = ỹ(i) ), i = 1,...,M ρ (i) q ← D(x (i) , ỹ(i) , z(i) ), i = 1,...,M Compute discriminator predictions ρ (j) p ← D( x(j) ,y (j) ,z (j) ), j = 1,...,M L D ← -1 M M i=1 log(ρ (i) q )-1 M M j=1 log(1-ρ (j) p ) Compute discriminator losses L Gx(y,Gz(y)) = L Gz(y) ← -1 M M i=1 log(ρ (i) p ) Compute x generator losses L Gy(x) = L Gz(x,Gy(x)) ← -1 M M i=1 log(1-ρ (i) q )

Compute y and z generator loss

θ D ← θ D -∇ θ D L D Update discriminator parameters θ Gx(y,Gz(y)) ← θ Gx(y,Gz(y)) -∇ θ Gx (y,Gz (y)) L Gx(y,Gz(y)) Update generator parameters θ Gz(y) ← θ Gz(y) -∇ θ Gz (y) L Gz(y) θ Gy(x) ← θ Gy(x) -∇ θ Gy (x) L Gy(x) θ Gz(x,Gy(x)) ← θ Gz(x,Gy(x)) -∇ θ Gz (x,Gy (x)) L Gz(x,Gy(x))

2.3. SEMI-SUPERVISED ADVERSARIALLY LEARNED MIXTURE MODEL:

The Semi-Supervised Adversarially Learned Mixture Model (SAMM) is an adversarial generative model for supervised or semi-supervised clustering and classification of data. The objective for training SAMM involves two adversarial games to match pairs of joint distributions. The supervised game matches inference distribution (4) to synthesis distribution (11) and is described by the following value function:

EQUATION

The unsupervised game matches either of the inference distributions, ( 6) or ( 7) to the synthesis distribution (11). In the case using distribution (6), the unsupervised game is described by ( 13).

The generator for semi-supervised learning has three components: encoders G z (x, G y (x)) and G z (x,y) map the labeled and unlabeled data samples, respectively, to the latent space, and a decoder G x (y,G z (y)) maps samples of y and z to the input space, where G z (z) can either be a learned function or be specified by a prior. The encoder for labeled data again consists of two generators (Figure 1 ). A detailed description of the training algorithm is given in algorithm 2 of the appendix.

In practice, optimization of each of the generators and the discriminator can be done simultaneously for both the unsupervised and semi-supervised updates. 

3. RELATED WORKS:

{ ẋi = G xi (G y (x i ),G z (x i

))}, while the inference generator is optimized using both the L2 objective and an adversarial objective. For InfoGAN, the inference generator is optimized by maximizing the per-example Mutual Information (MI) between samples of categorical latent variables {y i ∼ p(y)} and continuous latent variables {z i ∼ p(z)} and their "reconstructions" {{ ẏi , żi } = G y,z (G x (y i ,z i )}, while the synthesis generator is optimized using both the MI objective and an adversarial objective.

Recent approaches using self-supervision with data augmentation have also been proposed for unsupervised discrete representation learning by Kilinc & Uysal (2018) (LALNets) and Hu et al.

(2017) (IMSAT). In IMSAT, a network is trained to maximize the Mutual Information between input data and a discrete representation, similar to InfoGAN. In addition to this objective, the network is regularized by encouraging the representation of original, unperturbed data {x i } to be close to that of transformed data {T (x i )}. In the LALNets framework, a network is trained to distinguish unperturbed data and augmented data, and then k-means clustering is performed on the latent space representation of the unperturbed data to obtain cluster labels.

On the other end of the generative spectrum, The adversarial and hybrid-adversarial approaches thus far discussed all model y and z as being conditionally independent from each other. This may be an ideal prior structure for inference, for example, in learning disentangled representations of x sampled from a limited domain (Chen et al., 2016) . However, the independence assumption cannot account for the notion of proximity between categories because z is identically distributed for each category in y. Therefore, the distance between categories is equal and indeterminate. AMM and SAMM are presented as adversarial approaches to model conditional dependencies between y and z, but they do not preclude the independence assumption. The proposed methods can model y and z as conditionally independent with inference distribution We optimize all networks using Adam (Kingma & Ba, 2014) with α = 0.0002 and β 1 = 0.5. All kernel weights are initialized using a Gaussian distribution with standard deviation 0.02, all biases are initialized to 0.0. Following the criteria in Jiang et al. (2017) , the performance is evaluated as follows:

EQUATION

EQUATION

)

where M is the number of samples, y i is the true label, and ỹz,i is the predicted cluster label, and N is the set of all mappings between cluster labels and true labels.

4.1. GRADIENT PENALTY:

The gradient penalty introduced by Gulrajani et al. ( 2017) is added to the discriminator loss to help stabilize training of AMM and SAMM models. This penalty keeps the gradients of the discriminator with respect to the inputs x, y, and z on the same order of magnitude. The penalty applied to the discriminator loss is

EQUATION

)

where points ( x, ŷ, ẑ) are drawn at random on straight lines between real or prior samples (x, ỹ, z) and synthesized or inferred samples ( x,y,z). The gradient penalty for Jensen-Shannon GAN introduced by Roth et al. ( 2017) has also been explored, but did not produce better results. The regularization term is set to λ = 10.0, and λ = 0.01 for MNIST and SVHN experiments, respectively.

4.2. MNIST:

In this section, the AMM is evaluated on the task of unsupervised clustering of hand-drawn digits using the MNIST dataset. To model p(y)p(z | y), a 64 dimensional mixture of Gaussians is used with 10, 20, and 30-components across 3 experiments. A multinomial prior is used for p(y) with uniform probability for each class. The mean and variance of each component distribution are learned using the reparameterization trick via (12). Table 1 reports the test-set clustering error-rate mean and standard deviation over 10 trials, with the 10-component AMM achieving a 3.32 ± 0.39 percent error rate. Figure 3a shows random samples drawn from each component distribution generated by G x . Figure 3b is a t-SNE projection of test samples drawn from G z onto a 2D manifold. We show in 3c that AMM learns a smooth latent manifold as we interpolate between examples from SVHN. The cluster matrix for the SVHN test set is shown in figure 3d , demonstrating an overall classification error rate of 20.4% ± 0.80. From these results, we can make a few observations. First, the generated images are realistic looking. This is the case for every cluster other than cluster 5, which appears to be a noise cluster. Secondly, each cluster only contains a single digit. Finally, we observe that for each digit there are multiple clusters, one containing a dark background with light numbers (ie. cluster 28), and another containing a light background with dark numbers (ie. cluster 12). This is important, as it shows that the AMM has learned semantically meaningful clusters.

4.3.2. SEMI-SUPERVISED CLUSTERING AND CLASSIFICATION:

It is evident from the last experiment that the confounders introduced by the SVHN dataset made unsupervised semantic clustering more difficult. In this section we show how SAMM can be used to guide clustering along predefined categories using only a small amount of labeled data. To this end, we limit the samples drawn from q(x,y) to a random selection of 1000 examples from the training set. To model p(y)p(z | y) we use a 64 dimensional mixture of 10 spherical Gaussians, where the the mean and variance of each component distribution are learned using the reparameterization trick via (12). There is considerable class imbalance in the SVHN dataset, so a multinomial prior is used for p(y) with each class probability set to the frequency observed in the labeled subset of the training data. 

EQUATION

Figures 4e and 4f compare the confusion matrices for predictions given by ỹz and those given by ỹ from G y . The similarity between each is further evidence that the inference network has learned to embed data according to the desired distribution. 

5. CONCLUSION:

The AMM is presented as a generative model for unsupervised or semi-supervised data clustering with several contributions. The AMM is the first fully adversarially optimized method to model the conditional dependence between categorical and continuous latent variables, providing impressive unsupervised clustering and competitive semi-supervised classification results on benchmark datasets.

In contrast with other semi-supervised approaches, we have shown that the use of additional losses or discriminators (Gan et al. (2017) ; Li et al. (2017) ; Deng et al. (2017) ) are redundant additions to frameworks, as the AMM yields similar or better results to these methods. In contrast with Dai et al. (2017) , our strong semi-supervised performance and qualitatively good image generation demonstrate that semi-supervised performance and image generation are not necessarily opposing goals. As a fully adversarial framework, the AMM provides a simple, yet powerful formulation as a foundation for future work. For example, the distribution of mixture components can be something be other than a Gaussian without the need to derive an new evidence lower bound. Furthermore, by learning conditional dependencies between y and z, the AMM preserves the notion of proximity between classes in the latent space, which could be useful for applications in metric space learning and few shot learning. On top of this, learning the mean and variance of mixture components gives access to subsequent analysis of the likelihood function, which could provide an interpretable insight of the learned distributions. With that, the mixture coefficients are the only terms to be integrated into the framework. As a whole, the AMMs demonstrably strong performance validates its use in future work and extension into other domains.

:



Figures and Tables:

Figure 1: Overview of the unsupervised (AMM) and semi-supervised (SAMM) model with the first option (Equation (6)) for the inference distribution. AMM consists of two generators, encoder G z (x,G y (x)) and decoder G x (y,z), and a discriminator D(x,y,z). SAMM includes an additional generator for labeled data, G z (x,y).

and synthesis distribution p(x,y,z) = p(y)p(z)p(x | y,z); (16) however, analysis of this graphical model is left for future work. 4 EVALUATION AMM and SAMM are evaluated using two image datasets: MNIST (LeCun & Cortes, 2010) and SVHN (Netzer et al., 2011). The provided training and testing splits are used for MNIST experiments with 5000 randomly selected examples left out of the training set for validation. The same training, testing, and validation splits as Dumoulin et al. (2016) are used for SVHN. Preprocessing is limited to scaling image intensities on the range [0,1]. Detailed architectures for each experiment are shown in Section B of the appendix.

visualizes results from 1 of the 10 trials.

Figure 2: Unsupervised clustering of MNIST data with 10 mixture components. (a) Comparing test image membership and randomly generated digits for each mixture component. (b) Cluster matrix. (c) Reconstructions of input images: original data on the left of each pair. (d) Interpolation between examples: original data samples are shown in the first and last columns with linearly interpolated generations between. (e) t-SNE projection of testing samples, color-coded for the MNIST class labels (0 to 9).

Figure 3: Unsupervised clustering of SVHN data with 30 mixture components. (a) Randomly generated images for each mixture component, mixture component indices are indicated to the left of each row. (b) t-SNE projection of testing samples, color-coded for the SVHN class label (0 to 9). (c) Interpolation between examples: input images are shown in the first and last columns. (d) Cluster matrix.

Figure 4: Semi-supervised clustering and classification of SVHN data with 10 mixture components. (a) Test image predictions: rows correspond to the predicted class. (b) Randomly generated images for each mixture component. (c) Interpolation between examples: original data samples in first and last columns. (d) t-SNE projection of testing samples, color-coded for the SVHN class label (0 to 9). Confusion matrix for predictions given an image embedding (e) and given the generator G y (e).

Dilokthanakul et al. (2016) andJiang et al. (2017) offer non-adversarial, VAE-based approaches for unsupervised clustering. Like in the AMM, the combination of priors for the latent variables y and z is modeled as a Gaussian mixture model, where y corresponds to the mixture components.

Table content: <table/>

Test set clustering error rate and standard deviation for MNIST data. Methods using data augmentation are denoted with * .In this section, unsupervised clustering is revisited. The SVHN dataset is used to investigate how confounding attributes, such as color and contrast, affects the semantic separation of digits. To model p(y)p(z | y) a 64 dimensional mixture of Gaussians is used with 10, 20, and 30-components across 3 experiments. A multinomial prior is used for p(y) with uniform probability for each class, and the mean and variance of each component distribution are learned using the reparameterization trick via (12).

Table content: <table><tr><td>MODEL</td><td>K</td><td>MNIST</td><td>SVHN</td></tr><tr><td colspan="3">GMVAE (DILOKTHANAKUL ET AL., 2016) 30 7.23±1.60</td><td>-</td></tr><tr><td>VADE (JIANG ET AL., 2017)</td><td>10</td><td>5.54</td><td>-</td></tr><tr><td>CATGAN (SPRINGENBERG, 2015)</td><td>20</td><td>9.70</td><td>-</td></tr><tr><td>INFOGAN (CHEN ET AL., 2016)</td><td>10</td><td>5.00</td><td>-</td></tr><tr><td>AAE (MAKHZANI ET AL., 2016)</td><td colspan="2">30 4.10±1.13</td><td>-</td></tr><tr><td>IMSAT (HU ET AL., 2017)  *</td><td colspan="3">10 1.59±0.40 42.7±3.90</td></tr><tr><td>LALNETS (KILINC &amp; UYSAL, 2018)  *</td><td colspan="3">10 1.68±0.08 23.2±1.30</td></tr><tr><td>AMM</td><td colspan="2">10 3.32±0.39</td><td>-</td></tr><tr><td>AMM</td><td colspan="3">20 3.99±0.79 35.0±5.24</td></tr><tr><td>AMM</td><td colspan="3">30 4.01±1.11 20.4±0.80</td></tr><tr><td>4.3 SVHN</td><td/><td/><td/></tr><tr><td>4.3.1 UNSUPERVISED CLUSTERING</td><td/><td/><td/></tr></table>



Table content: <table><tr><td>reports the test-set error-rate mean and variance over 10 trials. SAMM achieves 5.60±0.45</td></tr><tr><td>percent error rate, which is an improvement over the ALI baseline. Figure 4 shows visualizations</td></tr><tr><td>of results from 1 of the 10 trials. Finally, given that we have defined p(y)p(z | y) we can use Bayes' theorem to derive p(y | z) and get a classifier given an image embedding z:</td></tr><tr><td>ỹz = argmax</td></tr><tr><td>k</td></tr></table>

Semi-supervised test set missclassification rate and standard deviation for SVHN data. † and ‡ denote similar encoder/classifier architectures. AAE (MAKHZANI ET AL., 2016) 17.70±0.24 IMPROVEDGAN (SALIMANS ET AL., 2016) ] † 8.11±1.30 ALI (DUMOULIN ET AL., 2016) 7.42±0.65 VAT SMALL (MIYATO ET AL., 2017) † 6.83±0.24 TRIPLEGAN (LI ET AL., 2017) ‡ 5.77±0.17 SGAN (DENG ET AL., 2017) ‡ 5.73±0.12 VAT LARGE (MIYATO ET AL., 2017) ‡

Table content: <table><tr><td>MODEL</td><td>SVHN</td></tr><tr><td/><td>(N = 1000)</td></tr><tr><td/><td>4.28±0.10</td></tr><tr><td>(DAI ET AL., 2017)  †</td><td>4.25±0.03</td></tr><tr><td>SAMM  †</td><td>5.60±0.45</td></tr></table>

Peer review (iclr2019):
 The paper presents a generative model that can be used for unsupervised and semi-supervised data clustering. unlike most of previous method the latent variable is composed of both continuous and discrete variables. Unlike previous methods like ALI the conditional probability p(y|x) of the labels given the object is represented by a neural network and not simply drown from the data. The authors show a clustering error rate on the MNIST data that is better than previously proposed methods.