GAN-based Gaussian Mixture Model Responsibility Learning
Authors: Wanming Huang
Shuai Jiang
Xuan Liang
Ian Oppermann
Richard Yi Da Xu
Venue:  (2020)
Keywords: Generative Adversarial Networks

Abstract: Mixture Model (MM) is a probabilistic framework which allows us to define a dataset containing K different modes. When each of the modes is associated with a Gaussian distribution, we refer it as Gaussian MM, or GMM. Given a data point x, GMM may assume the existence of a random index k ∈ {1, . . . , K } identifying which Gaussian the particular data is associated with. In a traditional GMM paradigm, it is straightforward to compute in closed-form, the conditional like- lihood p(x|k, θ), as well as responsibility probability p(k|x, θ) which describes the distribution index corresponds to the data. Computing the responsibility allows us to retrieve many important statistics of the overall dataset, including the weights of each of the modes. Modern large datasets often contain multiple unlabelled modes, such as paintings dataset containing several styles; fashion images containing several unlabelled categories. In its raw representation, the Euclidean distances between the data do not allow them to form mixtures naturally, nor it’s feasible to compute responsibility distribution, making GMM unable to apply. To this paper, we utilize the Generative Adversarial Network (GAN) framework to achieve an alternative plausible method to compute these probabilities at the data’s latent space z instead of x. Instead of defining p(x|k, θ) explicitly, we devised a modified GAN to allow us to define the distribution using p(z|k, θ), where z is the corresponding latent representation of x, as well as p(k|x, θ) through an additional classification network which is trained with the GAN in an “end-to-end” fashion. These techniques allow us to discover interesting properties of an unsupervised dataset, including dataset segments as well as generating new “out-distribution” data by smooth linear interpolation across any combinations of the modes in a completely unsupervised manner.

Under review as a conference paper at ICLR 2020
GAN-BASED GAUSSIAN MIXTURE MODEL RESPONSI- BILITY LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Mixture Model (MM) is probabilistic framework which allows to define dataset containing K different modes. When each of the modes is associated with Gaussian distribution, we refer Gaussian MM, GMM. Given data point v, GMM may assume the existence of random index & E {I,..., K} identify- ing which Gaussian the particular data is associated with. In a traditional GMM paradigm. it is straightforward to compute in closed-form, the conditional like- lihood p(r|k, 0). as well as responsibility probability p(k|22. 0) which describes the distribution index corresponds to the data, Computing the responsibility al- lows us to retrieve many important statistics of the overall dataset, including the weights of each of the modes. Modern large datasets often contain multiple unla- belled modes, such as paintings dataset containing several styles; fashion images containing several unlabelled categories. In its representation, the Euclidean distances between the data do not allow them to form mixtures naturally, nor it's feasible compute responsibility distribution, making GMM unable apply. To this paper, we utilize the Generative Adversarial Network (GAN) framework achieve an alternative plausible method to compute these probabilities the data's 's latent space instead of æ. Instead of defining p(r)k: kr 0) explicitly, we de- vised modified GAN to allow us define the distribution using p(z)k; 0), where 2 is the corresponding latent representation of x, as well as p(k(a, 0) through an additional classification network which is trained with the GAN in an "end-to- end" fashion. These techniques allow us discover interesting properties of an unsupervised dataset, including dataset segments as well as generating new "out- distribution" data by smooth linear interpolation across any combinations of the modes a completely unsupervised manner.
INTRODUCTION
Gaussian Mixture Model (GMM) is of the commonly used probabilistic models for dataset enjoy multiple modes. It assumes that all data points come from a mixture finite number Gaussian distributions. The density function of the GMM is defined below:
K (z = > aaN((z po. Ex)
(1)
k=
where K is the total number of Gaussians in the mixture, and bthh component is characterized by Gaussian distribution with weight ag, mean μ4 and covariance matrix Σt.
Given x is the data and & is the (latent) index of the mixture density, GMM allows us to compute the conditional likelihood p(z)k, 0) well responsibility probability p(k) æz 8). In the Bayesian paradigm, one may refer (kk] , 6) as the posterior density where the prior is p(k) (([,...,, K)).
This will further allow for example the segment membership of the data, the overall weights of the modes, we may even synthesizing meaningful "out-distribution" data using convex combination of any two modes.
retrieve many important statistics and properties about the dataset,
However, the conditional likelihood is hard to compute when dealing with high dimensional data, such images, data in its form do not form mixtures naturally.

---

Under review as a conference paper at ICLR 2020
At the same time, Generative Adversarial Network (GAN) gives way compute the latent representation 2 associated with the data x: GAN (Goodfellow et al. (2014)). introduces a 2-player non-cooperative game by generator G' and discriminator D. The generator produces samples from the random noise vector 2. The Discriminator differentiates between true samples and fake samples. The objective function of the game given follows:
DiimaavV(D.G) = (2)) + (x)px(s) log((1 - ((G())))) GD
The problem of the original GAN methodology is that the association between the latent vector from a cow-dimension latent space to a data sample a (in high-dimension data space) is one way, i.c., must generate before generating æ; making the backward association from - infeasible. Since the generation of z is independent of x, any arbitrary distribution should theoretically achieve its take, making uniform distribution or standard Gaussian distribution a popular choice.
In our proposed work, we are to devise methods which we are able to compute p(z[k, 0) and p(z|2, to replace p(r)k, kk, 0) and p(k|a, 0) respectively, through the use of GAN training. p(=|ar 0) is achieved by upgrading prior distribution p(z) to a GMM instead of a simple Gaussian. At the same time, the responsibility probability p(k] zz, ()) learned through a classification network which is trained from end-to-end with the GAN. The learned classifier can subsequently be used to classify new data points or perform segmentation on the test set. We also observe through experiments that smooth linear interpolation can be performed across multiple distribution modes to create the desired effects of "out-distriuutio" data.
Below we introduce some previous research that are related to our work.
1.1 GAUSSIAN MIXTURE MODEL IN GAN
Eghbal-zadeh & Widmer (2017) integrate a GMM into the GAN framework. Both of the means and covariance matrices trainable through the generator loss. Besides, instead of applying the classic adversarial loss as in Equation the authors proposed to use GMM likelihood.
Ben- Yosef & Weinshall (2018) also proposed method named GM-GAN which used Gaussian Mixture to model the distribution over the latent space, in addition to its variant for the conditional generation. The supervised GM-GAN modifies the Discriminator, so that instead of single scalar, it returns a vector o E NRN. Each element o represents the probability of the given sample being in each class. This will allow that images generated from the generator will be classified by the discriminator as certain class.
In addition, in the absence of prior knowledge, the GMM is assumed to be uniform for both works above, i.e.. Vk E {1....,K K} aş = K. On the contrary, our work makes no such assumption, the segmentation of data can be inferred from the trained model.
1.2 LINEAR INTERPOLATION GAN
Previously researchers have studied linear interpolation in the trained GAN models, where linear interpolations in the noise space lead to semantic interpolations et al. (2017) models the latent space as a unit sphere and learns the correspondence from the sphere space to the data space without the adversarial loss. Such a trained model is able to achieve smooth linear interpolation output between any two random vectors the unit sphere.
generated images. [Bojanwskki
Chen et al. (2016) proposed InfoGAN which learns to maximize the mutual information between subset of latent variables and the observation, i.e. I(c; G(z, c)), where is the class of the real sample. A learned InfoGAN model can disentangle discrete and continuous latent factors. Thus, linear interpolation
be performed using the continuous latent code, for example, from thin digit to a wide digit; but not possible on the discrete codes, such as across categories of images.
In the section, explain in details how each part of the proposed algorithm works.
2

---

Under review as a conference paper at ICLR 2020
1.3 PAPER ORGANIZATION
In Section
describe how each component of the proposed mechanism works. In Section 3 we demonstrate the performance the proposed method by comparing it against several baseline models. Section 2 concludes the paper.
2 ARCHITECTURE
The proposed architecture consists of three networks. First, a classifier C' which outputs the possi- bility that given image belonging each Gaussian, and the possibility is used to construct GMM distribution. Second, a Generator G which produces synthetic samples from GMM random vectors. Third, a Discriminator D which encodes samples feature vectors and discriminates between real and synthetic samples. The overall architecture design is shown below in Figure | In the following sections, we explain the details of the three networks.
.2
C
L classifier
X
a a pry a.!
Σg K
2
C aadversarial
Z1
3
02 4
L cadversarial
D
L2 cadversarial
L
radversarial
Figure 1: The overall architecture. The feed-forward logic of the Classifier C², the Generator G and the Discriminator D are marked with different colours.
2.1 THE CLASSIFIER
For an image H; the classifier C' outputs the probability (a:...., a . aff of the image belonging to each Gaussian indexed from K. These probability values are considered to be the mixture weights of the GMM which corresponds to the specific image, i.e, the density function
of the GMM model which corresponds to image x; is NfN((u Ex). Me and Σ₂ for k
(1.2.... K} are trainable parameters. During test time, the learned Classifier can easily output p(k]a x 0) for any test image x.
The design of the classifier has two alternatives, the first is to share the feature encoding layers with Discriminator: The shared feature encoding network encodes each image a feature vector, and the Classifier will be simple standard linear Soft-max classifier built top of the features. The second is to build stand-alone network which contains multiple CNN layers to classify images.
Details of both network designs be found below in Table [ In Table we use acronyms for operations in the table: "Conv" is the convolution operation, which the kernel and stride size are in the bracket; "Batch norm" is short for batch normalization; "Flatten" refers the operation that flattens a tensor to 1D array. The same acronyms are also used in Table and Table Dang. Dj, and D. related to the datasets use, the exact value of each reported in Section 3
In Section 3.3, we report the results of both classifier designs in terms of generation performance and computation costs.

---

Under review as a conference paper at ICLR 2020
Solsstage
Tersors
Output
Eneoding
(kerele5 LaakyReUU
Dag
Dax Deu/
(kemnlll, sinde=2)
LeskyReLU
Dug/2 Dayf
Day/A Druf
Encoding Network not shared)
(kerneld,, strdee1)) Rel.U
Fle Deg
Der Dlag
MaxPool (pool_sizees sinde=2)
Die Dog
Dae/? Dey/
Coav (kerneled, strdee2) KelU Homen
Dea/ Max /2
1 x Pag/ x
Linear a Rel.Il
Dug/ x Dhag/4 /4 16
1034
Classification Network (if shured)
Linear
Dieg x Deeg 128
K
Classifiattion network (if not shared)
Linear
1024
Table 1:
The network structure of the Classifier C. Each column from left right reports the network, sub-stage name, name of operations and shapes of imput and output tensors.
Parameters of the classifier are optimized by both the adversarial loss and the mutual information loss, the details of both loss functions are discussed in Section 2.2 and how the update is performed is introduced in Section 2.4
The Classifier is not required for generating new images after the model is fully trained. During testing, a random vector is sampled directly from the GMM model for the generation. The Classifier can then used to assign an unseen image Gaussians and perform segmentation on the testing set.
2.2 THE GENERATOR
Given the classification result a; for the input image Fi, ideally, multinomial sampling is performed
to select one Gaussian out of K. The density function of this sampling is Σ a * *N (Ju. Es). and
random vector 2; is sampled from the chosen Gaussian distribution to perform the image generation.
However, our design expects the training to be performed in an "end-to-end" fashion, and the sam- pling process would break the chain of the back- pack-propagation from the adversarial loss the classi- fier parameters. Therefore, we instead sample one random vector from each Gaussian represented as 2, [=],.... ). These vectors are used generate K synthetic images [.t....; . . J. a used weigh the adversarial loss calculated from each pair of generated sample a and the real image 24.
In addition, the reparameterization trick is applied to the 2; sampling process so that the back- propagation can be used to update the parameters μ4g and Σn of each Gaussian. Instead of sampling z (14 ) we define 2 A&E μA Vk [1, K], where sampled as e (0, 7), and A
Σ
The adversarial loss for the proposed framework is thus calculated over K pairs of real and synthetic samples as:
K
Cadversarial
ExzeEaaaco Epanto
Σ
a x (logD(x;) + +lo((1 D(
(3)
K
In addition to the adversarial loss, also encourage the classifier to restore the input classi- fication output from the generated image. Therefore, we also employ mutual information loss between the classification result from the real and the generated image. Following InfoGAN by Chen et al. (2016) where the mutual information loss is formulated as I(c, G(z,c)) = Ee- [c]z] + Hc)). where is the class of real sample. We define the mutual information loss in our work as:
L E, Er Epdato
Σ at
Iaa at))
(4)
K
The Generator structure used in our experiments is as below in Table In the table, "LeakyReLU" which is short for "leaky rectified linear unit" and "Tanh" are activation functions.
4

---

Under review as a conference paper at ICLR 2020
Stage
Name
Input Tensors
Output Tensors
Linear Batch norm LeakyyRLLU + Reshape
1 x D2 Ding/ Dang! Ding! x Dime! x 128
Duas/ x Dau /4 x 256
Trans-posed Conv (kemel=5, stride=1) + Batch Leak Transposed Conv (kerel=5,, stride=2) stride)) Batch norm LeakyReLU
Dang/ x Dang!
Ding! hm x Dine!
64
Transposed Conv (kemel=5,, stride=2) + Tanh
Ding/2 x Diu! 64
Dinx x Ding x Dr.
Table The network structure the Generator G.
2.3 THE DISCRIMINATOR
As mentioned in Section 2.1 the design of the Discriminator has two options: whether not the image encoding layers are shared with the classifier. The image encoding network is followed by standard linear logistic regression identify the given image to be real or fake. Details of the network are shown below in Table
Stage
Sub-stage
Name
Inpur Tensors
Ousput Tensors
Encoding Network
Cony (kernel-S,, stride-) LeakyReLU
Daeg Dive Daag 22 Dreel2 Dagl 4 Dagg/ /4 x 128
Dave? Dag!? 64 Ding DugfA 128
Con (kernelm4, stride=2) Batch norm LeakyRel.U Halten Lineat
Discriminator Network
Table 3: The network structure of the Discriminator D.
2.4 TRAINING PARAMETERS FOR THE PRIOR DISTRIBUTION
In our setting, the prior distribution a GMM model, the two trainable variables are means for K Gaussians E IR R D, and standard deviations for K Gaussians A E IR xxxxx Both variables are updated by adversarial loss addition to the mutual information loss as the training performed "end-to-end".
Below we give the pseudocode about how the updates are performed on each network in one itera- tion.
Algorithm Training the proposed model for iteration
Require: X = [x] ................................ .M A training images in one batch
for = ... MM do
2: Classify into a; = ja rel.....de ........................ 1
3: for 1...K do
4:
~ I).
5:
2k Age pAc
6:
25 - G(zk)
7:
Classify do into 6: Ja a..
8:
end for
9:
Calculate Cadversarial from x; and [+] K
in Equation
10:
Calculate L' from and â as in Equation
Update the Discriminator with Cadvensarial
12:
Update both the Generator and the Classifier with Cadversarial Update the Classifier with Cl
13:
14:
end for
3 EXPERIMENTS
In this section, we evaluate the performance the proposed method by comparing it with several baselines.
3.1 EXPERIMENT SETUP
The datasets we use are the MNIST (LeCun et al. (2010)), Fashion-MNIST (Xiao et al. (001)) and Oxford-102 Flower (Krizhevsky (2009) datasets. The details are listed below in Table In
5

---

Under review as a conference paper at ICLR 2020
particular, only select subset of Oxford- 102 to perform the training, which is the images which belong to the first 10 classes. For experiments performed on each dataset, we used different hyper- parameters, the details are listed in Table
Dataset
Number of Classes
Data Dimension
Train Samples
Validation Samples
Test Samples
MNIST
1
60,000
10,000
Fashion-MNNN Miiion
x 28 1
60,000
10,000
Oxford- 102 Flower
102
64 x 3
1,020
1,020
6,149
Table 4: Statistics of the different datasets used in the empirical evaluation.
Dataset
Number of Epochs
Learning Rate y
Dase
Du
MNIST
200
0,0002
Reshion-MNIST
200
0.0002
Oxford-102
10,000
0.0002
3
Table Hyper-parameters for the training performed on each dataset.
3.2 LINEAR INTERPOLATION ACROSS GAUSSIAN
Below in Figure we show samples generated by the proposed model trained on several datasets in the completely unsupervised manner. We set the number of Gaussians equal to the total number of classes of the dataset in all experiments. When are performing the linear interpolation the right panels, the random vector z for each image generation is calculated as z = A&c + fi vk (1...., K), where is sampled as N(O, I). is kept the same for all images for each dataset.
in
We can draw two conclusions from the results in Figure First, fully trained proposed method can learn to "allocate" each class of image to a Gaussian. Second, the trained model can be used to perform smooth linear interpolation between Gaussians and even among more than two Gaussians. In Figure 3 we demonstrate the linear interpolation performed over 3 categories, The proportion of Gaussians of the synthetic images can be set manually.
3.3
IMAGE GENERATION QUALITY
The generation performance is measured with two commonly used metrics: Inception score (Sali- mans et al. (2016)) and Frehet Inception Distance (FID) score (Heusel et al. (2017)).
and a higher value generally indi- Inception score is calculated as (ppplEEDDxu(elel)lllll
cates better performance.
where x is a generated image and y is the label predicted by the Inception model (Szegedy et al. 2015
FID score is another metrics that measures the image generation quality. A lower value shows better image quality and diversity. It calculates the difference between real images and generated images g as FID(x,g) g) = I|и 1 1 + Tr( + Σn 2))
Note that limitations of both Inception and FID score have been pointed out in several previous literature (Barratt & Sharma (2018) 8), Lucic et al. (2018)), and there is currently no "perfect" metrics at this moment. These two metries are used as an indication rather than a hard measure.
This evaluation is performed the Oxford-102 dataset. As the Inception score is suggested to be evaluated on a large enough number of samples, we generate 5K synthetic samples to calculate both values. Below in Figure 4 show the plot of Inception scores and FID scores calculated over the training epochs. In Table 6 we report the number of trainable parameters, the best Inception and FID score.
While the four algorithms use the same network settings for the Generator and Discriminator, is clear to that the proposed method, whether not the encoding layers are shared between the

---

Under review as a conference paper at ICLR 2020
0
0
O
O
0
0
O
O
1
1
/
O
2
2
2
2
2
2
2
2
2
2
2.
2
2
2 3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
4
6
5
5
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
Q
7
7
7
7
7
7
7
7
7
7
7
7
7
7 8
0
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
9
4
4
4
9
4
9
8
9
9
Figure 2: Samples generated by the proposed models trained on the MNIST (top panels), Fashion- MNIST (middle panels) and CIFAR-10 (bottom panels) datasets. In the left panels, the Gaussian mixture contains K = 10 Gaussians. and each row contains images sampled from a different Gaus- sian. In the right panels, each row shows the linear interpolation result from one Gaussian to another. Note that the index of Gaussian does not necessarily correspond to the actual digit, generated images are reordered the order of the digit only for demonstration purpose.
7

---

Under review as a conference paper at ICLR 2020
2
2
2 2
2
2
2
2 2
2
2
2
J
) 2
2
2
2
)
2 2
2
2
2
2 2
2.
2.2.2
2
2
2
2
2
2
2
2.
2
2
2. 2
2
Figure 3:
Linear interpolation over 3 Gaussians on the MNIST, Fashion-MNIST and Oxford-102 dataset. Each row and each column contains linear interpolation performed between 2 Gaussians. Images along the diagonal shows the interpolation across 3 Gaussians.
classifier and the Discriminator, is able out-perform previous baseline models in terms of the im- age generation quality. The shared feature encoding layers would further improve the performance and reduce the size of the network.
number of parameters
Inception Score f
FID score |
Proposed (encoding not shared)
13, 005, 411
2.9664 1: 0.2188
231.0577 1: 7.5371
Proposed (encoding shared)
8, 794, 835
3.1368 1. 0.1596
205.9776 1 7.8587
GM-GAN Vanilla GAN
8, 467, 145
2.6770 + 0,1079
239. 239.3936 + 6.7672
8, 3,360, 145
2.4882 + 0.1065
247.0610 + 7.2361
Table 6: Number of parameters, Inception scores and FID scores of the proposed method and the baselines.
Inception Score over Epachs
FID Score over Epachs
Proposed (ancoding not shares; (encading shard:)
2.4
2.2
110 30ote
Propoued (enecding not shared)
12-
Proposed (enocding shared)
CN CAN
Vanila GAN
2000
4000
5000
8000
20000
2000
4000
5000
5000
1000C
Fportsx
Fanetrx
Figure 4: Inception score and FID score over training epochs.
4 CONCLUSION
In this paper we propose a novel framework which we incorporate the GMM to model the latent prior distribution. We also use a Classifier that learns to categorize an image to Gaussians. The Classifier is trained with the GAN model in "end-to-end" fashion. We demonstrate through experiments that the proposed method surpass previous baselines in terms of the image generation performance with only minor growth the size of the network. A trained model is also able to perform smooth linear interpolation across Gaussians, i.c. generate images with mixed styles from multiple categories.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv: arXiv:1801 1001.11100000 01973, 2018.
Matan and Daphna Weinshall. Gaussian mixture generative adversarial networks for diverse datasets, and the unsupervised clustering of images. CoRR, abs/1808 10356, 2018. URL. aatp//bss/1///
http://arxiv. htt.:/iii...g///////88/0 10356
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space 2017.
of generative networks. arXiv preprini
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2172-2180. Curran Associates, Inc., 2016.
Hamid Eghbal-zadeh and Gerhard Widmer. Probabilistic generative adversarial networks. CoRR, 0tss/7708.01888 2017. URL
01886
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014. URL pdf
nips.cc/papez. ottp:://.......aaaaoaaeeeeeerrra
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. CoRR, abs/1706.08500, 2017. /tt.
08500
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. lecun. com/exdb/nmisst 2, 2010.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 700-709, Curran Associates, Inc., 2018. /RL /papers.pppp....e/ nips cc/paper/ large scale study. pdd
7350 are wwwaaare-aaa-aaaeee gans www.erreeeeeeeueeeeeeeeeeeeeeeee
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2234-2242. Curran Associates, Inc., 2016. URL
nips cc/paper/
6125-
pdf
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shiens, and Zhigniew Wojna. Re- thinking the inception architecture for computer vision. CoRR, abs/1512.00567,, 2015. URL
http:://rxivv
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/1708.077477 2017. URI. http:/tttt/a..v.rrrrrte/e
abs / 1708. 07747


Reviews:
[{"review_id": "rylsua2Q9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use GMM as the latent prior distribution of GAN. The model adds the means, covariances and the discrete priors of the GMM as learnable parameters to the GAN, which are jointly optimized with other GAN parameters. The model also adds a discrete classifier to the training process. During training, the classifier predicts the probability of each image falling into each of the GMM clusters, and uses these probabilities to re-weight the GAN generated samples.\n\n# motivation\n\nThe motivation of this work is not clear to me. Even with an isotropic Gaussian prior, a fully connected neural network is already sufficient to (approximately) simulate the GMM sampling. Thus, explicitly modeling GMM doesn't seem to be necessary, and could make the learning more difficult. \n\nI also don't quite understand why the authors have to add a discrete classifier to the modeling. It appears that the discrete classifier is only used for controlling the relative weights of clusters in the GAN training. If that's the case, then what's truly needed is just the prior distribution of each cluster, which doesn't depend on the individual images. For concrete datasets, this prior is usually known. For example, in MNIST each cluster has an equal prior of 10%.\n\n# experiments\n\nThe model is evaluated on MNIST and Oxford-102. I'd like to see it tested on more realistic and higher resolution images, and compared with state-of-the-art GAN models. Since the motivation of the modeling design is unclear, the bar on the empirical results should be much higher.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlnIgtxqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the Gaussian mixture model at the latent space to have a better GAN training result.  The proposed architecture consists of three networks, a classifier, a generator, and a discriminator. Every input image goes through the classifier and gets the softmax output. The softmax output is considered as the mixture weights of the GMM model and controls the loss function of the generator accordingly. \n\nAlthough this paper has some positive sides, I recommend \"weak reject\" because of the following reasons.\n\n1. This paper is hard to follow. Many concepts are not discussed enough. For instance, how to train mu_k and \\Sigma_k, why should we use Eq. (3) as the loss of the generator, where do we use L^I in Eq.(4),...\n\n2. The experiment section requires more works. It would be much better to do experiments with much higher dimensional data sets and compare with many other GAN algorithms e.g. InfoGAN.\n\n3. The pseudo-code has many errors. For instance,  what is \\alpha_i^0? It is not introduced. What is \\hat{\\alpha}?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HklCTYa0KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a modification of GANs where the latent space follows a distribution modelled by a Gaussian Mixture Model. While the idea of using GMMs in GANs is not novel, the main contribution of the paper is to add a classification models that enables posterior inference. The whole model is trained jointly end-to-end using both an adversarial loss and a mutual information loss. The procedure is then tested on MNIST, Fashion-MNIST and a subset of Oxford-102 Flower. \n\nWhile the problem of posterior inference is interesting, the novelty of the paper is quite limited. The overall structure is clear, although writing can be improved. \n\nHowever, a part from the form, I have other concerns about the paper: \n- The main purpose of the paper is to tackle large image datasets that are hard to address by classical GMMs. This being said, the used datasets are composed of small and modal enough images that it seems hard to validate the claim of the paper using only these data. It seems to me that for the claim of the paper to be verified, larger scale/more complex datasets are needed. \n- In all experiments, the authors suppose they have access to the number of classes/modes in the data, which is a huge assumption. It would be interesting to see if it would be possible to automatically accurately select the number of modes, e.g. on a held out validation set.\n- One problem of GANs that the authors do not seem to consider is mode collapse. It would be interesting to do experiments with unbalanced datasets (e.g. MNIST 1 vs all) to see if the proposed architecture will model the data correctly. \n- I am confused about the use of Mutual Information loss. The author claim that they would like to enforce each of the generated images to be from the same class as the input image. This would make sense if the authors used the multinomial sampling for the latent variable generation. However, the K generated samples are from K different Gaussians. It seems unreasonable to require of the classifier to render the same result. \n- On the same note, in order to both use the classical multinomial sampling in GMMs and not break the backpropagation, have the authors considered updating the classifier and the GAN in separately in expectation-maximization fashion?\n- Finally, I don't see the point of weighting the adversarial loss by the weights of Gaussians. All the generated images are fake and should be equally detected as such. \nAlthough the idea is interesting, I think the paper, at its current status, not ready for publication. \n\nMinor: \n- P. 4: Generator: The sampling density from the multinomial distribution seems incorrect. However, as the authors skip the sampling step to be able to back propagate through the model, this is not significant. \n- P. 1: The paragraph before the last: may even synthesizing -> synthesize\n- P. 3: Architecture: Possibility -> Probability\n- P. 7: Figure title: CIFR10 -> Oxford-102\n- Algorithm notation: \n    *Indexes for alpha and alpha_hat from 1 and not 0 to be consistent with the rest of the text \n    *Add hats to the entries of alpha_hat\n    *  LI from alpha_i and alpha_hat_i as in Equation 4 -> maybe change alpha_hat by x_hat to be consistent with the notation of eq.4 (although the meaning is clear here). \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use GMM as the latent prior distribution of GAN. The reviewers unanimously agree that the paper is not well motivated, explanations are lacking and writing needs to be substantially improved. 
Decision: Reject