Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is concerned with unsupervised image generation using GANs. They devise a solution (SLOGAN) for addressing the problems current models have, especially when dealing with datasets with unbalanced features/attributes. In addition, to learn attributes from data and improve the conditional generation, it suggests using a new contrastive loss (U2C). 


main_review: The paper is fairly organized, overall supported by theoretical foundations and setups the right experiments for evaluating unsupervised conditional generation GANs.

However, I find some of the paper's claims are neither trivial (or well-accepted IMO) nor are they supported/backed in the paper:

For example, the abstract says "However, existing unsupervised conditional GANs cannot cluster attributes of these data in their latent spaces properly because they assume uniform distributions of the attributes." This claim needs to be supported by some reasoning/intuition and evidence/references as to why this is true. I can argue that this is not necessarily true, for example, [Self-Cond-Gan](https://openaccess.thecvf.com/content_CVPR_2020/papers/Liu_Diverse_Image_Generation_via_Self-Conditioned_GANs_CVPR_2020_paper.pdf), shows that clustering is possible using D's features in an unsupervised manner.

Some of the novelties and tricks described in the paper (e.g. the reparametrization trick, or derivation and usage of the Theorem 1,2 are fairly trivial. However, the application of these ideas to Unsup GANs is a novel contribution of the paper.

The experiments are well designed, however, some of the right baselines have not been compared against. 
The two models: 

1- [Self-Cond-Gan](https://openaccess.thecvf.com/content_CVPR_2020/papers/Liu_Diverse_Image_Generation_via_Self-Conditioned_GANs_CVPR_2020_paper.pdf) which is an unsupervised conditional GAN, based on the DCGAN architecture, and 
2- [PGM GAN](https://openaccess.thecvf.com/content/CVPR2021/papers/Armandpour_Partition-Guided_GANs_CVPR_2021_paper.pdf) which is also a GAN for unsupervised conditional image generation and also uses contrastive clustering 

Are both missing in the comparisons, both in the literature review and empirical comparisons. 

Since these two works are not discussed, based on my understanding and comparing the empirical results, It seems that SLOGAN does not achieve a comparable clustering performance to either (in terms of NMI over CIFAR-10) nor in terms of over all generation quality (FID). It would also be interesting to see how the models compare in terms of the newly introduced ICFID.


summary_of_the_review: In summary, the paper needs some adjustments before it is ready for publication. The related works need to be updated with the relevant works (Self-Cond-GAN and PGMGAN mentioned above). The experiments and empirical results need to be compared with these relevant works or discussed why the comparison might not be fair. 

EDIT: Given the proposed method is moderately novel and the newly presented comparisons after the discussions with authors, I increased my rating from 5 to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a Stein Latent Optimization for Generative adversarial networks, which can perform conditional generation in an unsupervised manner. The core innovation is the reparameterizable gradient estimations. The proposed unsupervised conditional contrastive loss further ensures the single attribute generation capacity. The idea is somewhat novel and motivated. The authors perform empirical studies on diverse datasets to conclude that the proposed algorithm is effective in learning balanced or imbalanced attributes.
main_review: Strength:
1. It is interesting to introduce Stein’s lemma to provide a first-order gradient identity for multivariate Gaussian distribution.
2. The way of using a reparameterization trick to estimate gradients of the parameters of Gaussian mixtures is quite neat.
3. The paper is well-written and clearly presented.

Weakness:
1. It is reasonable to conduct visualization comparisons with the most recent methods. 
2. The authors could present failure cases that may be associated with different attributes.
3. How to obtain a suitable attribute ratio for the probe data? What will happen if we change the ratio from 14:1 to 1:14 for eyeglasses?
summary_of_the_review: Overall, the paper is somewhat novel and easy to follow. The assumptions and decisions are well supported. The stepwise experiments are helpful and provide good insights to evaluate the proposed algorithm.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper uses Gaussian mixture as a prior to address the scenario of imbalanced attributes in GANs latent space clustering. The study derives Stein latent optimization that provides reparameterizable gradient estimations when assuming a Gaussian mixture prior for the latent space. The results show that the proposed model achieves superior performance compared to the baselines. 
main_review: Strengths:
1. The idea of using Gaussian mixture as a prior for imbalanced attributes is intuitive and technically sound. 
2. In the theoretical aspect, this paper derives Stein latent optimization based on Stein’s lemma to estimate the gradient of parameters.
3. The paper is well-written, and the result looks promising.

Weaknesses:
1. The imbalance datasets (MNIST and CIFAR) contain two classes. I would expect to evaluate over more imbalance settings of the datasets, such as one class is 0.1 fraction of the other nine classes. 
2. Two important baselines in GANs latent space clustering are missing. “Effect of The Latent Structure on Clustering with GANs” and “Mixture of GANs for Clustering”.
3. Compared with ClusterGAN, this paper address the importance of generated image quality. The study may provide a comparison to Self-conditioned GAN or IC-GAN in terms of both generated image quality and clustering quality. 
4. In the current version, it is difficult to infer how U2C loss helps the performance of the proposed model. I would expect the study presents the quantitative results (e.g., Tables 1 and 2) of the proposed without U2C loss.

*****************************

Post Rebuttal Comments:
The authors have addressed my concerns in the revised version. I am satisfied with the response and change my rating from 5 to 6.

summary_of_the_review: Although the derived Stein latent optimization for imbalanced attributes is interesting and promising, the current experiments are not enough to support the claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a method for performing unsupervised conditional generation from imbalanced attributes. The key components of this work are the following:

* A (architecture agnostic) GAN with a GMM latent space, where we have one component per attribute in the data. 

* The parameters of the components $(\mu, \Sigma)$ are learned via implicit reparameterization, and leveraging stein's lemma to derive gradients for $\mu$ and $\Sigma$. 

* The components are encouraged to be distinct by introducing a contrastive objective. An encoder is used to obtain a low dimensional vector representation of a generated sample, and this sample is encouraged to be close to the mean vector of the corresponding component in the latent space. 
main_review: I found this approach to be quite reasonable, and it consists of components that work well together. While GANs with mixture latent spaces have been proposed before, I am not aware of a method that efficiently updates the prior components. Previous work has used the reparameterization trick to derive component-specific updates, so the novelty of this specific component in using implicit reparameterization is a bit low, but the benefit is still there. I thought the introduction of the contrastive loss was interesting, but I had some concerns about the diversity of generated samples when the encodings are encouraged to match just the mean vector, ignoring the covariances. In the experiments, it looks to perform quite well, but I wonder how this would perform with less tightly-clustered data (even 8 Gaussians but with larger covariances).  

From the experiments, I mostly wanted to see how this method handles a large imbalance on high-dimensional data. I am somewhat satisfied with the CelebA-HQ results with a ratio of 1.7:1, but this seems a bit low and does not stress the model at all. I wonder where does the model start to struggle. Is 2:1 possible?  Or even 5:1? Figure 5 examines this a little, but eyeglass is an attribute with global features almost identical to no-eyeglasses, so I don't think it is very informative. 

-----------------------------------------
### Post Rebuttal Comments

I thank the authors for their additional efforts in addressing my concerns. I am satisfied with this response and I will be retaining my score. 

summary_of_the_review: I think this is a reasonable combination of components that make progress on the task: facilitating learning of samplers from data with highly imbalanced class attributes. Given the performance increase compared to [1], it looks like using implicit reparameterization and a contrastive loss (the main novelty here) make a clear difference in performance. I think each of the components are justified and perform well. 

[1] Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 166–174, 2017.
review_title: null

Meta-review: This paper proposes an unsupervised learning method for GANs, called SLOGAN, which allows conditional generation of samples, by utilizing clustering structures of training data in a latent space. The main significance of the proposal over existing unconditional conditional GANs is that it is capable of dealing with training data with imbalance in the latent space. The proposal consists of the use of implicit reparameterization based on the generalized Stein lemma, which makes learning of the mixing coefficient parameters possible, as well as introduction of the U2C loss.

The initial review score distribution is such that two of them are just above the acceptance threshold, and two others are just below it. Upon reading the review comments and the author responses, as well as the paper itself, I think that the evaluations of the reviewers are more or less coherent with each other:

1. The proposed method is moderately, if not significantly, novel: The differences from DeLiGAN are the use of implicit reparameterization based on the generalized Stein lemma, learning of the mixing coefficient parameters, and introduction of the U2C loss.
2. The experimental results, while demonstrating effectiveness of the proposed method to some extent, were not convincing enough.

As for the item 2, the authors have provided results of additional experiments in their responses, as suggested by the reviewers, and two reviewers have revised their scores upward accordingly.

Yet another point I would like to mention is that in some numerical results summarized in Tables 1 and 2, as well as in several other places, one can notice somewhat large errors, so that one might be able to question the statistical significance of the claimed best-performing methods, shown in bold. (If my guess would be correct, the authors regarded the *best in the mean* as the best, ignoring the standard error, and did not perform any statistical testing to confirm the significance.) I would therefore appreciate additional assessment of significance of the numerical results via proper statistical testing.

Because of the above, I would like to recommend acceptance of this paper.
Decision: Accept (Poster)