Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a GAN variant with structural adversarial objectives for self-supervised (SS) representation learning, which aims to achieve a general SS representation learning, especially escaping dependence upon the hand-crafted elements guiding data augmentation or proxy task design. In particular, at a coarse scale, a JSD divergence between the discriminator features of real samples and generated samples is defined as an adversarial objective for GAN.  At a finer scale, an adversarial clustering objective is defined, which groups adjacent real embeddings (features extracted by the discriminator) to form a cluster and adversarially attracts the generated embedding towards the nearby cluster center. Furthermore, an efficient regularization scheme that approximates the spectral norm of the Jacobian is introduced to regularize the discriminator’s smoothness. Experiments demonstrate their proposed GAN achieves results that compete with networks trained by state-of-the-art contrastive approaches.
strength_and_weaknesses: The proposed structural adversarial objectives are interesting and seem novel. The performance of the proposed GAN especially on K-mean clustering is promising. However, I have the following concerns:
1. Actually, this is a kind of work that utilizes a generative model together with clustering for SS representation learning. Therefore, deep generative clustering, a.k.a., generative model (GAN/VAE) with clustering, e.g., [1,2,3,4], is closely related to this work, which is missed in the Related Work part of this paper.
2. I think the proposed work can be better positioned in the literature if the motivation is to target deep generative clustering instead of SS representation learning. I would expect a detailed analysis about the superior performance of the proposed generative clustering to the existing generative clustering works.
3. [1] also uses discriminator features for clustering. From this aspect, the idea claimed in this work that tasks the discriminator with additional structural modeling is not novel.  
4. The experimental study is not totally consistent with the original motivation. First, augmentation is adopted though simple. Second, the experiments can be more convincing if conducted on different modalities of data, like text. It can verify the generality of the proposed method and also demonstrate the data augmentation should be specified in terms of data domains.
5. Gaussian assumption encouraging smoothness and diversified representation is not well justified.
 

Some questions:
1. Why spectral normalization used in Miyato et al. 2018 will harm model capacity but the spectral normalization used in this work will not?
2. How to decide which layers of D as clustering features?
3. What is the size of the maintained memory bank $f_m$?
4. For coarse-scale optimization with Gaussian, is it using mini-batches to determine the covariance and mean? If so, is it problematic since Eq. (4)/(5) is supposed to be defined on the whole data?
5. Why “regularizing JD(x) on l2 normalized embedding f will enlarge the norm of f˜ throughout training and eventually destabilizes the system; while regularizing JD(x) on f˜operates oppositely”?


Errors
1. In Section 2.2, “and demonstrates their sensitivity to the parameters of augmentation schemes.”-→demonstrate
2. In Eq. (2), JSD should equal to $1/2*D_{KL}$…
3. In $L_{advgroup}$, $f_i^g$ is supposed to be $f_i$?
4. $\\{f_{i,j}\\}_{j=1}^{k}$, $k$ is supposed to be $K$.
5. In Alg 1, $s$ should be $S$.
6. Fig. 2, (4) should be (d).

References

[1] Liu, S., Wang, T., Bau, D., Zhu, J. Y., & Torralba, A. (2020). Diverse image generation via self-conditioned gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14286-14295).

[2] Noroozi, M. (2020). Self-labeled conditional gans. arXiv preprint arXiv:2012.02162.

[3] Guo, X., Gao, L., Liu, X., & Yin, J. (2017, August). Improved deep embedded clustering with local structure preservation. In Ijcai (pp. 1753-1759).

[4] Jiang, Z., Zheng, Y., Tan, H., Tang, B., & Zhou, H. (2017, January). Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering. In IJCAI.

clarity,_quality,_novelty_and_reproducibility: 1. The idea that tasks the discriminator with additional structural modeling is not novel.
2. The proposed structural adversarial objectives are interesting and seem novel.
3. Some claims are not clarified clearly.
4. The setting of some hyperparameters is not clear.
5. There exist formula/grammar errors.
summary_of_the_review: The proposed structural adversarial objectives are interesting and seem novel. The performance of the proposed GAN especially on K-mean clustering is promising. However, the authors do not well position their work and miss the comparison with closely related works. Thus, the contributions of this work cannot be clearly demonstrated. In addition, the empirical support is not consistent with the original motivation. Last, the clarity of this work needs improvement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes to use the discriminator in a GAN as a feature extractor for self-supervised representation learning. Assuming that both real and fake features from the discriminator follow a gaussian distribution, the authors propose a loss based on the distance between the real and fake gaussian distributions. They also propose performing clustering on the extracted features and use this as an extra loss term for the discriminator optimization. Their proposed method outperforms SOTA self-supervised learning approaches on small-scale datasets and using the K-means metric.
strength_and_weaknesses: Strengths:
1. The paper is well-written and easy to follow.
2. The paper is well-motivated, and it is technically and theoretically sound.
3. The proposed model outperforms BigGAN in terms of image generation quality.
4. The idea of self-supervised feature representation learning using a discriminator is interesting and novel to the best of my knowledge.

Weaknesses:
1. There are some grammatical errors in the text, and some sentences are too long to follow. E.g.: Page 1, Yet, to improve...
2. There is no discussion on different discriminator architectures, such as the patchGAN discriminator or feature extractors as discriminators [1,2]. Also, a more detailed comparison against ICGAN can improve the paper.

[1] Sungatullina, Diana, et al. "Image manipulation with perceptual discriminators." ECCV 2018.

[2] Mao, Xin, et al. "Is discriminator a good feature extractor?." arXiv 2019.

3. The evaluation setting is not common in self-supervised approaches. Usually, linear classification and kNN classifiers are used. In this paper, they use SVM and Kmeans.

4. The computational complexities of the networks in Tab. 1 are not compared.

5. The proposed model only outperforms previous work on one metric.

Minor: 
The references for related work in Tab. 1 could be added.
clarity,_quality,_novelty_and_reproducibility: 1. The paper is clear, well-written, and easy to understand.
2. The idea of using a discriminator for self-supervised representation learning is novel to the best of my knowledge.
3. The architecture and experimental setup are clear and reproducible. The losses may be a bit tricky to re-implement, but they are well explained mathematically.
summary_of_the_review: The paper has both strengths and weaknesses. I have some concerns regarding the evaluation setting and discussion on existing literature.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of representation learning with GANs. To do so, it introduces a novel adversarial objective that is composed of a distribution-matching objective and a clustering objective. In addition, they propose a novel regularization algorithm to ensure the smoothness of the discriminator. Compared to the widespread spectral normalization, the proposed regularization does not control the spectral norm at each layer individually, allowing the discriminator to retain more capacity. The method is evaluated against contrastive state-of-the-art, on CIFAR-10, CIFAR-100, and ImageNet-10.
strength_and_weaknesses: Strengths:
- Representation learning is a timely topic. Tackling it with generative models has been a promising approach but their usefulness had yet to be proven compared to their contrastive counterparts. The current submission is a big step in that direction.
- The approach is original in significant ways. It focuses on the discriminator, which is not usually exploited in GAN literature, and manages to use it as a feature learner.
- It also proposes a new GAN loss and a new GAN regularization scheme, which both seem useful for training GANs.

Weaknesses:
- The presentation of the technical parts could be made clearer:
  - In section 3.2, it seems that $f$ represents alternatively the individual outputs of $D$ as well as their distribution. It could be useful to clarify the notation. It can be especially confusing since the original adversarial objective, i.e. the binary classification objective between fake and real samples, can sometimes be presented as a JSD minimization of the distributions.
  - It could be useful to remind the reader of how VJP and JVP can be computed in the supplementary. I for one would welcome it.
  - Table 2 contains important information as it serves as an ablation study regarding the capacity of GAN discriminators to learn features. I don't think this aspect is discussed anywhere in the paper. Also, it isn't mentioned how the representations for the baselines in Table 2 have been extracted.
  - Have the contrastive baselines been trained on the datasets they are being evaluated on, like the GAN would be? Or have they been trained on ImageNet and evaluated only on those datasets, as is often the case in contrastive benchmarks? This information is important to put the results in context and should be made explicit.
- The paper should compare to other generative representation learning methods such as those they cite in their related work, or a more recent one [1].
- The authors might also want to discuss how their objectives relate to and differ from Feature Matching [2]. Here are some more recent examples of where it has been used [3,4].
- The BigGAN backbone is quite generic and useful, but somewhat outdated, especially for unconditional generation. Have the authors considered StyleGAN-2 [5] or FastGAN [6] backbones? Can the author discuss the potential issues of switching to such backbones?

---

[1] Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola. Generative Models as a Data Source for Multiview Representation Learning. ICLR 2022.

[2] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen. Improved Techniques for Training GANs. NeurIPS 2016.

[3] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro. High-Resolution Image Synthesis and Semantic Manipulation With Conditional GANs. CVPR 2018.

[4] Liming Jiang, Changxu Zhang, Mingyang Huang, Chunxiao Liu, Jianping Shi, Chen Change Loy. TSIT: A Simple and Versatile Framework for Image-to-Image Translation. ECCV 2020.

[5] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila. Analyzing and Improving the Image Quality of StyleGAN. CVPR 2020.

[6] Bingchen Liu, Yizhe Zhu, Kunpeng Song, Ahmed Elgammal. Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis. ICLR 2021.

clarity,_quality,_novelty_and_reproducibility: The work is novel in multiple ways, the delivery is quite clear but it leaves out information hurting both understanding and reproducibility.

summary_of_the_review: The work has relevant contributions for both GANs in general as well as their application to representation learning.
It tackles the problem in a novel and interesting way.
However, too many details are missing, as well as comparisons, and some related work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by three knowledgeable referees. The reviews highlighted the relevance of the proposed approach but raised important concerns w.r.t. the positioning of the work within the literature (XWaT, 9r54, GyrJ), and the evaluation of the proposed approach - in terms of comparisons, as well as unconventional evaluation protocol and/or dataset choices (XWaT, 9r54, GyrJ). The authors actively engaged in discussion with the reviewers. After discussion, the reviewers remain unconvinced and reiterate their concerns w.r.t. the positioning of the contributions and experiments. The authors ask the AC to take an in-depth look to their discussion with the reviewers as they believe there might have been a mischaracterization of their work. The AC takes into consideration the authors request during the discussion with reviewers. After going over the paper and the reviews, the AC notes the following:
1. The paper emphasizes in several places that the proposed approach is competitive in terms of image generation quality (this appears in the abstract already).
2. The proposed approach is based on the GAN framework and does leverage a clustering loss to improve the representation learning capabilities of the discriminator. Although clustering GAN approaches might have had a different motivation when introduced (e.g. improving image generation as opposed to improving representation learning), they do seem very reasonable baselines to consider as the representations learned by their discriminators might be useful in downstream tasks. The motivation of introducing the proposed new GAN-based approach would be strengthened by showing it is effective when compared to already existing methods which could benefit the same problem. Even if those methods were introduced in a different context, comparing against them (from the perspective of representation learning) is important to justify the contributions of this paper.
3. The evaluation of the proposed approach (again, from a representation learning perspective) is not standard. The authors claim their work should be seen as pioneering and does not need large scale validation, yet using unconventional training and evaluation protocols makes it really hard to assess the value of the contribution. For example, if the baselines have been retrained to follow a different experimental protocol, then it is unclear whether these baselines were given the same opportunity to shine (i.e. was there sufficient hyper-parameter search to make performance claims).
4. If the authors would like to make claims w.r.t. the quality of the generations achieved by their proposed approach, then they should consider baselines beyond BigGAN (Table 2) for that. In this case, a comparison with clustering GAN approaches becomes even more important, as those are very natural baselines to compare the proposed framework against in terms of image generation quality.

Unfortunately, the authors chose to not include those comparison results during the discussion phase.

Therefore, the AC agrees with the reviewers' assessment and recommends to reject. The AC strongly encourages the authors to consider the reviewers' feedback to rethink the presentation of their work (what key messages they would like to emphasize) and improve its execution to make it more convincing.
justification_for_why_not_higher_score: Contributions are not well positioned within the literature: clustering GAN literature is disregarded under the claim that the paper's goal is to perform representation learning from a GAN discriminator, yet the paper proposes to train the discriminator by leveraging clustering losses, raising the question whether the discriminator of any clustering GAN model could be used out-of-the-box and achieve comparable/better results. The proposed approach is probed from a representation learning perspective and compared against contrastive SSL methods. However, the experimental protocol is not standard, making the claims in the paper unconvincing.
Decision: Reject