Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: 
The authors argue that GAN discriminators experience catastrophic forgetting due to instability during training (the exact source of the instability the authors refer to is not clear). The authors claim that self-supervised methods, that provide an auxiliary loss (via an auxiliary self-supervised task),for GAN training can help prevent catastrophic forgetting, however they point out two weaknesses with the current methods:
(1) The auxiliary self-supervised losses do not consider the data synthesised by the generator.
(2) They have additional hyper-parameters that need to be fine-tuned to balance losses.

In this paper the authors train the self-supervised classifier on the transformed generate data as well as the transformed real data. Rather than having a separate classifier the discriminator predict the label, this avoids the need for balance hyper-parameters.
main_review: **Originality**

- The authors extend self supervised GANs to learn a join distribution p(real, label) rather than learning each independently.

**Quality**

- It’s good that the authors compare to other data augmentation methods to distinguish the benefits of self-supervised training from those of data augmentation.

- Figure 2: It would be helpful if all figures shared the same y axis this would highlight better that benefits of SSGAN-LA over other the baselines.

- Synthetic dataset: in general (and also according to this paper) when training a generative model the objective is to learn a distribution that matches the training data distribution. Given this, I’m unsure why experiments in section 5.1 are trying to recover the distribution of the augmented data? From my understanding of the paper the goal should be to recover the original training data distribution, the data augmentation used in self-supervised learning should only improve the model’s ability to recover that distribution. Ultimately, these results suggest that the SSGAN-LA model learns to generate samples from outside the original training data distribution? Some clarification on this would be really helpful, thank you.

Further in Section 5.2.1, qualitative results: the authors suggest the fact the DAGAN synthesises transformed data samples is a weakness, adding additional confusion to the motivation behind results in Figure 2.

- The authors compare SSGAN-LA to a large and relevant number of baselines models on three natural image datasets.

- Table 1: The authors suggest SSGAN-LA outperforms all baselines on all tasks, reporting FID and IS in all cases. It’s not clear that SSGAN-LA is statistically significantly better than DAGAN-MD on the STL-10 task, may the authors please clarify? How reproducible are these results? How many runs did you do?

- In some cases the authors have put answers to the Checklist in the checklist rather than in the paper. For example the hardware used for training and the implementation details. These should be in the main paper.

- 5.2.2 Image classification, Table 2: the authors show that the discriminator learns representations useful for classification. The SSGAN-LA appears to show some improvement, however the authors do not provide stdev. so it is hard to tell if these improvements are statistically significant. For example, CIFAR-100 SSGAN-MS performance is very close to SSGAN-LA.

**Clarity**

- Line 23: What do the authors mean by a “non-stationary learning environment”? Is there a more precise definition? Are you referring to the learning dynamics or the distribution of synthesised data?

- Line 31: What do you mean by “stable representations”? Are these representations that help to improve stability during training? Or representations that don’t change significantly between training steps? 

- Line 48 - 54: From the introduction it’s not clear what the discriminator will predict. Line 48-49 suggests it predicts only the label, but 52-53 suggests it predicts “real” or “fake”

- Line 100: What is meant by “enforces the generator to produce rotation-detectable image”? Do you mean images, whose orientation can be easily predicted (by a classifier)? 

- Figure 1: What is SNGAN?

- Proposition 1: To have notation more consistent with the GAN literature,  it would be good to super-script D with a * e.g. D*(k, 1|y) = … .

- Line 159 - 161: The authors claim “Compared with standard GANs, the transformation prediction especially on real samples of SSGAN LA establishes stationary self-supervised signals to prevent the discriminator from the forgetting problem.” - what is means by stationary self-supervised signals? At this point it is still not clear how training signals from the self-supervised tasks help to prevent catastrophic forgetting?

 - Does Theorem 5 apply to DAGAN-MD,  SSGAN-LA or both?

- On a first parse it was not clear that the role of the  “The importance of Self-supervision” section was to introduce baselines? It may make sense to refer to this section as “Related Work” or “Related Work: The importance of Self-supervision” or to include is in the “Experiments” section. 

**Significance**

- The approach presented in the paper is an incremental improvement on existing SS-GAN architectures and the improvements.

- There is no clear theoretical contribution in the main body of the paper.

- The link between the proposed approach and catastrophic forgetting are not made clear in the paper. 

- The authors do show some improvements on image synthesis, and show the application of the learned representations to image classification.

- The proposed approach does not require a hyper-parameter to trade off between the self-supervision loss and the discriminator loss.
limitations_and_societal_impact: The authors do not discuss the limitations of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a label augmentation GAN that combines the discriminator and the classifier in self-supervised learning GANs and show the priority on a variety of generative modeling tasks.
main_review: originality: The idea is novel. Combination of the classifier and the discriminator sounds natural and is an improvement over self-supervised GANs.

quality: The theorems look correct and the experiments seems solid. The only thing is that the numbers reported by the authors in Table seems to be different from BigGAN and SSGAN-MS. For example, BigGAN authors reportThe author may need to explain why. For example, the BigGAN authors report IS 9.22 and FID 14.73 which are different from the authors' IS 8.42 and FID 10.83. It may be an architecture difference but why not use the same architecture as BigGAN.

clarity: The paper is clearly written. The authors may cite [45] for theorem 2 and change "discriminator" to "classifier" in line 108. It seems that "classifier" and "generator" are competing against each other in the context.

significance: The only concern I have about the paper is the significance. The method seems to be novel but not significant. It seems to be a natural extension of self-supervised GANs. The paper seems to contain some theorems but there are still many loose points. It's unclear to me why self-supervised GAN can learn "stable" and "high-level" representations. I may be careless and skip some important details. Could the authors help me find which theorem in the paper gives the guarantee that the representation learned is "stable" and "high-level"?
limitations_and_societal_impact: The authors do not mention negative societal impact. One negative impact can be some improper use of generative models, such as DeepFake.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper presents an approach for training GANs with transformation-based self-supervision. It unifies the original GAN task with the self-supervised task into a single multi-class classification task for the discriminator by augmenting the real/fake labels with the self-supervised labels. The authors demonstrate that the proposed approach outperforms some existing transformation-based self-supervised and data augmentation GANs using experiments on CIFAR-10, STL-10 and Tiny-ImageNet. They also provide a theoretical analysis of their method. 
main_review: **Strengths**:


- The proposed approach is intuitive, and outperforms some of the existing self-supervised and augmentation GANs. 


**Weaknesses**:


- There are other works not included in comparisons. Karras et al. [A] present a method for training GANs with limited data, which achieves an FID of 2.92 on CIFAR-10. This is better than the FID of 5.87 reported in this paper. Another relevant work is [53] (differentiable augmentation GAN) which is not included in comparisons. The authors mention that "we do not aim to compare with state-of-the-art data augmentation GANs in this work" (L 179). This is unclear to me as they already provide a comparison with another augmentation method (DAGAN). Both self-supervised and augmentation GAN approaches aim to improve the final generation quality, so comparison with methods in both of these domains is expected. 


-  Contributions of the paper are modest for this venue. The proposed approach is quite similar to SSGAN(-MS), with some relatively minor differences such as applying self-supervision to real samples as well. Theoretical analysis in the paper is also similar to the original GAN paper. 


- Resolution and complexity of the datasets used in the paper are limited. The maximum resolution used is 64x64 for Tiny ImageNet. [A, 53] show results on higher resolution datasets.    


[A] Training Generative Adversarial Networks with Limited Data; Karras et al.; NeurIPS 2020
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In the context of using self-supervision signals in the GAN training framework, this paper proposes to use a multi-class classification approach in the discriminator paired with the task of predicting the rotation of an image.  This is proposed instead of using two separate discriminators: one for the rotation prediction and the other for the real/fake classification.
The idea supposes a small and simple change to existing methods that, although in terms of novelty it is limited, it is justified theoretically and quantitative empirical results support the idea.
main_review: The paper is generally well written and clear. In my opinion, the experimental section is sufficient and it is tested on enough datasets. The idea is simple but well motivated, and it improves over the self-supervised baselines. 

Although the paper states that it does not aim to compare against state of the art data augmentation techniques, it is a very important comparison to do. If data augmentation works better than self-supervision in GANs, why use self-supervision to begin with? In this case, it is not clear that the data augmentations can directly be combined with the self-supervised rotation signals. 

Moreover, the section regarding data augmentations in the paper (and the baseline comparisons) revolve around rotations, which is not the best data augmentation strategy one could use. Moreover, the paper talks about transformation leaking, when in [1], experimental results show that, as long as the non-transformed (0 degrees) image is seen enough times, leakage should not occur. Why was this not considered in this paper, whether it is in the discussion part and/or the experimental results? Could the authors elaborate on this? My concern is that the data augmentation baselines might be at a disadvantage and therefore, be an unfair comparison. 

As an additional thought, the idea of using rotations as self-supervision signals in the self-supervision community is arguably outdated. There is other self-supervision tasks or transformations to predict in the discriminator that could be borrowed from the self-supervised learning community to apply them to GAN training, and perhaps be more competitive with the extensive data augmentation proposed in [1], for example. 

L.90: if I understand correctly, the generator wants to fool the rotation classifier as much as if wants to fool the discriminator, so 
I suggest changing the term “collaborate” for another more appropriate word. 

[1] Zhao, S., Liu, Z., Lin, J., Zhu, J. Y., & Han, S. (2020). Differentiable augmentation for data-efficient gan training. arXiv preprint arXiv:2006.10738.

limitations_and_societal_impact: The authors have not addressed the societal impact.
review_title: null

Meta-review: This was a borderline paper with a good amount of discussion both between the reviewers and reviewers and authors. I'll try to summarize some of the pros and cons of this work:

* novel yet simple extension to SSGAN. Simplicity is good, but some reviewers have argued that contribution is incremental.
* the theoretical contribution is not in the main body of the paper (this is easily fixed)
* the link with catastrophic forgetting should be made more obvious in the paper itself.
* the results are on small resolution sets only (some reviewers argued that access to resources is not an excuse, but I respectfully disagree).

All in all, I am inclined to accept this work based on the fact that it's a simple, principled, theoretically grounded change that seems to work well in comparable empirical settings on lower-resolution image sets. And I think self-supervised GANs are an interesting practically useful line of research, where novel insights may be useful to many researchers.

 
Decision: Accept (Poster)