Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The demonstrate that effectiveness of the results on a wide variety of datasets and outperform current SOTA on the several benchmarks. They show that they get faster convergence than competing works like StyleGAN2 and show excellent qualitative results on very small datasets like the provided pokemon.
main_review: # Originality: 
I am not aware of any GANs that use pre-trained ImageNet embeddings in a GAN like so. Projects real and generated samples into a fix pre-trained feature space. There are some work that has used pretrained imagenet features to create knn clusters that are later used an auxilary classification task for the discriminator, but this approach seems sufficiently different to those approaches. The authors

# Quality:
 Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

Most of the claims in the paper are technically sound. The one issue I have is the entanglement between using ImageNet feature and entanglement with FID score. It's true they use a different architecture, EfficientNet than the InceptionNet does help a bit in that regard as stated in the paper. "We do not choose an inception network [59] to avoid strong150correlations with the evaluation metric FID." Inherently, EfficientNet is trained on the same data as InceptionNet though, so I would prefer if a non-ImageNet trained feature space was used for evaluation. Even another similar dataset that could be use for classification like Google OpenImages, MS CoCo classification, etc. That would strengthen the results in the paper. I do appreciate that they include other metrics like SwAV-FID (although this should be moved to the main paper as it's the most compelling quantitative result. 

Given the use of self-supervised FID assuages my concerns though. Also the Pokemon dataset results are pretty compelling. I'd also like to point that ImageNet and StyleGAN's architecture have similarly object centric biases (the objects are always in the center of the frame even in the Pokemon example). 

# Clarity: 
The submission is fairly well written and the qualitative examples are very convincing. The paper is very well written and the supplement is detailed.


# Significance:
The paper proposes a pre-trained feature layer to accelerate GAN training and improve results on small datasets. They show impressive results and demonstrates which techniques work and some that don't on guiding the GAN's using the feature spaces. It also shows how to utilize feature space like ImageNet on categories that are not included within. It also demonstrates some of the limitations of the approach like that fact that it actually hampers the results on certain datasets like FFHQ.


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates the use of pre-trained imagenet classifier networks as feature extractors for GAN discriminators. The author propose to use multiscale fixed random projections combined via an FPN-like approach, and explore which levels of the feature hierarchy are best fed into an ensemble of discriminators under different projection and mixing settings. Results are presented on a range of unimodal datasets (CLEVR, FFHQ, LSUN, etc) and compared against FastGAN and StyleGAN baselines, showing improvements particularly wrt sample efficiency.
main_review: My take:
This is a decent paper. For the most part, it is clear (with a few sharp edges, noted in my detailed notes) and easy to follow. The experiments are sensible and the results are reasonably strong, particularly with respect to sample efficiency. 

My primary concern with this paper would be that using pretrained features from classifiers in any way tends to strongly improve performance on *any metric which involves a pretrained classifier net,* (even one trained in a self-supervised fashion), but does not necessarily lead to actual sample quality improvements. E.g., for ImageNet GANs, this type of supervision leads to generating images which are more centred and emphasize the foreground, strongly improving both the rate of observed FID reduction and the final performance, but direct visual comparison makes it quite clear that the overall sample quality is not actually changed meaningfully.

While I appreciate that the authors include multiple metrics aside from FID in the appendix, these metrics are still nonetheless based on pretrained features which are susceptible to this phenomenon. I think this manuscript would be made substantially more airtight if non-pretrained-net-based metrics were used: reporting LPIPS and the sliced Wasserstein distance would go a long way on this front. Normally I wouldn’t much care to see these metrics, but because of this specific concern I think it pertinent. Based on the presented samples I do not expect the relative ordering of the comparisons to change.  In particular it is clear that regardless of final performance, the sample complexity is sharply improved by this method, so this is to me more a matter of adhering to standards of scientific rigour by ensuring that one cannot explain away the improvements by appealing to the aforementioned confounding factor.

Ultimately I think this paper will be of interest to practitioners operating in low-resource (data or compute settings), but not of especial interest to the field of (deep) generative modeling at large. That’s not a negative per se, but it does bear thinning about when guessing at its potential impact at NeurIPS. I’m rating this paper around a 6.5, and I expect a healthy discussion phase; if the authors address my concerns wrt metrics I would likely be willing to move up to a solid 7.


Notes:

-Section 3 acknowledges that if the feature extractor tosses information then the generative modeling objective is changed; it would be good to devote slightly more discussion to this on a theoretical level, perhaps in future work. I appreciate the discussion in Section 6 which demonstrates artifacts and potential negative downsides of the approach in this context.

“We observe that features at deeper layers are significantly harder to fit, as evidenced by our experiments in Section 4. We hypothesize that a discriminator can focus on a few descriptive feature channels while wholly disregarding others.”

This couplet needs a rework for several reasons. First, it seems the authors don’t mean that these features are harder to fit, but harder to *cover* (in the reverse KL sense); it sounds like these features are easier to overfit to. Second, this hypothesis is odd: focusing on a subset of the channels does not necessarily mean that D is overfitting, as it’s not only possible but highly likely that the feature space of any given pretrained network is not wholly utilized, so focusing just on a few channels may simply be because those channels are indeed the most informative. I think I understand the point the authors are trying to make, but they need to be careful and very specific with language here. Perhaps “focus on a subset of the feature space” would be a better choice

-” We do not choose an inception network [59] to avoid strong correlations with the evaluation metric FID”
While I understand the authors motivations here, this is misleading--using *any* classifier network will lead to strong correlations with the FID metric, regardless of if it is the same inception network used for the metric or not. This reviewer speaks directly from empirical experience, but evidence in the literature (that most classifier nets tend to learn approximately the same thing when trained on a given dataset) abounds. The supplementary provides several other metrics which should ameliorate this to some degree, but as mentioned in my main review the core issue is the use of classifier features to evaluate a model fit to classifier features, not the particular distance/divergence/Precision-recall curve used to evaluate the discrepancy between said features.


-In Table 1, the rows are unlabeled, and even after spending quite a bit of time reading the text, the caption, and examining the table, I was unable to ascertain with certainty exactly what each row corresponds to. Please label the rows of table 1 or explicitly explain in clear and direct language what they represent. I *think* the table is only meant to show what happens when you apply discriminators to different feature layers (with the highlights indicating which of the seven stages in an EffNet are being used as features), but this is not explained and doesn’t explain the meaning of the different values as one goes down a column. The column labels for this table are also problematic: they’re labeled as “FD” but they actually represent the ratio of performance relative to an RGB only baseline; while this is explained in the text, this is impossible to infer from the table itself and further adds to the confusion.


-The reduction in FD performance as discriminators are added to later layers is indeed interesting, but could potentially also be explained away by the known fact that (without contrary intervention) classifier networks tend to heavily rely on local structure and texture rather than global structure, and this is emphasized even in their deeper layers. While non-FID metrics are included in the appendix for the final models, it would be good to have them for this table as well.

-The authors mention wallclock time in several places but do not provide direct comparisons in terms of per-step training latency. These should be measured and included: since the sample complexity of the proposed method appears to outstrip the baselines by a large margin, then even if the use of the pretrained model were to incur a modest relative compute overhead, this would still favor the authors. In other words, what is the value of training imgs/s on a given GPU for each of the different methods?  A small table in the supplementary showing this would suffice and shouldn’t require any additional experiments, just the authors checking their training logs.

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The papers proposes a technique for faster training of GANs while achieving state-of-the-art FID scores on several datasets. The main idea is to use pre-trained features for the discriminator. Four improvements are recommended: 1) Multi-scale discriminators, 2) cross-channel mixing 3) cross-scale mixing 4) choice of pre-trained network.

FID scores are shown for several datasets, CLEVR, FFHQ, Cityscapes, Bedroom and Church.
main_review: The paper presents very impressive FID results across numerous datasets. However, it is unclear why these improved results were achieved. It is possible the paper could be accepted if the results could be clarified, but in its current state it is below the bar for acceptance.

Positives:

\++ Very good FID scores on many datasets.

\+ Very fast to train when compared to other approaches.

\+ Good job in calling out the approach's limitations.

Negatives:

\- It is unclear why CCM would make any difference in the network. A linear transformation before a linear layer is equivalent to a single linear layer. Why would CCM help training? This is particularly confusing since the labels in Table 1a are missing.

\- For CSM, did the authors compare against a discriminator that has access to all feature layers? It is unclear if this would help if the discriminators weren't independent.

\- The y-axis labels are missing in Table 1a and 1b. Without these labels it is hard to determine what the numbers actually mean.

\- Why are the values in Table 2 different from Table1? I.e. the FID scores are >1 in Table 2 but below 0.5 in Table 1. Is it because training was done for 10M images instead of 1M? If so, why are the results worse for larger training set sizes?

\- Is the reason the FID scores are so high due to the discriminator features being essentially classifier features? If so, the use of FID scores is suspect for evaluation. To be convincing, it would be better to run human studies or another metric that indicates the generated images are actually better. It is not clear from the supplementary material that the images generated are actually improved.


limitations_and_societal_impact: The paper did a great job of calling out their limitations in Section 6. No societal impact was provided beyond line 326.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to leverage pretrained image feature representations to improve GANs by training the discriminator on random projections of the pretrained features instead of directly on RGB images. This simple substitution results in improved image quality and diversity while also significantly reducing training time compared to existing state-of-the-art methods.
main_review: Strengths:
- Paper is technically sound. Authors do a good job of first motivating their solution by demonstrating that existing methods cannot leverage deep features from pretrained representations, and then demonstrate that their proposed solution successfully bypasses these limitation. 
- Ablation study compares several different SOTA feature representations.
- Generation quality is evaluated on a wide variety of datasets, compared to several other SOTA GAN architectures using a variety of GAN evaluation metrics.
- While the idea of leveraging pretrained feature representations to improve GAN training is certainly not new, the key insight that these representations need to be diluted in order to work well appears to be a novel contribution.
- The reduction in training time afforded by the techniques proposed in this paper could have significant positive impact in several ways, such as: lower barrier to entry for those who would like to train high resolution GANs but don't have the compute budget, faster iteration time for researchers, and less energy consumption/carbon emissions for training models. This is very promising as one of the main challenges with GANs currently is their excessive training time. 
- The paper is nicely organized and well written for the most part. 

Weaknesses:
- Some clarity issues - see below.

Unfortunately, I had a very difficult time understanding Section 4.1 and Table 1. I was not entirely clear on the experimental setup, particularly in relation to the rows in Table 1, which I could not find descriptions for. 
The discriminator architecture could also be described in more detail, even if only in the appendix. Currently the discriminator is described as "a simple convolutional architecture with spectral normalization" (lines 105-106), which is quite ambiguous. Considering that the discriminator is the focus of this paper, I think it is important to include sufficient detail to allow readers to replicate it. Releasing code would certainly help in this regard, but it is nice when a paper is self-contained.

Other questions/comments:
- I found it interesting that most of the performance improvement came in the form of better sample diversity (i.e. recall), rather than just image quality. I think this is a significant result and could be worth mentioning in the main paper rather than having it hidden in the appendix.
- How important is it to use multiple discriminators? i.e., how well would a perceptual discriminator perform if combined with CCM? Or alternatively, how well would FastGAN or StyleGAN2 perform with a separate discriminator for each resolution? (This information may have been in Table 1 but I couldn't tell).
- Considering that it was found that compact representations worked better than high performing classifier representations it would be interesting to test pretrained VGG, especially since it has been so successful for perceptual tasks in the past. (No need to run this experiment now, I am just curious).
- Do you have any intuition as to why FastGAN benefits more from projection compared to StyleGAN2? 
limitations_and_societal_impact: Authors briefly mention benefits of lower computational requirements and the possibility of malicious use such as in deep fakes.
review_title: null

Meta-review: The submission discusses projected GANs, i.e., GANs which project generated and real samples into a fixed, pretrained feature space. The initial reviewer assessment is generally positive. The reviewers provided further clarifications in the rebuttal which clarified questions. Reviewers remained assured about their initial assessment. AC thinks that this is a valuable study that will be of interest.
Decision: Accept (Poster)