Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a post-processing method to improve the GAN results. The post-processing consists of a projected gradient descent step to update the generated image to fit its target class, which works with both conditional/unconditional generative models. For the unconditional generative model, the authors design a de-bias method to force the model to uniformly generate images in each class. The results show this model can significantly improve the FID and IS scores on CIFAR-10 dataset. Additionally, the authors show this post-processing also works for image interpolation.

main_review: 
Advantages:
1. The improvement on the CIFAR-10 dataset is significant. And the refined images look much better than baselines.
2. The model can work with both conditional/unconditional generative models. 

Weakness:
1. In algorithm 1, it is not clear how to use epsilon. Please add more details.
2. In section 2.2, the robust classifier method optimizes both the input images and the network, but in section 3, it only optimizes the input images. The definition is not aligned.
3. Have the authors normalized the input images? Will the input pixel value between [0,1] or [0,255]? In the supplementary, the epsilon of VAE is 25. if the model uses a very large epsilon value, will the refined image still look realistic?
4. This paper only shows the CIFAR-10 results. Will this method generalize to large-scale and high-resolution datasets, such as ImageNet? Currently, people are interested in high-resolution image generation. For example, BigGan can generate high-resolution images. Please the authors test this post-procession method on these generated high-res images.
5. The projected gradient descent has already been used in some image generation works [1].
6. The FID and IS scores of cGAN-PD and BigGAN in this draft are different from the original papers. Please add more details. Also, missing the definitions of '10K' and '50K'.



[1] Xia, Weihao, et al. "Gan inversion: A survey." arXiv preprint arXiv:2101.05278 (2021).
summary_of_the_review: Given the results and novelty are marginal, I think this paper is between boardline. Please the authors address my questions. I am happy to change my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model-agnostic method for improving the quality of images produced by generative models. The method requires a robust classifier trained on the same data source as the generative models. Based on the perceptually aligned gradients phenomenon, the proposed method improves the quality of a generated image by the targeted projected gradient descent method with the help of the robust classifier. Experiments on CIFAR-10 show that the proposed method does improve several generative models both quantitively and qualitatively.
main_review: The proposed method has some nice properties. It is model-agnostic and can be used to improve generative models without re-training. The only requirement is a robust classifier trained on the same dataset as the generative models. As stated in the paper, training a classifier is much easier than training a generator. Even better, the same classifier can help all generative models trained on the dataset. These characteristics are attractive in applications. 

There are two weaknesses of the paper. First, the experiments are on the low-resolution CIFAR-10 dataset. Although the method is effective on the CIFAR-10 dataset, it is uncertain how well the proposed method performs on normal quality images. For 32x32 images, it is impossible to check out details. Also, low-resolution images are not very useful in real applications. As classifiers could focus more on high-level features, it is not clear how they help synthesize realistic details for high-resolution images. 

Second, as stated in the paper, the method is similar to prior work, Santukar et al. 2019 and Turner et al. 2019. The paper lists three differences: (1) the proposed method builds upon the generated images; (2) the proposed method is model-agnostic; and (3) the proposed method can be performed on every image generated by the generative models without throwing out anyone. Although the technical novelty is not significant, the proposed method has advantages and could be useful in applications.

summary_of_the_review: Overall, I like the proposed method because it is model-agnostic, simple, and effective. However, as stated in the previous section, since experiments are only performed on low-resolution images, it is not clear how the proposed method performs on images with ordinary resolutions and quality. Novelty is another potential issue, although I think that the paper has made sufficient contribution given its nice characteristics listed above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes BIGRoC (Boosting Image Generation via a Robust Classifier), a method to refine samples from a base generative model using a robust classifier. In this context, a robust classifier means one that has been trained to be robust to adversarial samples. Given a robust classifier and a base generative model (both trained on the same dataset), the key idea involves using the gradients of the robust classifier to update samples generated from the base generative model in the direction that maximizes the conditional probability of the samples' class. Experiments on the CIFAR-10 dataset have been presented that demonstrate that BIGRoC improves the sample quality from several base generative models in terms of the FID and IS metrics.
main_review: [Writing]

The manuscript is well-written and easy to understand. 

[Novelty, Significance, and Prior Work]

The authors have explored an interesting idea of using robust classifiers for sample refinement. However, the key technical contribution is minor compared to Santurkar et al. [1] — it only involves using a robust classifier on pre-trained generative models. One might argue for the novelty of application, however, the authors have ignored several works that address the problem of sample refinement, e.g., using GAN discriminators [2, 3, 4]. In contrast to these works, BIGRoC also has the additional overhead of training a robust classifier. Moreover, these prior works bring in new technical perspectives to the problem of sample refinement (e.g., from optimal transport [2], energy-based modeling [3], gradient flows [4]) unlike the proposed method which hardly brings in any new technical insights.

[Empirical Evaluation]

It is appreciable that the experiments have been conducted on several base models. However, only the CIFAR-10 dataset has been studied. It is unclear:

- How the method will perform on other datasets?
- How sensitive is the method to the training of the robust classifier?
- What is the additional time overhead of training a robust classifier?

Furthermore, the method has not been evaluated against any baseline methods (e.g., [2, 3, 4]). 

[1] Santurkar, Shibani, et al. "Image synthesis with a single (robust) classifier." *arXiv preprint arXiv:1906.09453* (2019).

[2] Tanaka, Akinori. "Discriminator optimal transport." *arXiv preprint arXiv:1910.06832* (2019).

[3] Che, Tong, et al. "Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling." *arXiv preprint arXiv:2003.06060* (2020).

[4] Ansari, Abdul Fatir, Ming Liang Ang, and Harold Soh. "Refining deep generative models via discriminator gradient flow." *arXiv preprint arXiv:2012.00780* (2020).
summary_of_the_review: The paper explores an interesting idea of using robust classifiers to refine samples from generative models; however, the technical contribution is limited. Several directly relevant methods of sample refinement have neither been discussed nor compared against. Overall, the contribution is not significant enough to warrant acceptance.
review_title: null

Meta-review: The paper proposes to improve generated images via a post-processing update procedure guided by gradients from a robust classifier.  After the author response and discussion, all reviewers agree that the paper is below the acceptance threshold.  Reviewer concerns include limited technical novelty and missing experimental comparison to relevant baselines.
Decision: Reject