BIGRoC: Boosting Image Generation via a Robust Classifier
Authors: Roy Ganz
Michael Elad
Venue: ICLR 2022 Submitted (2022)
Keywords: Image Generation
Adversarial Robustness
Perceptually Aligned Gradients

Abstract: The interest of the machine learning community in image synthesis has grown significantly in recent years, with the introduction of a wide range of deep generative models and means for training them. Such machines’ ultimate goal is to match the distributions of the given training images and the synthesized ones. In this work, we propose a general model-agnostic technique for improving the image quality and the distribution fidelity of generated images, obtained by any generative model. Our method, termed BIGRoC (boosting image generation via a robust classifier), is based on a post-processing procedure via the guidance of a given robust classifier and without a need for additional training of the generative model. Given a synthesized image, we propose to update it through projected gradient steps over the robust classifier, in an attempt to refine its recognition. We demonstrate this post-processing algorithm on various image synthesis methods and show a significant improvement of the generated images, both quantitatively and qualitatively.

Under review as a conference paper at ICLR 2022
BIGROC: BOOSTING IMAGE GENERATION VIA A Ro- BUST CLASSIFIER
Anonymous authors Paper under double-blind review
BigGAN output
BIGRoC output
7.D: IS: 9.38
FID: 6.79, IS: 9.47
Figure 1: BIGRoC results: left: images of 3 classes (birds, cars and trucks) generated by BigGAN Brook al. 2019). trained on CIFAR-10. Right: the enhanced versions of these images, attained by the proposed BIGRoC algorithm.
ABSTRACT
The interest of the machine learning community in image synthesis has grown significantly in recent years, with the introduction of a wide range of deep gen- erative models and means for training them. Such machines' ultimate goal is match the distributions of the given training images and the synthesized ones. this work, we propose a general model-agnostic technique for improving the image quality and the distribution fidelity of generated images, obtained by any generative model. Our method, termed BIGRoC (boosting image generation via robust classifier), is based a given robust classifier and without a need for additional training of the genera- tive model. Given synthesized image, propose to update through projected gradient steps over the robust classifier, in an attempt to refine its recognition. We demonstrate this post-processing algorithm various image synthesis methods and show a significant improvement of the generated images, both quantitatively and qualitatively.
post-processing procedure via the guidance of
INTRODUCTION
Deep generative models (DGMss are class of deep neural networks trained to model complicated high-dimensional data (Bond-Taylor et al. 2021). Such models receive a large number of samples that follow certain data distribution, x PD x). and aim to produce samples from the same statis- tics. One of the most fascinating generative tasks is image synthesis, which is notoriously hard, due to the complexity of the natural images' manifold. Nevertheless, deep generative models for image synthesis have gained tremendous popularity recent years, revolutionized the field and became state-of-the--at in various tasks (Brock al. 2019 Karras et al. 2020b) (hhu et al. 2017). Energy- based models, variational autoencoders, generative adversarial networks (GANs), autoregressive likelihood models, normalization flows, diffusion-based algorithms and more, all aim to synthesize natural- natural-lookig images, ranging from relatively simple to extremely complicated generators, often containing millions of parameters (Kingma & Welling. 2014 Goodfellow al., 2014 Oord al.)
2016 Rezende & Mohamed, 2015 Ho et al. 2020).
When operating on a multiclass labeled dataset, as considered in this paper, image synthesis can be either conditional unconditional. In the unconditional setup, the generative model aims to

---

Under review as a conference paper at ICLR 2022
produce samples from the target data distribution without receiving any information regarding the target class of the synthesized images, i.e., sample from Pp(x). In contrast, in the conditional setup, the generator goal is to synthesize images from designated class, i.., sample from Pp (zlyy) where y is the label. As such, conditional generative models receive additional class-related information.
Most of the work in the deep generative models' field has been focusing on improving the quality and the variety of the images produced by such models, tackled by seeking novel architectures and training procedures. In this work, while still aiming to improve the performance of trained generative models, we place a different emphasis than in most of these studies and propose method for boosting generative models without any re-training or fine-tuning. More specifically, our method improves the perceptual quality of the images synthesized by any given model via iterative post- processing procedure driven by robust classifier.
With the introduction of learning- based machines into "real-world" applications, the interest in the robustness of such models has become a central concern. While there are abundant of definitions for robustness, the most common and studied is the adversarial one. This definition upholds if a classi- fier is robust to small perturbation of its input, made by an adversary in order to fool it. Previous work (Szegedy et al., 2014) has demonstrated that deep neural networks are not robust at all and can be easily fooled by an adversary. In light of this observation, many robustification methods were proposed, but the most popular among these is adversarial training (Goodfellow et al., (011)). Ac- cording to this method, in order to train a robust classifier, one should generate adversarial examples and incorporate them into the training process. While examining the properties of such classifiers, researchers have revealed a fascinating phenomenon, called perceptually aligned gradients (Tsipras iet al. 2019). According to this tendency, a modification of an image that sharpens such a classifier's 's decision yields visual features that are perceptually aligned with the target class. In other words, when drifting an image content to be better classified, the changes obtained are visually pleasing and faithful to natural image content.
In this work we harness and utilize the above described phenomenon - we propose to iteratively modify the images created by a trained generative model, so as to maximize the conditional prob- ability of certain target class, approximated by given robust classifier. This modification can potentially improve the quality of the synthesized images, since it emphasizes visual features that are aligned with images of the target class, thus boosting the generation process both in terms of perceptual quality and distribution faithfulness. We hypothesize that given an image dataset, the su- pervised training of robust classifier is much simpler and effective than the unsupervised training of a generative model, thus enabling an indirect yet powerful improvement of generative models. We term this method "BIGRoC" - Boosting Image Generation via a Robust Classifier.
The method presented in this article is general and model-agnostic, and it can be applied any image generator, both conditional or unconditional. In the unconditional case, since we do not have a target class to guide the boosting process, we propose to estimate it via the trained robust classifier. The marked performance improvement achieved by our proposed method is demonstrated in series of experiments on a wide range of image generators. We show that this approach enables us to significantly improve the quality of images synthesized by relatively simple models, boosting them to a level of more sophisticated and complex generators. Furthermore, we demonstrate the ability of our method to enhance the performance of higher-quality generative architectures, both qualitatively and quantitatively. In addition to our contribution on enhancing image generation performance, we leverage the same robust classifier and its perceptually aligned gradient property for proposing an adversarial technique for image interpolation.
Our approach is inspired by two main works: Santurkar et al. (2019) and Turner et al. (211)) San- turkar et al. (2019) have shown that single robust classifier is capable of tackling various computer vision problems, such as generation, inpainting and image-to-imane lizing the perceptually aligned gradients phenomenon. Turner et al. (2019) aims at improving the generation quality of GANs, by discarding low quality images, identified by the GAN's discrimina- tor. While our work is related to the above two papers, it differs substantially in the following main characteristics: (i) While our boosting technique relies on the perceptually aligned gradient property as in Santurkar et al. 2019). it builds on the outcome of pre-trained generative models, thus getting
translation, all achieved by uti-
2

---

Under review as a conference paper at ICLR 2022
much higher quality images; (ii) Our proposed boosting operate any image synthesizer; and (iii) All images produced by the generator are taken into account, none being discarded.
2
BACKGROUND
2.1 ADVERSARIAL EXAMPLES
Adversarial examples are instances that intentionally designed by an attacker cause false prediction by a machine learning-based classifier (Szegedy et al. 2014). The generation procedure of such examples is based on applying modifications to given training examples, while restricting the allowed perturbations A. Ideally, the "threat model" A should include all the possible perturbations that are unnoticeable to a human observer. As it is impossible to rigorously define such set, in practice simple subset of the ideal threat model is used, where the most common choices are the l and the foc balls: = {6 : |61l2/0= S <}. Given A, the attacker receives an instance x 8 s.t. 8 A. while aiming to fool the classifier. Adversarial attacks can
and generates 2 be both untargeted or targeted: An untargeted attack perturbs the input p(yl&) with respect to 5. In contrast, targeted attack receives in addition the target class ú, and perturbs x to maximize p()).). There are diverse techniques for generating adversarial examples, yet, in this work, we focus on targeted attacks using the Projected Gradient Descent (PGD) method (Madry et al. 2018) an iterative method for creating adversarial examples that operates as shown Algorithm
way that minimizes
Algorithm 1: Targeted Projected Gradient Descent (PGD) for adversarial targeted attacks
Input: classifier fo, input x, target class ú. c, step size a, number of iterations T
5u 40
for 1. from 0 do 6,+1 II, (6, - (V
end
Tade
= =x 8T
Output: Tadv
The operation 11, stands for a projection operator onto A, and E(-) is the classification loss.
2.2 ADVERSARIAL ROBUSTNESS
Adversarial robustness is property of classifiers, according to which, applying small perturbation on a classifier's input in order to fool does not affect its prediction (Goodfellow et al. 2015). To attain such classifiers, should solve the following optimization problem:
min Σ max FVEDD
(1)
Namely, train the classifier to accurately predict the class labels of the "toughest" perturbed images, allowed by the threat model A. In practice, solving this optimization problem is challenging, and there are several ways to attain an approximated solution. The most simple yet effective method based on approximating the solution of the immer-mxximizatio via adversarial attacks, such as PGD. According to this strategy, the above optimization is performed iteratively, fixing the classifier's parameters and optimizing the attacks for each example via PGD, and then fixing these and updating f. Repeating these steps results in a robust classifier, as we use in this work.
2.3 PERCEPTUALLY ALIGNED GRADIENTS
Perceptually aligned gradients is a phenomenon that occurs in adversarially trained models when modifying an image to maximize the probability assigned to a claget class. Tsipras et (2019) show that performing the above PGD process such models yields meaningful visual features that
3

---

Under review as a conference paper at ICLR 2022
perceptually aligned to the target class. It is important to note that this phenomenon does not occur in non-robust models. The perceptually aligned gradients property indicates that the features learned by robust models are more aligned with human perception. Figure presents a visual demonstration of this fascinating phenomenon.
Input
Targeted POB
plane
dear
frog
Robust
model
2=10,.-100 P-
P-1A.--101
P=18.-=200 P-10,r-178 P-10.
P-10,x-201 p-10.--156 P-10,e-196 P-10.r-206
plane
bird
cac
deer
dog
horse
ship
truck
Non- robust
model
P=10.c=45 P=10..-6G
P=10..
F-10.cw45
P+18 5-52
F=10.--42 P-10 P-10.c-45 1=10
P=10.
Figure 2: Demonstration of the perceptually aligned gradients phenomenon on CIFAR-10: the origi- nal image (left) is perturbed via a targeted PGD attack maximize the probability of different target classes. This experiment is done with a robust and non-robust classifiers of the same architecture ResNet50 (H. et al. 2016). The top row shows the results for the robust classifier and the bottom for the non-robust one. We specify below each image the relevant classifier's 's certainty (denoted as generated images, the attacked classifier reaches absolute certainty (P = 1). As can be seen, an adversarial attack (PGD) on a robust classifier leads to a significant addition of perceptual features aligned with the target class. This phenomenon does not occur at all the non-robust classifier. Although both networks use the same threat model A, we see that the attack on the non-robust classifier does not utilize the entire threat model and perturbs the image in seemingly insignificant way, but still able to completely fool it. Experines innetal settings. A = {8 ||ō1|2 I c), c = 30, T = 60, f 0.5.
P), and the effective &e norm the perturbation 5 (denoted as e). For each
BOOSTING IMAGE GENERATION VIA ROBUST CLASSIFIER
We propose a method for improving the quality of images synthesized by trained generative models, named BIGRoC: Boosting Image Generation via Robust Classifier. Our method is model agnostic and does not require additional training fine-tuning of the generative model, and can be viewed as post processing step. Thus, BIGRoC can easily applied to any generative model, both condi- tional or unconditional. This mechanism harnesses the perceptually aligned gradients phenomenon to further tweak the generated images to improve their visual quality. To do so, we perform an iterative process of modifying the generated image to maximize the posterior probability of given target class ÿ. po (ū) (.). where pe is modeled by an adversarially trained classifier. This can be achieved by performing PGD-like process, but instead of adversarially changing an image x of class to a different class ÿ + we propose to modify it in a way that maximizes the probability that xr belongs y. Therefore, our method requires a trained robust classifier that operates on the same data source as the generator we aim to improve.
In the conditional generation process, the generator G receives the class label y. from which suppose to draw samples. Hence, in this setup, have information regarding the class affiliation of the image and we can maximize the corresponding conditional probability. In the unconditional generation process, the generator does not receive class labels all and its goal is to draw samples from p(x). Thus, in this case, we cannot directly maximize the desired posterior probability, as our method suggests. To bridge this gap, propose to estimate the most likely class via our robust classifier fe. and afterward modify the image via the suggested method to maximize its probability. The proposed image generation boosting described in Algorithm for both the conditional and the unconditional schemes.
While the above-described approach for unconditional sampling works well, it could be further im- proved. We have noticed that in this case, estimating the target classes Y of X gen via so leads
4

---

Under review as a conference paper at ICLR 2022
to unbalanced labels. For example, in the CIFAR-10, when generating 50,000 samples, expect approximately 5,000 images per each of the 10 classes, and yet the labels' estimation do not dis- tribute uniformly all. This imbalance causes bias in the target labels estimation of the boosting algorithm, affects the visual content of boost and limits the quantitative improvement attained by BIGRoC. We emphasize that this issue is manifested only in the quantitative metries, and when qualitatively evaluating the boosted images, the improvement is significant, as can be seen in Figure 3
To further enhance the quantitative results of our algorithm in the unconditional case, we propose to de-bias the target class estimation of Xgen, and attain close to uniform class estimations. naive solution to this can be achieved by generating more samples and extracting subset of these im- ages with a labels-balance. This approach is computationally heavy and does not use the generated images as-ís, which raises questions regarding the fairness of the quantitative comparison. Thus, we propose a different debiasing technique- we modify the classifier's 's class estimation to become more balanced by calibrating its logits. More specifically, we shift the classifier's logits by adding per-class pre-calculated value, de, that induces equality of the mean logits value across all classes. For simplicity, we denote logite, as the logit of class Gj corresponding to generated sample 2 genn- We approximate Exg.. [logite, for each class Cs using validation set of generated images, and cal- culate a per-class debiasing factor: de, = a Te-y [Logite,] (WLOG, a = 1), where Ezzc.. [logite,] is a mean estimator. After calculating de,, given a generated image ggen we calculate its logits and add de, to it to obtain debiased logits (logite.) ). from which we derive the unbiased class estimation via softmax. The following equation shows that, given correct estimation of the per-class logits' mean, the per-class means of the debiased logits are equal:
B Byyen [Logite]] = Bergen [de; + -logite;] = Exyen (a Feryen [logite]] + logile,] =
a By gen [logite:] + Exzen [logite,] =
Figure presents demonstration that verifies the validity of this method and shows its qualitative effects on the unconditional generation boosting.
Algorithm 2: BIGRoC: Boosting Image Generation via a Robust Classifier
Input: Robust classifier fo. Fgens Bgens e, step size a, number iterations T, de
if Ugen is None then
Ugen orgmarr So (Xgge)) ) + de)
end
Tboost Targeted PGD genn Ogens e, a,
Output: Thoost
As can be seen in Algorithm 2 it receives as input the generated images and their designated labels (if exist) and returns improved version of them. As such, this method can be applied at the inference phase of generative models to enhance their performance, in a manner totally separated from their training. As can be from the algorithm's 's description, it has several hyperparameters that determine the modification process of the image: e sets the maximal size of the perturbation allowed by the threat model A, a controls the step size at each update step and T is the number of updates. Another choice is the norm used to define the threat model A.
The byperparameter c is central in our scheme when is too large, the method overrides the input and modifies the original content in an unrecognizable way, as be seen in Figure 2 On the other hand, when é is too small, the boosted images remain very similar to the input ones, leading to a minor enhancement. As our goal is obtain significant enhancement to the synthesized images, careful choice of e should be practiced, which restricts the allowed perturbations in the threat model.
Another important choice is the threat model A itself. Two of the most common choices of A for adversarial attacks are the foo and the & balls. Due to the desired behavior of our method, using the foe ball is less preferable: it allows a change of te to every pixel and as such it will not focus on meaningful specific locations and might not preserve the existing structure of the synthesized input image. Thus, we choose the &g ball as our threat model, with relatively small e. Such a choice restricts the allowed perturbations and leads to changes that may concentrate in specific locations,
5

---

Under review as a conference paper at ICLR 2022
Boesting without Debiasing
Boosting without Debiasing
Boosting Debiasing
X gen
class ids
Boosting with Debiasing
Figure 3: Demonstration of the debiasing technique: We show 100 generated images by an uncon- ditional SNGAN and the results of the BIGRoC algorithm, with and without the proposed debi- asing. As can be seen, the outputs of the boosting algorithms are perceptually superior, while the histograms expose the fact that the suggested debiasing algorithm induces a more uniform labels' distribution. In the "Boosting without Debiasing" experiment, 36 out of 100 images are classified as deers, and only are horses. The most prominent deer images are marked in red. However, when applying the debiased boosting, the number of deers reduced to 9, and the number horses increased to 15. We mark the boosted images that remain deer in red, and images that are modified to other minority classes in green. As can be seen, many of the deers were changed to horses, perceptually similar class.
while preserving most of the existing content in the generated images. visual demonstration of these considerations is given in Figure
Generated Images
Lop based boost
Le bo-based boost
Figure 4: The effect of the threat model's choice: generated images via conditional GAN (left) are perturbed via a targeted PGD attack to maximize the probability of their target classes (planes, and birds) using either fee or le threat models. The boosted outputs attained by the &go entirely change the structure of the images and lead to unnatural results. Using the E2 threat model leads to pleasing and better results,
claimed. Experimental settings: In the l2 case, we use
10, T 30, a = T 3. which leads to good visual results. In order to make sensible comparison with the lac. we measure the perturbation each pixel made by the based boosting algorithm and calculate the 70% percentile of the perturbations, denoted 40.7. Namely, 40.7 is larger than 70% of the perturbations done by our method using the &a threat model. For the Ex case, set to be equal to the 90.7, while leaving the other hyperparameters unchanged.

---

Under review as a conference paper at ICLR 2022
4 ROBUST CLASSIFIER BASED IMAGE INTERPOLATION
As byproduct of the boosting algorithm for improving the quality of synthesized images, de- velop a method for image interpolation, which stems from a similar rationale. In image synthesis, image interpolation is an operation that aims generate intermediate pictures between two given images, while remaining meaningful and of gradual change. We propose to use features extracted by our robust classifier different scales in order to guide a PGD-based interpolation process. More specifically, given a souree image 2, and a target image 2;, we extract their corresponding features {FI,..... Fr., } and {Fa,.. ....... F } from our trained robust classifier fo- To perform interpolation, we modify Is such that its features will be more similar to those of 24 at the different scales, using
PGD. Mathematically, we aim to solve Equation
arg minses
|
2.45
F ll2
(3)
where {w; ) is an importance vector to the different features' scales. Namely, this optimization aims to find perturbation ó E that minimizes the fz-distance between the images' representa- tions. In the above formulation, the choice of the determines the level of change allowed in the process. Thus, solving the above optimization with different e values leads to different intermediary images as we increase the interpolation resembles the target image more.
5 EXPERIMENTS
Image Generation Enhancement: We conduct extensive experiments to demonstrate the per- formance improvement achieved by the proposed method. In this work, we focus on CIFAR-10 10 (Krizhevsky 2012), the most common benchmark for image synthesis. Since BIGRoC is model- agnostic, it can be easily applied to any generative model, given a robust classifier, trained on the same data source. Although our proposed method is a general technique, we experiment mainly with Generative Adversarial Networks (GANs), due to their popularity and generation quality. In all of our experiments, we use single adversarially trained ResNer-50 (H. al., 2016) as the ro- bust classifier. We further elaborate regarding the experimental setups in Appendix [ We utilize the agnel-ag property to examine the effects of applying the proposed boosting over wide variety of image generators of different qualities: from relatively simple ones to sophisticated and complex ones. Moreover, we test our approach on both conditional and unconditional generative models to show that the proposed scheme can enhance different synthesis procedures.
Given a generative model, we use it to synthesize a set of images X Xgen and apply our method to generate Xboost according Algorithm 2 We compare between Xg and X bboost quantitatively using two image synthesis evaluation methods: Fréchet Inception Distance (FID) (Heusel et al.) (017) and Inception Score (IS) (Salimans al.) (2016). Our quantitative results (Tables [ indicate that our method achieves a substantial improvement across wide range of tested generator architectures, both conditional and unconditional, demonstrating the method's 's versatility and validity. In addition, the fact the same robust classifier is capable of enhancing the performance of such a wide range of generative models, demonstrates the power of BIGRoC. Furthermore, we show in Figure [] S and in Appendix _ the qualitative results that indicate that the "boosted" results are more pleasing and clear to human observers. All the conducted experiments verify the significant enhancemenn attained by method, both quantitatively and qualitatively.
We conduct image interpolation experiments according to the technique pro- posed in Section 4 We use the ImageNet dataset (Deng et al., 2009) with resolution 224 x 224 and ResNet-50 classifier. We compare the visual results attained by interpolation method us- ing both robust and non-robust classifiers of the same architecture. As can be seen in Figure 6 the interpolation results using robust classifier much more convincing and the intermediary images are perceptually plausible. In contrast, the interpolation using a non-robust model leads to much inferior results, and the final interpolation does not look like the target image all.
Image Interpolation:
7

---

Under review as a conference paper at ICLR 2022
Table 1: Conditional BIGRoC quantitative results on CIFAR-10, using FID (lower is better) and IS
(higher is better).
Architecture
FID 10K
FID 50K
IS 50K
cGAN Mirza & Osindero 2014)
33.28 + 0.31
29.26 + 0.10
6.95 E 0.03
Boosted cGAN
23.62 + 0.28
0.02 19.54
8.18 + 0.04
Improvement
29.03%
33. 33.22%
t 17.70%
cGAN-PD Miyato & Koyama Boosted cGAN-PD
2018)
15.01 + 0.14
11.03 E 0.08
8.36 E 0.05
12.99 + 0.11
8.89 + 0.05
8.57 + 0.05
Improvement
13.466
19.40%
2.51%
BigGAN (Brock al.
2019)
11.57 0.09
7.45 + 0.08
9.38 + 0.05
Boosted BigGAN
10.80 + 0.05
6.79 + 0.02
9.47 + 0.02
Improvement
+ 6.66%
4 8.86%
1 0.96%
Diff BigGAN (Zhao et al. 2020)
8.53 + 0.094
4.37 + 0.03
9.48 + 0.03
Boosted Dift BigGAN
8.03 + 0.043
3.95 + 0.02
9.61 + 0.03
Improvement
4. 12.45%
4.9.6%
1 1.37%
StyleGAN2 ADA (Krrra et al. 2020a)
6.60 + 0.04
2.44 0.01
10.02 E 0.04
Boosted StyleGANZ ADA Improvement
6.56 + 0.04
2.41 0.01
10.07 + 0.04
0.61%
1.23%
1 0.50%
Table 2: Unconditional BIGRoC quantitative results CIFAR-10, 10, using FID (lower is better) and IS (higher is better).
Architecture
FID 10K
FID 50K
IS 50K
VAE (Kingma & Welling Boosted VAE
2014)
155.05 0.55
152.04 + 0.19
3.05 0.01
90.93 + 0.60
88.68 + 0.37
6.27 0.04
Improvement
4. 41.36%
1. 42.99%
1 51.16%
DCGAN (Radford et al. 2015
42.44 + 0.06
38.34 + 0.11
6.10 + 0.01
Boosted DCGAN
33.66 + 0.26
0.05 29.93
7.28 0.04
Improvement
29.48%
4 21.9%%
1 19.34%
WGAN-GP (Salimans al. 2016)
26.59 = 0.09
22.62 = 0.09
7.49 = 0.03
Boosted WGAN-GP Improvement
20.16 + 0.22
16.28 + 0.08
8.15 + 0.03
24.18%%
4 28.03%
1 8.81%
SNGAN (Miyato et 2018)
21.33
0.38
17.19 0.07
8.04 0.02
Boosted SNGAN Improvement
17.09 + 0.20
13.25 + 0.10
8.61 + 0.04
19.88%
4. 1.22..%%
7.09%
InfoMaxGAN (Lee et al. 2021)
19.65 + 0.20
15.41 + 0.12
8.09 0.05
Boosted SSGAN
15.49 + 0.23
11.27 + 0.11
8.48 0.03
Improvement
21.15%
4. 26.82%
1 4.85%
SSGAN (Chen et al. 2019)
19.24 + 0.17
15.05 + 0.06
8.20 + 0.02
Boosted SSGAN
14.85 + 0.16
10.77 0.06
8.61 0.03
Improvement
22.82%
4. 28.44%
1 5.00%
DISCUSSION AND CONCLUSIONS
In this work, we propose a novel method that leverages the perceptually aligned gradients phe- nomenon for enhancing the visual quality of synthesized images. Our approach does not require additional training of the generative model and it is completely model agnostic. Thus, it can be applied during the inference phase of any model. Clearly, training robust classifier is a simpler task than training a generative model. Due to the core ability of such a robust classifier to better capture significant visual features, it capable of effectively and efficiently improving the output of generative models. In a line of experiments, we confirm this rationale and show that this method highly effective and capable of substantially enhancing both the qualitative and quantitative results of a wide range of generative models. addition, we develop an image interpolation method that stems from similar origin adversarial attacks robust classifier.
8

---

Under review as a conference paper at ICLR 2022
Unconditional
Conditional
WGAN-GP
SNGAN
SSGAN
cGAN
cGAN-PD
BigGAN
Airplane
Car
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Figure 5: The qualitative enhancement obtained by BIGRoC: We show 6 pairs of sets of images, each containing 10 images, one per CIFAR-I0 class. Each set consists images generated by certain GAN (left) and their BIGRoC boosted version (right). This demonstrates the significant visual enhancement achieved, both conditional and unconditional image generation schemes.
X,
Interpolate
X
Robust model
Non-Robust
model
Figure 6: Image interpolation demonstration on ImageNet images. The interpolation process is achieved by an alternation of according c c- || xall2, (0, 1].
REPRODUCIBILITY STATEMENT
In this work we have proposed post- procepsrcsssing algorithm for enhancing the visual quality of gen- erated images synthesized by generative models. We implemented approach using the official software or using implementations from Lee & Town (2020) each of the papers we have referred to, and using the same hyperparameters stated in their experiments. In addition, in all of our tests,

---

Under review as a conference paper at ICLR 2022
evaluated the performance multiple times using different seeds, to factor out randomness in the attained improvement. A code for running the conducted experiments for the unconditional gen- eration case is provided in the supplementary materials. A complete code package will be published upon acceptance of the paper.
REFERENCES
D. Bond-Taylor, A. Leach, Y. Long, and C. G. Willcocks. Deep generative modelling: A com- parative review VAEs, GANs, Normalizing Flows, Energy- Based and Autoregressive Models, 2021.
A. Brock, Donahue, and K. Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In 7th International Conference on Learning Representations, ICLR, 2019.
Chen, X. Zhai, M. Ritter, M. Lucic, and Houlsby. Self-Supervised GANs via Auxiliary Rota- tion Loss. In Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
J. Deng, W. Dong. R. Socher, Li-Jia Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, CVPR, 2009.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. In Advances Neural Information Processing Systems, NeurIPS, 2014.
I. Goodfellow, Shlens, and C. Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations, ICLR, 2015.
Kaiming H., Xiangyu Z., Shaoqing R., and Jian S. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition, CVPR, 2016.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge Local Nash Equilibrium. In Advances in Neural Informa- fion P Processing Systems, NeurlPS, 2017.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, NeurlPS, 2020.
T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training Generative Adversarial Networks with Limited Data. Advances in Neural Information Processing Systems, NeurIPS, 2020a.
T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and Improving the Image Quality of StyleGAN. In Conference on Computer Vision and Pattern Recognition, CVPR, 2020b.
D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, ICLR, 2014.
A. Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto Tech- nical Report, 2012.
K. Lee and C. Town. Mimicry: Towards the reproducibility of gan research. 2020.
K. S. Lee, Tran, and N. Cheung. Infomax-gan Improved adversarial image generation via information maximization and contrastive learning. In Winter Conference Applications of Computer Vision (WACV), 2021.
A. Madry, A. Makelov, L. Schmidt. D. Tsipras, and A. Vladu. Towards Deep Learning Models Re- sistant to Adversarial Attacks. International Conference on Learning Representations, ICLR, 2018.
M. Mirza and S. Osindero. Conditional Generative Adversarial Nets. CoRR, 2014.
10

---

Under review as a conference paper at ICLR 2022
T. Miyato and M. Koyama. cGANs with Projection Discriminator. In International Conference Learning Representations, ICLR, 2018.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral Normalization for Generative Adver- sarial Networks. In International Conference on Learning Representations, ICLR, 2018.
A. Van Oord, N. Kalchbrenner, and K. Kavukeuoglu. Pixel Recurrent Neural Networks. In Interna- tional Conference on Machine Learning, ICML, 2016.
A. Radford, L. Metz, and S. Chintala. Unsupervised Representation Learning with Deep Convolu- tional Generative Adversarial Networks. In International Conference on Learning Representa- fions, ICLR, 2015.
D. Rezende and S. Mohamed. Variational Inference with Normalizing Flows. In International Conference on Machine Learning, ICML, 2015.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved Tech- niques for Training GANs. In Advances in Neural Information Processing Systems, NeurIPS, 2016.
S. Santurkar, A. Ilyas, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Image Synthesis with a Single (Robust) Classifier. Advances in Neural Information Processing Systems, NeurIPS, 2019.
C. Szegedy, W. Zaremba, I. Sutskever, Bruna, D. Erhan, I. Goodfellow, and Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, ICLR, 2014.
D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness May Be at Odds with Accuracy. In International Conference on Learning Representations, ICLR, 2019.
R. Tumnrr, J. Hung, E. Frank, Y. Saatchi, and Yosinski. Metsopolis-Hastings generative adversarial networks. In International Conference on Machine Learning, ICML., 2019.
S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han. Differentiable augmentation for data-efficient GAN training, In Advances Neural Information Processing System, NeurIPS, 2020.
J.-Y. Zhu, Park, P. Isola, and A. A. Efros. Unpaired Image-to-Image Translation Using Cycle- Consistent Adversarial Networks. In International Conference Computer Vision, ICCV, 2017.

---

Under review as a conference paper at ICLR 2022
APPENDIX
.1 ADVERSARTAT. IMAGE GENERATION BOOSTING QUALITATIVE RESULTS
In this section, we show additional visual results that further demonstrate the qualitative enhance- ment attained by our proposed method. We use image generators of different qualities, both condi- tional and unconditional, and show the generated images and the boosted ones in Figures 7 and All the images shown below are simply the 100 first synthesized images from each class.
10
. ADVERSARIAL IMAGE GENERATION BOOSTING EXPERIMENTAL SETUP
We use an adversarially trained (Madry et al. 2018) ResNet-50 architecture with an bbaaeddd threat model for both image boosting and image interpolation, as Santurkar al. (2019). For image boosting, we use the same robust classifier to enhance all tested generative models, and we do not perform any hyperparameter tuning it. To best harness the perceptually aligned gradients, we use an & based threat model for the BIGRoC. Since different generative models require different magnitudes of cnhancement, we tune the hyperparameter enhancement each of them. The used values of e across different models are summarized in Table 3 Beyond the value e, we use the same number of steps (T=30) and set the step size a to be f.
Algorithm 2 to enable the proper
Table 3: The hyperparameter for BIGRoC for every tested generative model.
Architecture
cGAN
5
cGAN-PD
2
BigGAN
Diff BigGAN
StyleGAN2 ADA
0.28
VAE
DCGAN
5
WGAN-GP
5
SNGAN
SSGAN
3
.3 IMAGE INTERPOLATION ADDITIONAL RESULTS
We show Figure | additional image interpolation results, for both robust non-robust image
classifiers.
12

---

Under review as a conference paper at ICLR 2022
(a) comparison between BigGAN generated images class automobile and the corresponding boosted ones.
(b) A comparison between BigGAN generated images of class bird and the corresponding boosted
(c) A comparison between BigGAN generated images of class dog and the corresponding boosted ones.
Figure 7: qualitative comparison between BigGAN generated images of CIFAR-10 samples and the proposed BIGRoC algorithm.
13

---

Under review as a conference paper at ICLR 2022
Figure 8: qualitative comparison between an unconditional VAE generated images of CIFAR-10 10 samples and the proposed BIGRoC algorithm.
Figure 9: A qualitative comparison between unconditional WGAN-GP generated images of CIFAR-10 samples and the proposed BIGRoC algorithm.
Figure 10: A qualitative comparison between an unconditional SSGAN generated images of CIFAR- 10 samples and the proposed BIGRoC algorithm.
14

---

Under review as a conference paper at ICLR 2022
X,
Interpolate
X,
Robust model
Non-Robust model
Figure 11: Additional image interpolation results using robust and non-robust classifiers according our proposed method.
15


Reviews:
[{"review_id": "YZiIle1nMHJ", "reviewer": "Reviewer_v7Qv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nAdvantages:\n1. The improvement on the CIFAR-10 dataset is significant. And the refined images look much better than baselines.\n2. The model can work with both conditional/unconditional generative models. \n\nWeakness:\n1. In algorithm 1, it is not clear how to use epsilon. Please add more details.\n2. In section 2.2, the robust classifier method optimizes both the input images and the network, but in section 3, it only optimizes the input images. The definition is not aligned.\n3. Have the authors normalized the input images? Will the input pixel value between [0,1] or [0,255]? In the supplementary, the epsilon of VAE is 25. if the model uses a very large epsilon value, will the refined image still look realistic?\n4. This paper only shows the CIFAR-10 results. Will this method generalize to large-scale and high-resolution datasets, such as ImageNet? Currently, people are interested in high-resolution image generation. For example, BigGan can generate high-resolution images. Please the authors test this post-procession method on these generated high-res images.\n5. The projected gradient descent has already been used in some image generation works [1].\n6. The FID and IS scores of cGAN-PD and BigGAN in this draft are different from the original papers. Please add more details. Also, missing the definitions of '10K' and '50K'.\n\n\n\n[1] Xia, Weihao, et al. \"Gan inversion: A survey.\" arXiv preprint arXiv:2101.05278 (2021).", "summary_of_the_paper": "This paper presents a post-processing method to improve the GAN results. The post-processing consists of a projected gradient descent step to update the generated image to fit its target class, which works with both conditional/unconditional generative models. For the unconditional generative model, the authors design a de-bias method to force the model to uniformly generate images in each class. The results show this model can significantly improve the FID and IS scores on CIFAR-10 dataset. Additionally, the authors show this post-processing also works for image interpolation.\n", "summary_of_the_review": "Given the results and novelty are marginal, I think this paper is between boardline. Please the authors address my questions. I am happy to change my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O8Qh3D_Ff9_", "reviewer": "Reviewer_jEYy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method has some nice properties. It is model-agnostic and can be used to improve generative models without re-training. The only requirement is a robust classifier trained on the same dataset as the generative models. As stated in the paper, training a classifier is much easier than training a generator. Even better, the same classifier can help all generative models trained on the dataset. These characteristics are attractive in applications. \n\nThere are two weaknesses of the paper. First, the experiments are on the low-resolution CIFAR-10 dataset. Although the method is effective on the CIFAR-10 dataset, it is uncertain how well the proposed method performs on normal quality images. For 32x32 images, it is impossible to check out details. Also, low-resolution images are not very useful in real applications. As classifiers could focus more on high-level features, it is not clear how they help synthesize realistic details for high-resolution images. \n\nSecond, as stated in the paper, the method is similar to prior work, Santukar et al. 2019 and Turner et al. 2019. The paper lists three differences: (1) the proposed method builds upon the generated images; (2) the proposed method is model-agnostic; and (3) the proposed method can be performed on every image generated by the generative models without throwing out anyone. Although the technical novelty is not significant, the proposed method has advantages and could be useful in applications.\n", "summary_of_the_paper": "This paper proposes a model-agnostic method for improving the quality of images produced by generative models. The method requires a robust classifier trained on the same data source as the generative models. Based on the perceptually aligned gradients phenomenon, the proposed method improves the quality of a generated image by the targeted projected gradient descent method with the help of the robust classifier. Experiments on CIFAR-10 show that the proposed method does improve several generative models both quantitively and qualitatively.", "summary_of_the_review": "Overall, I like the proposed method because it is model-agnostic, simple, and effective. However, as stated in the previous section, since experiments are only performed on low-resolution images, it is not clear how the proposed method performs on images with ordinary resolutions and quality. Novelty is another potential issue, although I think that the paper has made sufficient contribution given its nice characteristics listed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1z0IOlg4Cl", "reviewer": "Reviewer_Ezr2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Writing]\n\nThe manuscript is well-written and easy to understand. \n\n[Novelty, Significance, and Prior Work]\n\nThe authors have explored an interesting idea of using robust classifiers for sample refinement. However, the key technical contribution is minor compared to Santurkar et al. [1] — it only involves using a robust classifier on pre-trained generative models. One might argue for the novelty of application, however, the authors have ignored several works that address the problem of sample refinement, e.g., using GAN discriminators [2, 3, 4]. In contrast to these works, BIGRoC also has the additional overhead of training a robust classifier. Moreover, these prior works bring in new technical perspectives to the problem of sample refinement (e.g., from optimal transport [2], energy-based modeling [3], gradient flows [4]) unlike the proposed method which hardly brings in any new technical insights.\n\n[Empirical Evaluation]\n\nIt is appreciable that the experiments have been conducted on several base models. However, only the CIFAR-10 dataset has been studied. It is unclear:\n\n- How the method will perform on other datasets?\n- How sensitive is the method to the training of the robust classifier?\n- What is the additional time overhead of training a robust classifier?\n\nFurthermore, the method has not been evaluated against any baseline methods (e.g., [2, 3, 4]). \n\n[1] Santurkar, Shibani, et al. \"Image synthesis with a single (robust) classifier.\" *arXiv preprint arXiv:1906.09453* (2019).\n\n[2] Tanaka, Akinori. \"Discriminator optimal transport.\" *arXiv preprint arXiv:1910.06832* (2019).\n\n[3] Che, Tong, et al. \"Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling.\" *arXiv preprint arXiv:2003.06060* (2020).\n\n[4] Ansari, Abdul Fatir, Ming Liang Ang, and Harold Soh. \"Refining deep generative models via discriminator gradient flow.\" *arXiv preprint arXiv:2012.00780* (2020).", "summary_of_the_paper": "The paper proposes BIGRoC (Boosting Image Generation via a Robust Classifier), a method to refine samples from a base generative model using a robust classifier. In this context, a robust classifier means one that has been trained to be robust to adversarial samples. Given a robust classifier and a base generative model (both trained on the same dataset), the key idea involves using the gradients of the robust classifier to update samples generated from the base generative model in the direction that maximizes the conditional probability of the samples' class. Experiments on the CIFAR-10 dataset have been presented that demonstrate that BIGRoC improves the sample quality from several base generative models in terms of the FID and IS metrics.", "summary_of_the_review": "The paper explores an interesting idea of using robust classifiers to refine samples from generative models; however, the technical contribution is limited. Several directly relevant methods of sample refinement have neither been discussed nor compared against. Overall, the contribution is not significant enough to warrant acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to improve generated images via a post-processing update procedure guided by gradients from a robust classifier.  After the author response and discussion, all reviewers agree that the paper is below the acceptance threshold.  Reviewer concerns include limited technical novelty and missing experimental comparison to relevant baselines.
Decision: Reject