BIGRoC: Boosting Image Generation via a Robust Classifier
Authors: Roy Ganz
Michael Elad
Venue: Accepted by TMLR ()

Abstract: The interest of the machine learning community in image synthesis has grown significantly in recent years, with the introduction of a wide range of deep generative models and means for training them. In this work, we propose a general model-agnostic technique for improving the image quality and the distribution fidelity of generated images obtained by any generative model. Our method, termed BIGRoC (Boosting Image Generation via a Robust Classifier), is based on a post-processing procedure via the guidance of a given robust classifier and without a need for additional training of the generative model. Given a synthesized image, we propose to update it through projected gradient steps over the robust classifier to refine its recognition. We demonstrate this post-processing algorithm on various image synthesis methods and show a significant quantitative and qualitative improvement on CIFAR-10 and ImageNet. Surprisingly, although BIGRoC is the first model agnostic among refinement approaches and requires much less information, it outperforms competitive methods. Specifically, BIGRoC improves the image synthesis best performing diffusion model on ImageNet $128\times128$ by 14.81%, attaining an FID score of 2.53 and on $256\times256$ by 7.87%, achieving an FID of 3.63. Moreover, we conduct an opinion survey, according to which humans significantly prefer our method's outputs.

Published in Transactions on Machine Learning Research (01/2023)
BIGRoC: Boosting Image Generation via Robust Classifier
Roy Ganz Electrical Engineering Deportment Technion
gans@caampss.tchhrionn....
Michael Elad
Computer Science Department
Technion
Reviewed on OpenReview:
Attps: // openreview. net/ met/foruue forum? id- 7RGNXhGSR
Abstract
The interest of the machine learning community in image synthesis has grown significantly in recent years, with the introduction of wide range of deep generative models and means for training them. In this work, we propose general model-agnostic technique for im- proving the image quality and the distribution fidelity of generated images obtained by any generative model. Our method, termed BIGRoC (Boosting Image Generation via a Ro- bust Classifier), is based on a post-processing procedure via the guidance of a given robust classifier and without a need for additional training of the generative model. Given a syn- thesized image, we propose to update through projected gradient steps over the robust classifier to refine its recognition. We demonstrate this post-processing algorithm on various image synthesis methods and show a significant quantitative and qualitative improvement on CIFAR-10 and ImageNet. Surprisingly, although BIGRoC is the first model agnostic among refinement approaches and requires much less information, it outperforms competi- live methods. Specifically, BIGRoC improves the image synthesis best, performing diffusion model on ImageNet 128 x 128 by 14.81%, attaining an FID score of 2.53 and on 256 x 256 by 7.87%, achieving an FID of 3.63. Moreover, we conduct an opinion survey, according to which humans significantly prefer our method's outputs.
1 Introduction
Deep generative models are class of deep neural networks trained to model complicated high-dimensional data (Bond-Taylor al.] 2021). Such models receive a large number of samples that follow a certain data distribution, z (x) and aim to produce new ones from the same statistics. One of the most fascinating generative tasks is image synthesis, which is notoriously hard due to the complexity of the natural images' manifold. Nevertheless, deep generative models for image synthesis have gained tremendous popularity in recent years, revolutionized the field, and becane state-of-the-art in various tasks (Isola et al.) 2017 Zhu et al. 2017 Karras et al.] 2018 2019 Brock et al.] 2019 Karras et al.) 2020). Energy-basee models, variational autoencoders (VAFs), generative adversarial networks (GANs), autoregressive likelihood models, normalization flows, diffusion-baseet algorithms, and more, all aim to synthesize natural-looking images, ranging from relatively simple to highly complicated generators (Kingma & Welling 2014; Goodfellow al.
2014 Rezende & Mohamed 2015 Oord et al. 2016 Ho et al. 2020).
When operating on a multiclass labeled dataset, as considered in this paper, image synthesis can be cither conditional or unconditional. In the unconditional selup, the generalive model aims to produce samples from the target data distribution without receiving any information regarding the target class of the synthesized images, i.c., sample from Pp(x). In contrast, the conditional case, the generator goal to synthesize images from target class, i.c., a sample from Pp(xly) where y is the label. As such, conditional generative models receive additional Mlass-relatdd information.

---

Published in Transactions on Machine Learning Research (01/2023)
Guided-Diffusion
BIGRoC
Figure 1: Qualitative demonstration of BIGRoC. Top: Images generated by guided diffusion trained on Image Net 128 x 128. Bottom: Refined images by applying BIGRoC.
Most of the work in the deep generative models' field focused on improving the quality and the variety of the images produced by such models, lackled by seeking novel architectures and training procedures. this work, while still aiming to improve the performance trained generative models, we place a different emphasis than in most of these studies and propose a method for refining the outputs of such models. More specifically, our model-agnostic method improves the perceptual quality of the images synthesized by any given model via iterative post-processing procedure driven by robust classifier.
With the introduction of learning baased based machines into wroal-world" applications, the interest in the robustness of such models has become a central concern. While there are abundant definitions for robustness, the most common and studied is the adversarial one. This definition upholds if a classifier is robust to a small perturbation of its input, made by an adversary to fool it. Previous work (Szegedy et al., 2014 Goodfellow et al. 2015 Kurakin et al. 2017) has demonstrated that deep neural networks are not robust all and can be easily fooled by an adversary. In light of this observation, many robustification methods were proposed, but the most popular among these is adversarial training (Goodfellow et al. 2015 Madry al. 2018 According to this method, to train robust classifier, one should generate adversarial examples and incorporate them into the training process. While examining the properties of such classifiers, researchers have revealed a fascinating phenomenon called perceptually aligned gradients (Tsipras et al.) 2019). This trait implics that modifying an image to sharpen such a classifier's decision yields visual features perceptuully aligned with the target class. In other words, when drifting an image content to be better classified, the changes obtained are visually pleasing and faithful natural image content.
In this work, harness and utilize the above-described phenomenon the images created by trained generative model so as to maximize the conditional probability of target class approximated by a given robust classifier. This modification can potentially improve the quality of the synthesized images since emphasizes visual features aligned with the target class, thus boosting the generation process both in terms of perceptual quality and distribution faithfulness. Due to the fundamental differences between classification and generation, we hypothesize that the robust classifier could capture different semantic features than the generative model. Thus, incorporating the two for image refinement can improve sample quality, Generation via a Robust Classifier.
propose to iteratively modify
be seen in Figure We term this method "BIGRoC" Boosting Image
Contrary to other refinement methods, BIGRoC is general and model-agnostic that can be applied to any image generator, both conditional or unconditional, without requiring access to its weights, given an ad- versarially trained classifier. The marked performance improvement achieved by our proposed method is demonstrated a series of experiments on a wide range of image generators on CIFAR-10 and ImageNet datasets. We show that this approach enables to significantly improve the quality of images synthesized
2

---

Published in Transactions on Machine Learning Research (01/2023)
Input
Targeted PGD
plane
bird
calt
deer
dog
frog
horse
ship
truck
Robust model
plane
bird
trog
ship
Non- robust
model
1-0.c-000
P-10,
P-10,2-42 P-10,
P=10, p=10
P-10, P-10,6-43
P-10,
Figure 2: PAG visualization. Demonstration of large 12-based adversarial examples robust (top) and non robust (bottom) ResNet50 classifiers (He et al. 2016)) trained on CIFAR-10, The classifier's certainty and the effective & norm the perturbation 6 are denoted as P and c, respectively. As can be seen, robust models with PAG guides the attack towards semantically meaningful features, whereas non-robust do not.
by relatively simple models, boosting them to level of more sophisticated and complex ones. Furthermore, show the ability of our method to enhance the performance of generative architectures of the highest quality, both quvell and quantitatively. Specifically, applying BIGRoC on the outputs of guided diffu- sion, (Dhariwal & Nichol 2021] significantly improves its performance on ImageNet 128 x 128 and 256 x 256 achieving FIDs of 2.53 and 3.63, an improvement of 14.81% and 7.87%, respectively. Moreover, to truly validate that BIGRoC perceptually refines samples, we present an opinion survey, which finds that human evaluators significantly prefer the outputs of our method. As such, our work exposes the striking generative capabilities of adversarially trained classifiers. These abilities were yet to be fully discovered and utilized in previous work. To summarize, our contributions are follow:
We introduce BIGRoC, the first model-agnoste approach for image generation refinement by harnessing the Perceptually Aligned Gradients phenomenon.
Extensive experiments on CIFAR-10 and ImageNet (both 128 x 128 and 256 256) show significant quantitative and qualitative improvements across a wide range of generative models.
We reveal the exceptional generative power of Adversarially Robust classifiers, the same model is capable of refining both low-quality and State-Of-The-AA Art image synthesizers.
2 Background
2.1
Adversarial Examples
Adversarial examples are instances intentionally designed by an attacker to cause a false prediction by a machine learning-based classifier (Szegedy et al. 2014 Goodfellow et al. 2015 Kurakin et al. 2017). The generation procedure of such examples relies on applying modifications to given training examples while restricting the allowed perturbations A. Tdeally, the "threat model" A should include all the possible unnoticeable perturbations
human observer. As it is impossible rigorously define such a set, in practice simple subset of the ideal threat model is used, where the most common choices the l2 and the e balls: instance and generates & = + ô s.t. 8 A.
A (0 |0012/00 < e}. Given A, the attacker receives while aiming to fool the classifier. Adversarial attacks can be both untargeted or targeted: An untargeted allack perturbs the input in a way that minimizes p(yl@) with respect to 8. In contrast, a largeted atlack receives in addition the target class ÿ, and perturbs r to maximize p.) There are diverse techniques for gencrating adversarial examples, yet. this work, we focus on targeted attacks using the Projected Gradient Descent (PGD) (Madry et al. [2018) an iterative method for creating adversarial examples that operates shown in Algorithm The operation II, is the projection operator onto A, and (()) is the classification loss.
3

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm Targeted Projected Gradient Descent (PGD)
Input: classifier fo, input 2, target class Ú, c, stop size a, number itcrations T
dg 0
for from
do di+1 = (6; a (V 66).0));
end
Tadv = x + dr
Output: Todv
2.2 Adversarial Robustness
Adversarial robustness is a property of classifiers, according to which applying a small perturbation on a classifier's 's input in order to fool it does not affect its prediction (Goodfellow et al. 2015). To attain such classifiers, one should solve the following optimization problem:
min Σ EyEDD
max {(Solx 8),
(1)
DEA
Namely, train the classifier accurately predict the class labels of the "toughest' perturbed images, allowed by the threat model A. In practice, solving this optimization problem challenging, and there are several ways to attain approximated solution. The most simple yet effective method is based approximating the solution of the inner-manee maximiaainiztioon via adversarial attacks, such as PGD (Madry et al. 2018). According to this strategy, the above optimization is performed iteratively, fixing the classifier's parameters and optimizing the perturbation 5 for each example via PGD, and then fixing these and updating 0. Repeating these steps results in a robust classifier, as we use in this work.
2.3 Perceptually Aligned Gradients
Perceptually aligned gradients (PAG) Tsipras et al. 2019 Kaur et al. 2019 Aggarwal et al, 2020 Ganz ct al. 2022) is a trait of adversarially trained models, best demonstrated when modifying an image to maximize the probability assigned to a target class. (Tsipras et al. 2019 show that performing the above process on such models yields meaningful features aligned with the target class. It important to note that this phenomenon does not occur in non-robust models. The perceptually aligned gradients property indicates that the features learned by robust models are more aligned with human perception. Figure 2 presents a visual demonstration of this fascinating phenomenon.
3 Boosting Image Generation via Robust Classifier
We propose a method for improving the quality of images synthesized by trained generative models, named BIGRoC: Boosting Image Generation via Robust Classifier. Our method is model agnostic and does not require additional training or fine-tuning of the generative model that can be viewed as post-processing step performed on the synthesized images. Thus, BIGRoC can be easily applied to any generative model, both conditional or unconditional. This mechanism harnesses the perceptually aligned gradients phenomenon to modify the generated images to improve their visual quality. To so, we perform an iterative process of modifying the generated image r to maximize the posterior probability of a given target class ÿ. po())z), where pe modeled by an adversarially trained classifier. This can be achieved by performing a PGD-like process that instead of adversarially changing an image of class y to different class # y, modifies it in way that maximizes the probability that belongs to class Therefore, our method requires a trained robust classifier that operates on the same data source as the images we aim to improve.
In the conditional generation process, the generator G receives the class label y, from which suppose to draw samples. Hence, in this sctup, we have information regarding the class affiliation of the image, and we can maximize the corresponding conditional probability. In the unconditional generation process, the generator does not receive class labels at all, and its goal is draw samples from p(x). Thus, in this case,
4

---

Published in Transactions on Machine Learning Research (01/2023)
Unconditional
Conditional
WGAN-GP
SNGAN
SSGAN
cGAN
cGAN-PD
BigGAN
Airplane
InfoMaxGAN
Car
Bird
Cat
SNGAN
Deer
Dog
Frog
BigGAN SSGAN
Horse
Ship
Truck
(a) CIFAR-10
(b) ImageNet
Figure 3: BIGRoC qualitative results. We present pairs of images where the left columns contain
generated images and the right ones the boosted results. Figures 3(a) and 3(b) show such results on CIFAR- 10 and ImageNet 128 x 128, respectively.
cannot directly maximize the desired posterior probability, our method suggests. To bridge this gap, propose lo estimate the most likely class via our robust classifier fo and afterward modify the image via the suggested method to maximize its probability. The proposed image generation boosting is described in Algorithm 2 for both setups.
While the above-described approach for unconditional sampling works well, it could be further improved. We have noticed that this case, estimating the target classes Y of X Xgen via fo might lead to unbalancod labels. For example, in the CIFAR-10, when generating 50,000 samples, we expect approxinately 5,000 images per each of the ten classes, and yet the labels' estimation does not distribute uniformly at all. This imbalance stems from the incompetence of the generative model to capture the data distribution, which leads to bias in the target labels estimation of the boosting algorithm, affecting the visual content of Xboost- Since quantitative evaluation is impaired by such class imbalances, this bias limits the quantilative improvement attained by BIGRoC. We emphasize that this issue is manifested only in the quantitative metrics, and when qualitatively evaluating the boosted images, the improvement significant, as can be seen in the supplementary material.
To further enhance the quantitative results of our algorithm in the unconditional case, propose to de-bias the target class ostimation of X Xgen and attain close to uniform class estimations. A naive solution to this can be achieved by generating more samples and extracting a subset of these images with a belance This approach is computationally heavy and does not use the generated images as-is, raising questions regarding the fairness of the quantitative comparison. Thus, we propose a different debiasing technique we modify the classifier's class estimation to become more balanced by calibrating its logits. More specifically, we shift the classifier's logits by adding per-class pre-caleulated value, de, that induces equality of the mean logits value across all classes. We define de as vector containing all da, values: de the number of classes. For simplicity, denote logile, as the logit of class corresponding to generated sample Zgen- We approximate [logitt) for each class C₁, using a validation set of generated images, and calculate a per-class debiasing factor: de, a Ezye, [logite,] (WLOG, 1), where Æz,., [logite,] is a mean estimator. After caleulating de,; given generated image Tgen; calculate its logits and add de; to
-ex--;] where N is
5

---

Published in Transactions on Machine Learning Research (01/2023)
Generated Images
Lo based boost
L2-based boost
Figure 4: Threat-model effect demonstration. Generated images via a conditional GAN (left) perturbed via a targeted PGD maximize the probability of their target classes (planes, cars and birds) using either foc l2 threat models. While the boosted outputs attained by the foo entirely change the structure of the images and lead to unnatural results, the l threat model leads to better results.
it to obtain debiased logits (logita)) from which derive the unbiased class estimation via softmax. The following equation shows that, given correct estimation of the per-class logits' mean, the per-class means of the debiased logits are equal:
Exyen [logite]] = Errgen e [de; + logite;] = Exyen [a Brgen [ogite,]] + logite,]
Exg.. [logila,] Exyen [ogite; Rs a
We present in Appendix an empirical demonstration that verifies the validity of this method and shows its qualitative effects on unconditional generation boosting.
Algorithm 2 BIGRoC
Input: Robust classifier Jes Egens Mgens e, step size a, number iterations T, debiasing vector de
if Ygen is None then
Vgen = arggaa((ooFgggm + de)
end
Thoost = T Tergeted PGD(Jo, & gen Mgens T))
Output: Tboost
As can be seen in Algorithm 2 it receives as input the generated images and their designated labels (if exist) and returns an improved version of them. As such, this method can be applied at the inference phase of generative models to enhance their performance in a complete soparation from their training. Furthermore. BIGRoC does not require access to the generative models at all and thus can be applied to standalone images, regardless of their origin. As can be seen from the algorithm's 's description, that determine the modification process of the image: sets the maximal size of the perturbation allowed by the threat model A, controls the step size at each update step, and T is the number of updates. Another choice is the norm used to define the threat model A.
has several hyperparameters
The hyperparameter e is central in our scheme when e is too large, the method overrides the input and modifies the original content in an unrecognizable way, as can be seen Figure 2 On the other hand, when too small, the boosted images remain very similar the input ones, leading to minor enhancement. As our goal is obtain a significant enhancement to the synthesized images, careful choice of e should be practiced, which restricts the allowed perturbations in the threat model.
Another important choice is the threat model A itself. Two of the most common choices of A for adversarial attacks are the foo and the & balls. Due to the desired behavior of our method, using the to ball is loss preferable: it allows a change of te to every pixel, and as such, it will not focus on meaningful specific locations, and might not preserve the existing structure the synthesized input image. Thus, we choose the l ball as our thrcat model, with relatively small c. Such a choice restricts the allowed perturbations and leads changes that may concentrate on specific locations while preserving most of the existing content in the generated images. A visual demonstration of these considerations is given in Figure

---

Published in Transactions on Machine Learning Research (01/2023)
4 Related Work
4.1 Image Generation Refinement
There are two main lines of work that aim to improve the quality of generated images. One is based on rejection sampling improving the generation quality of GANs by discarding low-quality images identified by the GAN's discriminator (Turner et al., 2019; Azadi et al., 2019). Contrary to such work that does not enhance the generated images but rather acts as selector, BIGRoC does not discard any synthesized images and improves their quality by modifying them.
The second line of work (Tanaka 2019 Che et al. 2021 Ansari et al. 2021), which is closely related to ours, addresses the task of sample refinement - modifying the generated images to attain improved perceptual quality. These papers propose methods for improving synthesized images using the guidance of the GAN's discriminator. More preciscly, given a latent code of a generated image, their strategy is to modify it maximize the score given by the GAN's discriminator. Therefore, to enhance the perceptual quality of a set of generated images, these approaches require access to both the generator and the discriminator weights and the corresponding latent code of the generated images. Thus, such methods rely on the existence of a discriminator, making them not model-agnostic. Moreover, as image refinement. is an applicative task, this constraint prevents such methods from operating directly on images without additional information, making their configuration less realistic. In contrast to these, our work offers a much simpler and different way of boosting generated images by an external pretrained robust classifier in a completely model-agnostic way. Our method can operate without requiring access to the latent vectors generating the images or the weights of the generative model that produces them. Thus, BIGRoC can be applied to standalone images realistic setup where none of the existing methods can operate. Moreover, while competitive methods couple the generator with its discriminator, we show that a single robust classifier is capable of improving a wide variety of sample qualities from the same dataset, making our configuration even more realistically appealing, as it requires only one model per dataset. In addition, while (Tanaka) 2019 Che et al. 2021 are limited to GANs only, our method is model-agnostic and capable of improving generated images of any source, e.g., diffusion models. In Section5..4 we empirically demonstrate that although the existing methods have much stricter requirements than ours, it leads to improved performance. Morcover, approach is the first to be successfully applied to ImageNet. 256 x 256, proving its scalability.
4.2 Perceptually Aligned Gradients in Computer Vision
PAG phenomenon was previously utilized for solving various computer vision tasks, such inpainting, image translation, super-resolution, and image generalion (Santurkar et al. 2019). However, in image synthesis, the obtained performance is relatively disappointing, d.e., Inception Score (IS) of 7.5 on CIFAR-10, far from state-of-tho-ae (SOTA) performance. Morcover, the qualitative rcsults demonstrated are far from pleasing. Thus, it raises the question if this is the generative capabilities performance limit of robust image classifiers. In our work, we provide a definite answer to this question by harnessing PAG to a different task sample refinement. To this end, we build upon any existing generative model, including high-performing ones, and empirically show that robust classifier can boost the performance of SOTA image generators. In particular, we demonstrate that robust classifier guidance can improve Diff BigGAN (Zhao el al., 2020) from IS of 9.48 to 9.61, well beyond the 7.5 obtained in (Santurkar et al., 2019). As such, our work exposes much stronger force that does exist in adversarially robust classifiers in capturing high perceptual quality features.
5 Experimental Results
In this section, we present experiments that demonstrate the effectiveness of our method on the most common datasets for image synthesis CIFAR-10 (Krizhevsky) 2012) and ImageNet. (Deng et al., 2009). Given a generative model, we use it to synthesize a set of images Xgen and apply our method to generate Xboost: according to Algorithm | Since BIGRoC is model-agnostic, it can be casily applied to any generative model, given robust classifier trained on the same data source. We utilize the model-agnostic property examine the effects of applying the proposed boosting over a wide variety of image generators of different
7

---

Published in Transactions on Machine Learning Research (01/2023)
qualities: from relatively simple to sophisticated and complex ones. We test our method on both conditional and unconditional generative models to validate that the proposed scheme can enhance different synthesis procedures. An application of our method without the ground truth labels is termed BIGRoCPL, as it generates pseudo labels (PL) using our robust classifier. In contrast, BIGRoCer refers to the case where ground truth (GT) labels are available (in the conditional image synthesis).
In all the experiments, for each dataset, we use the same adversarial robust classifier to refine all the generated images of different sources. The only needed adjustment is at tuning e, which defines the allowed size of the visual modifications done by BIGRoC. The striking fact that single robust classifier improves both low and high-quality images strongly demonstrates the versatility of our approach and the surprising refinemennt capabilities possessed by such a model. We analyze BIGRoC performance both quantitatively and qualitatively, using Fréchet Inception Distance (FID, (Heusel et al. 2017], lower is better), and Inception Score (TS, (Salimans et al., 2016), higher is better). Morcover, we conduct an opinion survey to validate that human observers find BIGRoCs outputs more pleasing. In addition, we compare our approach with other image refinement SOTA methods mentioned in Section
5.1 CIFAR-10
In this section, we evaluate the performance of the proposed BIGRoC on the CIFAR-10 dataset, using a single publicl--vaillale adversarially trained ResNet-50 on CIFAR-10 as the robust classifier (Engstrom ct al. 2019 We experiment with wide range of generative models, both conditional and unconditional.
Quantitative Results Table D contains BIGRoCrI. and BIGRoCGT quantitative results for both uncondi- tional and conditional image synthesis refinement, respectively. These results indicate that BIGRoC achieves a substantial improvement across a wide range of generator architectures, both conditional and unconditional, demonstrating the method's versatility and validity. Interestingly, the fact that the same robust classifier can enhance both low-quality models (that require focus on high-level features) and top-performing models (that require emphasis on low-level features) strongly attests to its generative powers.
Qualitative Results To strengthen the quantitative results, show in Figure 3(a) and in the supplemen- tary material qualitative results that verify that the "boosted" results indeed look better perceptually.
5.2 ImageNet
We turn to evaluate the performance of the proposed BIGRoC on the ImageNet 128 x 128 and 256 x 256 datasets, using a single publicly available adversarially trained ResNet-50 on ImageNet as the robust classifier (Engstrom el al. 2019 for both of the resolutions.
Geneeattd Images
Boosted Imagres
Quantitative Results Table 2 summarize our quantita- tive results on ImageNet 128 x 128 and 256 256. These results strongly indicate that BIGRoC is also highly ben- eficial for higher-resolution images from richer datasets. Specifically, BIGRoC improves the best-performing dif- fusion model (Dhariwal & Nichol, 2021) on ImageNet 128 x 128 by 14.81%, leading FID of 2.53, and on 256 x 256 by 7.87%, leading to FID of 3.63. The ob- tained results show a similarly marked improvement in IS. Interestingly, our approach (BIGRoCpL) is capable of refining the outputs of top-performing generative models even with the absence of the ground truth labels.
Daference
Figure 5: Qualitative comparison on Ima- geNet 256 256. Top: Images generated by guided diffusion. Middle: Images boosted by BI- GRoC. Bottom: contrast stretched differences.
Qualitative Results To verify that our method indeed leads to better perceptual quality, we show visual results on ImageNet 128 x 128 in Figure 3(b) and Figure Morcover, we present in Figure a qualitative comparison between images generated by guided-diffusion trained on ImageNet 256 x 256 and the outputs of BIGRoC applied to them. We also show the images' differences (after contrast stretching) to better grasp
8

---

Published in Transactions on Machine Learning Research (01/2023)
Table 1: BIGRoC quantitative results on CIFAR-10 for both conditional and unconditional generators.
Architecture
Cond.
FID
IS
VAE (Kingma & Welling 2014
152.04 0.19
3.05 = 0.01
w/ BIGRoCPL
88.68 0.37
6.27 0.04
A
-63.36
+3.22
DCGAN (Radford al. 2015
38.34 0.11
6.10 = 0.01
w/ BIGRoCPL
29.93 + 0.05
7.28 = 0.04
A
-8.41
+1.18
WGAN-GP (Sllmaan et al.
2016
X
22.62 + 0.09
7.49 0.03
w/ BIGRoCPL
16.28 + 0.08
8.15 + 0.03
A
-6.32
+0.66
SNGAN (Miyato et al. 2018
x
17.19 + 0.07
8.04 + 0.02
w/ BIGRoCPL
13.25 + 0.10
8.61 = 0.04
A
-3.94
+0.57
IAAA (Lee et al. 2021
15.41 0.12
8.09 0.05
w/ BIGRoCPL
11.27 0.11
8.48 0.03
A
-4.14
+0.39
SSGAN (Chen al. 2019
X
15.05 + 0.06
8.20 + 0.02
w/ BIGRROPL.
10.77 0.06
8.61 + 0.03
-4.28
+0.41
cGAN Mirza & Osindero 2014]
29.26 + 0.10
6.95 Σ 0.03
w/ BIGRoCeT
8.89 + 0.05
8.57 + 0.05
-20.37
+1.62
cGAN-PD Mirza & Osindero 2014)
11.10 + 0.07
8.54 + 0.03
w/ BIGRoCeT
8.33 = 0.06
8.76 0.05
A
-2.77
+0.22
BigGAN (Brock al. 2019
7.45 + 0.08
9.38 + 0.05
w/ BIGRoCeT
6.79 = 0.02
9.47 = 0.02
A
-0.66
+0.09
Diff BigGAN ZZaao al. 2020
v
4.37 + 0.03
9.48 + 0.03
w/ BIGRoCeT
3.95 + 0.02
9.61 + 0.03
A
-0.42
+0.13
the perceptual modifications applied by our method. As can be seen, BIGRoC focuses on the edges and textures and leads to sharper and more high-contrast images, which are more pleasing to human observer.
5.3 Opinion survey with human evaluators
As current quantitative metrics are limited proxies for human perception, we conduct an opinion survey with human evaluators to better validate that our proposed method leads to better visual quality. We randomly sampled 100 images from a guided diffusion, trained on ImageNet 128 128, and applied BIGRoC on them. Human evaluators were shown pairs of such images in a random order, and the question: "Which image looks better?" They were asked to choose option between "Left", "Right", and "Same". The image pairs were in random order for each evaluator. We collected 1038 individual answers in total, where 56.07% of the votes opted for BIGRoC's outputs, 19.17% the generated ones, and the rest chose the "same" option. When aggregating the voles per each image pair, in 83% of the pairs, human evaluators were in favor of our algorithm's 's outputs, while only in 12% of them, they prefer the generated ones. In the remaining 5%, the "same" " option obtained the most votes. These results indicate that human evaluators considerably prefer our method outputs, with significance level of 95%. We provide additional details regarding the conducted survey in Appendix B

---

Published in Transactions on Machine Learning Research (01/2023)
Table 2: Quantitative results on ImageNet 128
128.
Architecture
Resolution
FID
IS
SNGAN (Miyato et al. 2018 w/ BIGRoCPL
128 x 128
62.28
13.05
40.40
71.67
A
-21.88
+58.62
SSGAN (Chen al. 2019
128 128
63.60
13.75
w/ BIGRoCPL
38.93
73.94
A
-24.67
+60.19
InfoMaxGAN Lee et al. 2021
128 x 128
60.61
13.79
w/ BIGRoCPL
37.70
75.49
-22.91
+61.7
BigGAN- Brock et al. 2019
128 x 128
6.02
145.83
w/ BIGRoCPL
5.69
176.42
w/ BIGRoCeT
5.71
226.17
-0.31
+80.34
Guided Diffusion (Dhariwal & Nichol 2021 b
128 x 128
2.97
141.37
w/ BIGRoCpt.
2.77
150.43
w/ BIGRoCeT
2.53
169.73
A
-0.44
+28.36
BigGAN- deep Brock et al. 2019)
256 x 256
7.03
202.65
w/ BIGRoCPL
6.93
221.78
w/ BIGRoCeT
6.84
228.23
-0.19
+25.58
Guided Diffusion DDhariwal & Nichol 2021
256 x 256
3.94
215.84
w/ BIGRoOpL.
3.69
249.91
w/ BIGRoCeT
3.63
260.02
A
-0.31
+44.18
5.4 Comparison with other methods
Previous works, such as DOT (Tanaka 2019 DDLS (Che et al. 2021 h and DG-flow (Ansari et al. 2021 address image generation refinement, as we do. As mentioned in Section we propose a much simpler approach than our competitors and require much less information since BIGRoC can operate without access to the image generator, the discriminator, and without knowing the latent codes corresponding with the gen- erated images. Moreover, BIGRoC is the first model-agnostic refinement method that operate any generative model's output. Besides these advantages, we demonstrate that although our method utilizes less information than other methods and can operate in setups that competitive approaches can not, BIGRoC performs on par and even better. To this end, we compare our method with current SOTA image refinement methods on CIFAR-10 and ImageNet 128 x 128. For both these setups, we adopt the same publicly available pretrained models of EN-ResNet-GAN apply algorithm to the generated images and evaluate its performance quantitatively (see the supple- mentary material for additional implementation details). In Table 3 compare the quantitative results of the DOT, DDLS, and DG-flow with BIGRoC using IS, as this is the common metric reported in these papers. Our findings show that our approach significantly outperforms the other methods on the challenging ImageNet dataset while obtaining comparable results to DG-flow on CIFAR-10.
used in (Tanaka 2019 Che al. 2022 Ansari et al. 2021] and
6 Ablation Study
We conduct experiments better understand the performance improvements obtained and analyze the effects of the central hyperparameters in algorithm.
10

---

Published in Transactions on Machine Learning Research (01/2023)
Table 3: Quantitative comparison with competitive methods.
INCEPTION SCORE CIFAR-10 IMAGENET
ARCHITECTURE
SN-RESNET-GAN
8.38
36.8
W/ DOT
37.29
W/ DDLS
9.09
40.2
W/ DG-FLOW
9.35
W/ BIGROC
9.33
44.68
6.1
The effect of
As stated Section 3 the value of e defines the al- lowed perturbation. In Figure G we demonstrate the effect of different values of when applying BIGRoCoT to images generated by guided diffusion, trained ImageNet 256 x 256. As can be seen, it affects the trade-off between diversity and fidelity, demonstrated by IS versus FID values. Our method attains lower FTD scores in a range of tested values of while achieving better 1S, leading to better trade- ofls. We present additional ablations in the supple- mentary materials. This study shows that although its importance, BIGRoC is not highly sensitive to the choice of e.
e
BIORuC
c-3
Guided Dffuution
1=1
e
200
250
300
350
IS
Figure 6: Trade-off effects of e. Demonstrated on GuideddDiiffusion trained on ImageNet 256 256.
6.2
The effect of the guiding classifier
In this work, propose a method for perceptually improving generated images by using robust classifier. However, there are few other choices of models that can be used for guiding the refinement process. The simplest choice is using a standard non-robust classifier trained on the same dataset as the generative model. Another applicable option when refining the outputs of guided diffusion models is to use the inherent guiding classifier the diffusion. To validate that indeed robust classifiers have the best. refining capabilities, we apply Algorithm on 50,000 ImageNet 128 128 images generated by guided diffusson (Dhariwal & Nichol 2021 using 5e ("Vanilla clf"), a guiding classifier ("Guiding clf") proposed in (Dhariwal & Nichol
2021 and robust classifier ("RoC"). We summarize our positive findings in Table which show that robust classifier provides the best guidance among the tested alternatives.
Table FID scores of Algorithm using different classifiers.
w/ ALG.
w/o ALG.
VANILLA CLF
GUIDING
RoC
2.97
3.19
2.77
2.53
6.3
The effect of the threat model
In all of our experiments, we use an adversarially trained (AT) robust classifier with threat model A based on la norm with a predefined value. In this section, we study the effect of in the training of the AT robust classifier on BIGRoC's performance. In Table we compare the influence of using a non-adversarial classifier (i.c. < 0) and adversarial classifiers trained with different threat models' sizes on our proposod method, while the rest of the hyperparameters are fixed. As can be seen, using AT classifiers in BIGRoC enhances the results significantly.
11

---

Published in Transactions on Machine Learning Research (01/2023)
Table 5: Illustration of the robust classifier's threat model effect on BIGRoC's performance, measured in FID, using WGAN-GP trained on CIFAR-10 dataset.
w/ BIGRoC
w/o BIGROC
0
0.25
0.5
22.32
19.71
18.79
16.28
15.87
7 Discussion and Conclusions
In this work, we propose novel method that leverages the perceptually aligned gradients phenomenon for refining synthesized images. Due to the core ability of such robust classifier to better capture significant visual features, it is capable of effectively improving the output of generative models. Our approach does not require additional training of the generative model, and it is completely model agnostic, contrary lo highly effective and capable of substantially enhancing the qualitative and quantitative results of various generative models over multiple datasets. Moreover, we conduct an opinion survey that validates that the outputs of BIGRoC indeed more perceptually pleasing- Interestingly, work reveals the surprising generative capabilities of robust classifiers, single such model can refine the outputs of both simple and SOTA generators.
other methods. In a line of experiments, we show that our method
References
Gunjan Aggarwal, Abhishek Sinha, Nupur Kumari, and Mayank Kumar Singh. On the benefits of models with perceptually--ligneed gradients. ArXiv, Abs/2005.01449) 2020.
A. Fatir Ansari, M. Liang Ang, and H. Soh. Refining deep generative models via discriminator gradient flow, 2021.
S. Azadi, C. Olsson, T. Darrell, I. Goodfellow, and A. Odena. Discriminator rejection sampling, 2019.
Bond-Taylor, A. Leach, Long, and C. G. Willcocks. Deep generalive modelling: comparative review of VAEs, GANs, Normalizing Flows, Energy- Based and Autoregressive Models, 2021.
A. Brock, J. Donahue, and K. Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In 7th International Conference on Learning Representations, ICLR, 2019.
T. Che, R. Zhang, J. Sohl-Dickstein, H. Larochelle, T.. Paull, Y. Cao, and Y. Bengio. Your gan is secretly energy-based model and you should use discriminator driven latent sampling, 2021.
T. Chen, X. Zhai, M. Ritter, M. Lucic, and N. Houlsby. Sele-Supe GANs via Auxiliary Rotation Loss. In Conference on Computer Vision and Pallern Recognition, CVPR, 2019.
J. Deng, W. Dong, R. Socher, Li-Jia Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, CVPR, pp. 248-255, 2009.
P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. orXiv preprint arXiv:2105.0525 2021.
Logan Engstrom, Andrew Nlas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL http:p///gggitub...o///aaatt////
Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply adversarial robustness?, 2022. URL
11378
Goodfellow, J. Pouget-Abadic, M. Mirza, Xu, D. Wardc-Farley, Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. In Advonces in Neural Information Processing Systems, NewrIPS, pp. 2672- 2680, 2014.
12

---

Published in Transactions on Machine Learning Research (01/2023)
I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learving Representations, ICLR, 2015.
K. He, X. Zhang, S. Ren, and Sun. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition, CVPR, pp. 770-778, 2016.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time- Scale Update Rule Converge Local Nash Equilibrium. In Aduances in Neural Information Processing Systems, NewrIPS, 2017.
J. Ho, A. Jain, and P. Abbecl. Denoising diffusion probabilistic modols. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexci A. Efros. Image-to-image translation with conditional adversarial works In Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 5967-5976, 2017.
T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, ICLR, 2018.
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Conference Computer Vision and Pattern Recognition, CVPR, pp. 4401 4410, 2019.
T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and Improving the Image Quality of StyleGAN. In Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
Simran Kaur, Jeremy Cohen, and Zachary Lipton. Are perceptually-aignned gradients a general property of robust classifiers?, 10 2019.
D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, ICLR, 2014.
A. Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto Technicol Report, 2012.
A. Kurakin, Goodfellow, and S. Bengio. Adversarial examples in the physical world, 2017.
K. S. Lee, N. T. Tran, and N. M. Cheung. Infomax-gan: Improved adversarial image generation via infor- mation maximization and contrastive learning. In Winter Conference on Applications of Computer Vision (WACV), 2021.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations, ICLR, 2018.
M. Mirza and S. Osindero. Conditional Generative Adversarial Nets. CoRR, 2014.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral Normalization for Generative Adversarial Networks. In International Conference on Learning Representations, ICLR, 2018.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7176- 7185. PMLR, 2020.
A. Van Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel Recurrent Neural Networks. In International Conference on Machine Learning, ICML, pp. 1747 1756, 2016.
A. Radford, L. Metz, and S. Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In International Conference Learning Representations, ICLR, 2015.
D. Rezende and S. Mohamed. Variational Inference with Normalizing Flows. In International Conference on Machine Learning, TCML, pp. 1530 -1538,
1538, 2015.
13

---

Published in Transactions on Machine Learning Research (01/2023)
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved Techniques for Training GANs. In Advances in Neural Information Processing Systems, NewrIPS, 2016.
S. Santurkar, A. Ilyas, D. Tsipras, L. Engstrom, Tran, and A. Madry. Image Synthesis with Single (Robust) Classifier. In Advances in Neural Information Processing Systems, NewrIPS, 2019.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties neural networks. In International Conference on Learning Representations, ICLR, 2014.
A. Tanaka. Discriminator optimal transport. In Advances in Neural Information Processing Systems, NewrIPS, volume 32, 2019.
D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, Robustness May Be at Odds with Accuracy. In International Conference on Learning Representations, ICLR, 2019.
R. Turner, J. Hung, Frank, Y. Saatchi, and J. Yosinski. Metropolis- Hatropoli-HHastinss generative adversarial networks. In International Conference on Machine Learning, ICML, 2019.
S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han. Differentiable augmentation for data-efficient GAN training. Advances in Neural Information Processing System, NeurIPS, 2020.
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired Image-to-lmage Translation Using Cycle-Consistent Adversarial Networks. In International Conference Computer Vision, 1CCV, 2017.
14

---

Published in Transactions on Machine Learning Research (01/2023)
A Additional Ablations and Demonstrations
A.1 The effect of BIGRoC's number of steps
When é is fixed, increasing the number of steps leads to smaller steps. Fine-grained steps can lead to better performance, but with a computational cost. In Table G we summarize the effect of the number steps w.r.t fixed 6, As can be concluded, is a plausible value since it good trade-off between refinement performance and computational cost.
Table 6: The effect of the number of steps in BIGRoC's algorithm, measured in FID, using WGAN-GP trained CIFAR-10 dataset.
w/ BIGRoC
w/o BIGROC
1 STEP
STEPS
20 STEPS
30 STEPS
22.32
17.48
16.28
16.15
16.27
A.2 Visual demonstration of BIGRoC's iterations
BIGRoC is an iterative boosting algorithm, and as such, performs several update steps. In Figure 7 we visualize the optimization algorithm performed by our method. As can be seen, the perceptuul quality the images obtained by BIGRoC gradually improves during its application. More specifically, first, BIGRoC focuses the coarse shape and details of the image and then the fine details.
steps
Airplane
Car
Bird
Cat
Dear
Dog
Frog
Horse
Ship
Truck
Figure 7: BIGRoC's steps visualization, demonstrated using VAE trained CIFAR-10: On the left column we show the generated images of the model. The righmost column corresponds with the final output of BIGRoC. The middle 9 columns are the results obtained after each intermediate step of our algorithm.
15

---

Published in Transactions on Machine Learning Research (01/2023)
A.3 Qualitative comparison with non-robust models
As explained in the experimental part of the paper, study the effects of applying BIGRoC with non-robust classifiers. More specifically, we do so with non-robust ResNet-50. In Figure g we show that applying our algorithm with such non-robust network does not refine the image as a robust network that has PAG docs. We use exactly the same hyperparameters these experiments.
Generated Images
BIGRoC wi non-robust model
BIGRoC wi Robust model
Figure & Guiding classifier effect. Qualitative comparison between the refinement obtained by BIGRoC with robust and non-robust models guidance.
B
Opinion Survey Details
Which image looks better?
Left
Right
Same
Figure 9: Opinion survey. An example of question from the human-observer survey.
As mentioned in the paper, conduct opinion survey in which ask human evaluators to choose their preferred image from a pair of two a generated one and BIGRoC's output. In addition, the evaluators were allowed to choose that they have no preference. We randomly generate 100 images using guided- diffusion (Dhariwal & Nichol 2021 trained on ImageNet 128 x 128 and applied BIGRoC on them. We create pairs of original-boosted images in random order and shuffled the order of them for each evaluator. An example of the screen shown evaluators is displayed in Figure We provide all 100 pairs in the supplementary material.
16

---

Published in Transactions on Machine Learning Research (01/2023)
C Implementation Details
In this work, except for three basic generative models (see description in C.1 we did not train models and used only available pretrained ones from verified sources. For quantitative evaluation, use common implementations of IS and FID metrics. In all of our experiments, we use the same robust classifier to boost all the gencrative models that operate on the same dataset. We use the pretrained generators to synthesize sets of images and BIGRoC to refine them. The specific details regarding the experimental results are listed in the sections below.
C.1 CIFAR-10
Adversarial Robust Classifier We use a pretrained robust ResNet--0 on CIFAR-10, provided in (En- gstrom et al. 2019). This model is adversarially trained with threat model A {6 || S 0.5) with step size of 0.1.
Image Generators Besides VAE, DCGAN, and cGAN that we trained from scratch, using the relevant available codebases, we did not train any other generator and used only publicly available ones. For cGAN- PD. WGAN-GP, SNGAN, InfoMaxGAN, and SSGAN, we use the ones from mimicry repository We use the pretrained versions of BigGAN and Differential Augmentation CR BigGAN (Diff BigGAN) from the data-efficient GANs repository
BIGRoC hyperparameters As stated above, we use the same robust classifier in all our experiments. Thus, the remaining hyperparameters to tune step size, and the number of steps. We empirically find out that the number of steps is relatively marginal (see Section A.1 and thus we opt to use 7. In all the experiments, we fix the step size to be 1.5 qualities, each requires a particular amount of visual enhancement, defined by the value of E. Low-quality generators require substantial improvement and therefore benefit from high values, while better ones benefit from smaller values. In the below table, we summarize the value of e for each tested architecture.
steps" Since we test various image generators of different
Table 7: BIGRoC's values in CIFAR-10 experiments
ARCHITECTURE
NORMALIZATION
VAE
25
DCGAN
WGAN-GP
SNGAN
3
SSGAN
cGAN
eGAN-PD
BIGGAN
DIFF BIGGAN
Where normalization is referred to images in [-1, 1]. To better interpret the meaning of e in terms of pixels modification, the average change of a pixel value is expressed by V32x32x3' For example, in DCGAN, = 5 is equivalent to an average change of z 0.1. We note that in this example, the pixels are in a range between -1 to and therefore, the mean change is Rs 5%
C.2 ImageNet
Adversarial Robust Classifier We use a pretrained robust ResNet-50 on ImageNet, provided in (Engstrom el al. 2019), on both 128 x 128 and 256 x 256. This model adversarially trained with threat model A 0 8112 < 3) with step size of 0.5.
http:://gthub.cc///uteee/nn/
https://github. cww.eitt htt.a//eec...t....t....o./eeeenr
17

---

Published in Transactions on Machine Learning Research (01/2023)
Image Generators We did not train any generator and utilize the publicly available ones. For SNGAN, SSGAN, and InfoMaxGAN, we use the ones from the mimicry repository. As for BigGAN-deep (truncation= 1.0) and guided diffusion, we utilize the fact that BIGRoC can operate on standalone images and utilize the sets generated images, published in guided diffusion's 's repository and we apply our method upon these. We test our method using the aforementioned sets of generated images using two setups with and without the ground truth labels. When operating without the labels, we produce pscudo labels using our robust classifier and then apply BIGRoC.
BIGRoC hyperparameters As in CIFAR-10, only tune the value of e. In the below table, report the used values for our tested architectures.
Table 8: BIGRoC's values in ImageNet experiments
RESOLUTION ARCHITECTURE
NORMALIZATION
SNGAN
40
SSGAN
40
128
INROMAXGAN
40
BIGGAN
GUIDED DIFFUSION
1.5
X
256
BIGGAN-DEEP
GUIDED DIFFUSION
1.5
X
C.3
Comparison with other methods
Adversarial Robust Classifier For CIFAR-10 we use the same robust classifier as described in Appendix
C.1 and for ImageNet we use the same model as in Appendix C.2
Image Generators To fairly compare BIGRoC and the competitive methods, we use BIGRoC to refine the outputs of the same pretrained model as DOT, DDLSS and DG-flow SN-ReNNet-GANI We experiment in both CIFAR-10 and ImageNet and compare our results with the reported oncs of the other methods. The missing values in Table 3 stem from the fact that DOT did not report its results NeAR CIFAR-10, and DG-flow was not tested on ImageNet any other high-resolution dataset.
BIGRoC hyperparameters We report in the table below the value of epsilon used to attain the results in Table g
Table 9: BIGRoC's values in Table B
experiments
DATASET
NORMALIZATION
CIFAR-10 1.8
X
IMAGENET 15
X
D Fidelity-Diversity Analysis
In image synthesis, fidelity and diversity measure the photorealism and distribution faithfulness of the generated images. In the main paper, we considered FID and Inception Score as our quantitative metrics for evaluating the quality of the generated and boosted images. While IS mainly focuses on fidelity, FID accounts for both. However, to better understand BIGRoC's contribution in terms of fidelity and diversity, we report in Table 10 the precision and recall scores Naeem et al. (2020) of applying our method to different generative models, trained on ImageNet 128 128. Theoretically, using too large values when applying
18

---

Published in Transactions on Machine Learning Research (01/2023)
method can improve the samples' fidelity but potentially sacrifice the diversity. However, abstain from this by choosing proper values. This way, we improve the overall generation quality, expressed by better FID scores. We demonstrate the effect of e in Figure G using too large values boosts the fidelity (expressed via IS) but harms the diversity (reffected in FID). However, as can be seen from the table, a proper choice of e enables us improve the overall quality, expressed by the FID score, without trading off diversity for fidelity.
Table 10: Srecision-Recall results on ImageNet 128
128.
Architecture
Resolution
Precision
Recall
FID
SNGAN (Miyato et al. 2018
128 x 128
0.25
0.43
62.28
w/ BIGRoOPI.
0.32
0.52
40.40
+0.07
+0.09
-21.88
SSGAN (Chen et al. 2019)
128 128
0.27
0.43
63.60
w/ BIGRoCPL
0.33
0.51
38.93
A
+0.06
+0.08
-24.67
InfoMaxGAN
Lee et al. 2021
128 x 128
0.27
0.46
60.61
w/ BIGRoCPL
0.33
0.52
37.70
+0.06
+0.06
-2.911
ADM (Dhariwal & Nichol 2021
128 x 128
0.70
0.65
5.91
w/ BIGRoCGr
0.72
0.64
4.41
+0.02
-0.01
-1.5
Guided Diffusion (Dhariwal & Nichol 2021)
128 128
0.78
0.59
2.97
w/ BIGRoCGr
0.79
0.59
2.75
A
+0.01
-0.22
E Qualitative Results
this section, we show additional qualitative results to further demonstrate the qualitative enhancement attained by our method. We use image generators of different qualitics, both conditional and unconditional, and show the generated images and the boosted ones in Figures TO 11
12 and 13 The images below are
simply the 100 first synthesized images from each class.
F Debiasing
In Section 3 describe debiasing algorithm, which aims to induce uniform class distribution the outputs of BIGRoC. The rationale behind this procedure is expressed in Equation 2 To better understand the outcome of our debiasing, we present in Figure 14 its effect visually. We compare the results of applying BIGRoC in the unconditional case, with and without debiasing. One can clearly see that it reduces the amount of the majority class and leads to a more uniform class distribution, by modifying the images accordingly.
G Computational Resources
In this work, we do not train generative models (when possible) and use publicly available pretrained ones generate images and apply BIGRoC on their outputs. This way, we decouple the generative model's com- plexity from our algorithm since it is applied to the generated images themselves. In all of our experiments, the Google COLAB service with single GPU.
19

---

Published in Transactions on Machine Learning Research (01/202))
A comparison between BigGAN generated images of class automobile and the corresponding boosted ones.
(b) A comparison between BigCAN gencrated images of class bird and the corresponding boosted ones.
A comparison between BigGAN generated images of class dog and the corresponding boosted ones.
Figure 10: BIGRoC refinement of BigGAN. A qualitative comparison between BigGAN generated images of CIFAR-10 samples and the refined ones by BIGRoC algorithm.
20

---

Published in Transactions on Machine Learning Research (01/2023)
Figure 11: BIGRoC refinement of VAE. qualitative comparison between an unconditional VAE gen- erated images of CIFAR-10 samples and the refined ones by BIGRoC algorithm.
Figure 12: BIGRoC refinement of WGAN-GP. A qualitative comparison between an unconditional
WGAN-GP generated images of CIFAR-10 samples and refined ones by BIGRoC algorithm.
21

---

Published in Transactions on Machine Learning Research (01/2023)
Figure 13: BIGRoC refinement of SSGAN. A qualitative comparison between an unconditional SSGAN gencrated images CIFAR-10 samples and refined ones by BIGRoC algorithm..
Boosting without Debiasing
samples
class idx
Boosting without Debiasing
Boosting with Debiasing
num samples
gen
Boosting with Debiasing
Figure 14: Demonstration of the debiasing technique. We show 100 generated images by an uncon- ditional SNGAN and the results of the BIGRoC algorithm, with and without the proposed debiasing. As be seen, the outputs of the boosting algorithms perceptually superior, while the histograms expose the fact that the suggested debiasing algorithm induces more uniform labels distribution. In the "Boosting without Debiasing" experiment, 36 out of 100 images are classified as deers, and only 3 are horses. The most prominent deer images are marked in red. However, when applying the debiased boosting, the number of deers is reduced to 9, and the number of horses increased to 15. We mark the boosted images that remain deer in red, and images that are modified to other minority classes green. As can be seen, many of the deers were changed to be horses, a perceptually similar class.
22


Reviews:
[{"review_id": "kZbWBwiwjw", "reviewer": "Reviewer_Jnwa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n+ The motivation of this paper is clear and strong. Applying a robust classifier to boost visual generation is straightforward but meaningful.\n\n+ The whole paper is well written, with clear logic to follow. The discussion of related works is comprehensive and includes most of the related literature.\n\n+ The proposed model is efficient and is supposed to be computationally friendly. There is no finetuning of generation models, which usually takes much effort. \n\n+ I have not noticed a similar previous work to apply a robust classifier to boost the visual generation model. The overall idea is novel.\n\n\nCons:\n\n- The refinements in visual quality seem marginal (I have not seen much difference between the two methods in Fig.1). And some FID score also shows a trivial improvement.\n\n- There are many hyper-parameters to be assigned. This may need some hand-craft tricks to make it work, which is not friendly for new developers.\n\n- Some important experiments are missing. I recommend the authors establish a user study for comparison, which could be an important evaluation matrix since the authors claimed that the robust classifier could help generative results better align with human perception.\n\nQuestion:\n\nWill authors have any plans to extend this framework to text-to-image generation?\n\nIs it any discussion of weakness?\n\nWhat is the time/computation cost of your proposed post-processing?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a model-agnostic framework to boost existing visual generation models with a robust classifier. Only post-processing can improve the visual quality without any model finetuning. This is inspired by the adversarial robustness that the small perturbation would not change the prediction. The authors assume that modifying an image to sharpen such a classifier’s decision yields visual features perceptually aligned with the target class. Thus, this paper introduces the BIGRoC, which can be applied for both conditional and unconditional generation given an adversarially trained classifier. BIGRoC has shown large potential on a variety of benchmarks, including ImageNet and CIFAR.\n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L3vvoJrqL3", "reviewer": "Reviewer_AYWs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength：\n\n-This paper uses a robust classifier to enhance the ability of generative models. It is glad to see that this method can be combined with generative models, includings WGAN, SNGAN, SSGAN, etc.\n\n- The authors conduct extensive ablation study to investigate the effectiveness of each components.\n\nWeakness:\n\n-The classifier improve image's quality by discovering adversarial examples. There are many methods about adversarial examples, imposing the classifier can be designed with different formulation. It is interesting to investigate whether the formulation of classifier affect the quality of  generated images.\n\n-The robust classifier seems a general method. Besides image synthesis, could the method be used for other low-level tasks, such as Image super-resolution？", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose an new strategy to enhance the quality of synthesized images by introducing a robust classifier. The classifier can capture the semantic information, which covers the shortage of a generative model. It is interesting that a classifier for high-level vision task can help a low-level vision task. The authors conduct extensive experiments on ImageNet and CIFAR, where images with high quality are generated.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xvtRcKqX9v", "reviewer": "Reviewer_5TM6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n* Paper is clearly written \n* Shows improved image quality on benchmark datasets.\n\n## Weaknesses\n* There isn’t enough results (either theoretical or empirical) for validating and understanding the proposed method. See Requested Changes below. \n* The impact of this study can be limited as current SOTA image generation models based on large foundation models are substantially better than the models used in the paper.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method named BIGRoC, which performs image refinement by running PGD w.r.t. an adversarially trained (a.k.a. robust) classifier.  The authors further propose ways to class-balance the generation when not given GT class labels.  In terms of FID and IS, BIGRoC showed improved results on CIFAR and ImageNet datasets.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A major question that remains unanswered for me is whether the improved image quality comes with a cost of diversity.  Intuitively, increasing class-conditional likelihood will make the image look more specific to the object class, and appear more visually appealing. But from a generative modelling perspective, distribution coverage and sample diversity are equally important.  Both IS and FID are pretty simplistic metrics that do not capture image diversity well and the human preference study was done as an image-level where the human subject cannot judge distribution-level performance.  Here are some suggestions of how this question might be answered:\n1. In terms of evaluation, how does it do in terms of data likelihood? \n2. It’s intuitive that with class-dependent gradient the image quality improves due to moving towards a local optimum in the likelihood space, but does that come at a sacrifice of “coverage”?  \nWhat about class-dependent FID, and other non-parametric metrics like those suggested in “Reliable Fidelity and Diversity Metrics for Generative Models”?\n\nThe comparison to baseline methods in image refinement also seem to be very brief (only Table 3).  \n\nWhile I do think this paper studies an interesting problem, and the proposed method seems reasonable and shows some good results, I think this paper can benefit the readers more if the above question is answered. \n", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model-agnostic method to improve the quality of synthesized images. The proposed robust classifier is reasonable, whose effectiveness is verified by extensive experiments. All the reviewers recognize the contribution of this paper. 

During the rebuttal phase, the authors post more explanation (e.g., generalization of the proposed method) and experimental analysis (e.g., experiments with several generative models trained on ImageNet 128x128 ). These responses answer most of the questions proposed by reviewers. All the reviewers post positive recommendations (i.e., 'Leaning Accept' and 'Accept'). Therefore, I recommend acceptance of this paper.
Decision: Accept (poster)