Year: 2021

Abstract:
The interest of the machine learning community in image synthesis has grown significantly in recent years, with the introduction of a wide range of deep generative models and means for training them. Such machines’ ultimate goal is to match the distributions of the given training images and the synthesized ones. In this work, we propose a general model-agnostic technique for improving the image quality and the distribution fidelity of generated images, obtained by any generative model. Our method, termed BIGRoC (boosting image generation via a robust classifier), is based on a post-processing procedure via the guidance of a given robust classifier and without a need for additional training of the generative model. Given a synthesized image, we propose to update it through projected gradient steps over the robust classifier, in an attempt to refine its recognition. We demonstrate this post-processing algorithm on various image synthesis methods and show a significant improvement of the generated images, both quantitatively and qualitatively.

text: The interest of the machine learning community in image synthesis has grown significantly in recent years, with the introduction of a wide range of deep generative models and means for training them. Such machines’ ultimate goal is to match the distributions of the given training images and the synthesized ones. In this work, we propose a general model-agnostic technique for improving the image quality and the distribution fidelity of generated images, obtained by any generative model. Our method, termed BIGRoC (boosting image generation via a robust classifier), is based on a post-processing procedure via the guidance of a given robust classifier and without a need for additional training of the generative model. Given a synthesized image, we propose to update it through projected gradient steps over the robust classifier, in an attempt to refine its recognition. We demonstrate this post-processing algorithm on various image synthesis methods and show a significant improvement of the generated images, both quantitatively and qualitatively.

heading: 1 INTRODUCTION
text: Deep generative models (DGMs) are a class of deep neural networks trained to model complicated high-dimensional data (Bond-Taylor et al., 2021). Such models receive a large number of samples that follow a certain data distribution, x ∼ PD(x), and aim to produce samples from the same statistics. One of the most fascinating generative tasks is image synthesis, which is notoriously hard, due to the complexity of the natural images’ manifold. Nevertheless, deep generative models for image synthesis have gained tremendous popularity in recent years, revolutionized the field and became state-of-the-art in various tasks (Brock et al., 2019; Karras et al., 2020b; Zhu et al., 2017). Energybased models, variational autoencoders, generative adversarial networks (GANs), autoregressive likelihood models, normalization flows, diffusion-based algorithms and more, all aim to synthesize natural-looking images, ranging from relatively simple to extremely complicated generators, often containing millions of parameters (Kingma & Welling, 2014; Goodfellow et al., 2014; Oord et al., 2016; Rezende & Mohamed, 2015; Ho et al., 2020).
When operating on a multiclass labeled dataset, as considered in this paper, image synthesis can be either conditional or unconditional. In the unconditional setup, the generative model aims to
produce samples from the target data distribution without receiving any information regarding the target class of the synthesized images, i.e., sample from PD(x). In contrast, in the conditional setup, the generator goal is to synthesize images from a designated class, i.e., sample from PD(x|y) where y is the label. As such, conditional generative models receive additional class-related information.
Most of the work in the deep generative models’ field has been focusing on improving the quality and the variety of the images produced by such models, tackled by seeking novel architectures and training procedures. In this work, while still aiming to improve the performance of trained generative models, we place a different emphasis than in most of these studies and propose a method for boosting generative models without any re-training or fine-tuning. More specifically, our method improves the perceptual quality of the images synthesized by any given model via an iterative postprocessing procedure driven by a robust classifier.
With the introduction of learning-based machines into “real-world” applications, the interest in the robustness of such models has become a central concern. While there are abundant of definitions for robustness, the most common and studied is the adversarial one. This definition upholds if a classifier is robust to a small perturbation of its input, made by an adversary in order to fool it. Previous work (Szegedy et al., 2014) has demonstrated that deep neural networks are not robust at all and can be easily fooled by an adversary. In light of this observation, many robustification methods were proposed, but the most popular among these is adversarial training (Goodfellow et al., 2015). According to this method, in order to train a robust classifier, one should generate adversarial examples and incorporate them into the training process. While examining the properties of such classifiers, researchers have revealed a fascinating phenomenon, called perceptually aligned gradients (Tsipras et al., 2019). According to this tendency, a modification of an image that sharpens such a classifier’s decision yields visual features that are perceptually aligned with the target class. In other words, when drifting an image content to be better classified, the changes obtained are visually pleasing and faithful to natural image content.
In this work we harness and utilize the above described phenomenon – we propose to iteratively modify the images created by a trained generative model, so as to maximize the conditional probability of a certain target class, approximated by a given robust classifier. This modification can potentially improve the quality of the synthesized images, since it emphasizes visual features that are aligned with images of the target class, thus boosting the generation process both in terms of perceptual quality and distribution faithfulness. We hypothesize that given an image dataset, the supervised training of a robust classifier is much simpler and effective than the unsupervised training of a generative model, thus enabling an indirect yet powerful improvement of generative models. We term this method “BIGRoC” – Boosting Image Generation via a Robust Classifier.
The method presented in this article is general and model-agnostic, and it can be applied to any image generator, both conditional or unconditional. In the unconditional case, since we do not have a target class to guide the boosting process, we propose to estimate it via the trained robust classifier. The marked performance improvement achieved by our proposed method is demonstrated in a series of experiments on a wide range of image generators. We show that this approach enables us to significantly improve the quality of images synthesized by relatively simple models, boosting them to a level of more sophisticated and complex generators. Furthermore, we demonstrate the ability of our method to enhance the performance of higher-quality generative architectures, both qualitatively and quantitatively. In addition to our contribution on enhancing image generation performance, we leverage the same robust classifier and its perceptually aligned gradient property for proposing an adversarial technique for image interpolation.
Our approach is inspired by two main works: Santurkar et al. (2019) and Turner et al. (2019). Santurkar et al. (2019) have shown that a single robust classifier is capable of tackling various computer vision problems, such as generation, inpainting and image-to-image translation, all achieved by utilizing the perceptually aligned gradients phenomenon. Turner et al. (2019) aims at improving the generation quality of GANs, by discarding low quality images, identified by the GAN’s discriminator. While our work is related to the above two papers, it differs substantially in the following main characteristics: (i) While our boosting technique relies on the perceptually aligned gradient property as in Santurkar et al. (2019), it builds on the outcome of pre-trained generative models, thus getting
much higher quality images; (ii) Our proposed boosting can operate on any image synthesizer; and (iii) All images produced by the generator are taken into account, none being discarded.

heading: 2 BACKGROUND

heading: 2.1 ADVERSARIAL EXAMPLES
text: Adversarial examples are instances that are intentionally designed by an attacker to cause a false prediction by a machine learning-based classifier (Szegedy et al., 2014). The generation procedure of such examples is based on applying modifications to given training examples, while restricting the allowed perturbations ∆. Ideally, the “threat model” ∆ should include all the possible perturbations that are unnoticeable to a human observer. As it is impossible to rigorously define such a set, in practice a simple subset of the ideal threat model is used, where the most common choices are the `2 and the `∞ balls: ∆ = {δ : ‖δ‖2/∞ ≤ }. Given ∆, the attacker receives an instance x and generates x̂ = x + δ s.t. δ ∈ ∆, while aiming to fool the classifier. Adversarial attacks can be both untargeted or targeted: An untargeted attack perturbs the input in a way that minimizes p(y|x̂) with respect to δ. In contrast, a targeted attack receives in addition the target class ŷ, and perturbs x to maximize p(ŷ|x̂). There are diverse techniques for generating adversarial examples, yet, in this work, we focus on targeted attacks using the Projected Gradient Descent (PGD) method (Madry et al., 2018)– an iterative method for creating adversarial examples that operates as shown in Algorithm 1.
Algorithm 1: Targeted Projected Gradient Descent (PGD) for adversarial targeted attacks Input: classifier fθ, input x, target class ŷ, , step size α, number of iterations T δ0 ← 0 for t from 0 to T do
δt+1 = Π (δt − α∇δ`(fθ(x+ δt), ŷ)); end xadv = x+ δT Output: xadv
The operation Π stands for a projection operator onto ∆, and `(·) is the classification loss.

heading: 2.2 ADVERSARIAL ROBUSTNESS
text: Adversarial robustness is a property of classifiers, according to which, applying small perturbation on a classifier’s input in order to fool it does not affect its prediction (Goodfellow et al., 2015). To attain such classifiers, one should solve the following optimization problem:
min θ ∑ x,y∈D max δ∈∆ `(fθ(x+ δ), y) (1)
Namely, train the classifier to accurately predict the class labels of the ”toughest” perturbed images, allowed by the threat model ∆. In practice, solving this optimization problem is challenging, and there are several ways to attain an approximated solution. The most simple yet effective method is based on approximating the solution of the inner-maximization via adversarial attacks, such as PGD. According to this strategy, the above optimization is performed iteratively, fixing the classifier’s parameters θ and optimizing the attacks δ for each example via PGD, and then fixing these and updating θ. Repeating these steps results in a robust classifier, as we use in this work.

heading: 2.3 PERCEPTUALLY ALIGNED GRADIENTS
text: Perceptually aligned gradients is a phenomenon that occurs in adversarially trained models when modifying an image to maximize the probability assigned to a target class. Tsipras et al. (2019) show that performing the above PGD process on such models yields meaningful visual features that are
perceptually aligned to the target class. It is important to note that this phenomenon does not occur in non-robust models. The perceptually aligned gradients property indicates that the features learned by robust models are more aligned with human perception. Figure 2 presents a visual demonstration of this fascinating phenomenon.

heading: 3 BOOSTING IMAGE GENERATION VIA A ROBUST CLASSIFIER
text: We propose a method for improving the quality of images synthesized by trained generative models, named BIGRoC: Boosting Image Generation via a Robust Classifier. Our method is model agnostic and does not require additional training or fine-tuning of the generative model, and can be viewed as a post processing step. Thus, BIGRoC can be easily applied to any generative model, both conditional or unconditional. This mechanism harnesses the perceptually aligned gradients phenomenon to further tweak the generated images to improve their visual quality. To do so, we perform an iterative process of modifying the generated image x to maximize the posterior probability of a given target class ŷ, pθ(ŷ|x), where pθ is modeled by an adversarially trained classifier. This can be achieved by performing a PGD-like process, but instead of adversarially changing an image x of class y to a different class ŷ 6= y, we propose to modify it in a way that maximizes the probability that x belongs to y. Therefore, our method requires a trained robust classifier that operates on the same data source as the generator we aim to improve.
In the conditional generation process, the generator G receives the class label y, from which it suppose to draw samples. Hence, in this setup, we have information regarding the class affiliation of the image and we can maximize the corresponding conditional probability. In the unconditional generation process, the generator does not receive class labels at all and its goal is to draw samples from p(x). Thus, in this case, we cannot directly maximize the desired posterior probability, as our method suggests. To bridge this gap, we propose to estimate the most likely class via our robust classifier fθ, and afterward modify the image via the suggested method to maximize its probability. The proposed image generation boosting is described in Algorithm 2, for both the conditional and the unconditional schemes.
While the above-described approach for unconditional sampling works well, it could be further improved. We have noticed that in this case, estimating the target classes Y of Xgen via fθ leads
to unbalanced labels. For example, in the CIFAR-10, when generating 50,000 samples, we expect approximately 5,000 images per each of the 10 classes, and yet the labels’ estimation do not distribute uniformly at all. This imbalance causes a bias in the target labels estimation of the boosting algorithm, affects the visual content of Xboost and limits the quantitative improvement attained by BIGRoC. We emphasize that this issue is manifested only in the quantitative metrics, and when qualitatively evaluating the boosted images, the improvement is significant, as can be seen in Figure 3.
To further enhance the quantitative results of our algorithm in the unconditional case, we propose to de-bias the target class estimation of Xgen, and attain close to uniform class estimations. A naive solution to this can be achieved by generating more samples and extracting a subset of these images with a labels-balance. This approach is computationally heavy and does not use the generated images as-is, which raises questions regarding the fairness of the quantitative comparison. Thus, we propose a different debiasing technique – we modify the classifier’s class estimation to become more balanced by calibrating its logits. More specifically, we shift the classifier’s logits by adding a per-class pre-calculated value, dci , that induces equality of the mean logits value across all classes. For simplicity, we denote logitci as the logit of class ci corresponding to a generated sample xgen. We approximate Exgen [logitci ] for each class ci, using a validation set of generated images, and calculate a per-class debiasing factor: dci = a− Êxgen [logitci ] (WLOG, a = 1), where Êxgen [logitci ] is a mean estimator. After calculating dci , given a generated image xgen, we calculate its logits and add dci to it to obtain debiased logits ( ˆlogitci ), from which we derive the unbiased class estimation via softmax. The following equation shows that, given a correct estimation of the per-class logits’ mean, the per-class means of the debiased logits are equal:
Exgen [ ˆlogitci ] = Exgen [dci + logitci ] = Exgen [a− Êxgen [logitci ] + logitci ] = = a− Êxgen [logitci ] + Exgen [logitci ] ≈ a (2)
Figure 3 presents a demonstration that verifies the validity of this method and shows its qualitative effects on the unconditional generation boosting.
Algorithm 2: BIGRoC: Boosting Image Generation via a Robust Classifier Input: Robust classifier fθ, xgen, ygen, , step size α, number of iterations T , dc if ygen is None then
ygen = argmax(fθ(xgen) + dc) end xboost = Targeted PGD(fθ, xgen, ygen, , α, T ) Output: xboost
As can be seen in Algorithm 2, it receives as input the generated images and their designated labels (if exist) and returns an improved version of them. As such, this method can be applied at the inference phase of generative models to enhance their performance, in a manner totally separated from their training. As can be seen from the algorithm’s description, it has several hyperparameters that determine the modification process of the image: sets the maximal size of the perturbation allowed by the threat model ∆, α controls the step size at each update step and T is the number of updates. Another choice is the norm used to define the threat model ∆.
The hyperparameter is central in our scheme - when is too large, the method overrides the input and modifies the original content in an unrecognizable way, as can be seen in Figure 2. On the other hand, when is too small, the boosted images remain very similar to the input ones, leading to a minor enhancement. As our goal is to obtain a significant enhancement to the synthesized images, a careful choice of should be practiced, which restricts the allowed perturbations in the threat model.
Another important choice is the threat model ∆ itself. Two of the most common choices of ∆ for adversarial attacks are the `∞ and the `2 balls. Due to the desired behavior of our method, using the `∞ ball is less preferable: it allows a change of ± to every pixel and as such it will not focus on meaningful specific locations and might not preserve the existing structure of the synthesized input image. Thus, we choose the `2 ball as our threat model, with relatively small . Such a choice restricts the allowed perturbations and leads to changes that may concentrate in specific locations,
while preserving most of the existing content in the generated images. A visual demonstration of these considerations is given in Figure 4.

heading: 4 ROBUST CLASSIFIER BASED IMAGE INTERPOLATION
text: As a byproduct of the boosting algorithm for improving the quality of synthesized images, we develop a method for image interpolation, which stems from a similar rationale. In image synthesis, image interpolation is an operation that aims to generate intermediate pictures between two given images, while remaining meaningful and of gradual change. We propose to use features extracted by our robust classifier at different scales in order to guide a PGD-based interpolation process. More specifically, given a source image xs and a target image xt, we extract their corresponding features {F 1xs , . . . , F k xs} and {F 1 xt , . . . , F k xt} from our trained robust classifier fθ. To perform interpolation, we modify xs such that its features will be more similar to those of xt at the different scales, using PGD. Mathematically, we aim to solve Equation 3:
argminδ∈∆ k∑ i=1 wi · ‖F ixs+δ − F i xt‖2 (3)
where {wi}ki=1 is an importance vector to the different features’ scales. Namely, this optimization aims to find a perturbation δ ∈ ∆ that minimizes the `2-distance between the images’ representations. In the above formulation, the choice of the determines the level of change allowed in the process. Thus, solving the above optimization with different values leads to different intermediary images – as we increase , the interpolation resembles the target image more.

heading: 5 EXPERIMENTS
text: Image Generation Enhancement: We conduct extensive experiments to demonstrate the performance improvement achieved by the proposed method. In this work, we focus on CIFAR-10 (Krizhevsky, 2012), the most common benchmark for image synthesis. Since BIGRoC is modelagnostic, it can be easily applied to any generative model, given a robust classifier, trained on the same data source. Although our proposed method is a general technique, we experiment mainly with Generative Adversarial Networks (GANs), due to their popularity and generation quality. In all of our experiments, we use a single adversarially trained ResNet-50 (H. et al., 2016) as the robust classifier. We further elaborate regarding the experimental setups in Appendix .2. We utilize the model-agnostic property to examine the effects of applying the proposed boosting over a wide variety of image generators of different qualities: from relatively simple ones to sophisticated and complex ones. Moreover, we test our approach on both conditional and unconditional generative models to show that the proposed scheme can enhance different synthesis procedures.
Given a generative model, we use it to synthesize a set of images Xgen and apply our method to generateXboost, according to Algorithm 2. We compare betweenXg andXboost quantitatively using two image synthesis evaluation methods: Fréchet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016). Our quantitative results (Tables 1, 2) indicate that our method achieves a substantial improvement across a wide range of tested generator architectures, both conditional and unconditional, demonstrating the method’s versatility and validity. In addition, the fact the same robust classifier is capable of enhancing the performance of such a wide range of generative models, demonstrates the power of BIGRoC. Furthermore, we show in Figure 1, 5 and in Appendix .1 the qualitative results that indicate that the “boosted“ results are more pleasing and clear to human observers. All the conducted experiments verify the significant enhancement attained by our method, both quantitatively and qualitatively.
Image Interpolation: We conduct image interpolation experiments according to the technique proposed in Section 4. We use the ImageNet dataset (Deng et al., 2009) with resolution of 224 × 224 and a ResNet-50 classifier. We compare the visual results attained by our interpolation method using both robust and non-robust classifiers of the same architecture. As can be seen in Figure 6, the interpolation results using a robust classifier are much more convincing and the intermediary images are perceptually plausible. In contrast, the interpolation using a non-robust model leads to much inferior results, and the final interpolation does not look like the target image at all.

heading: 6 DISCUSSION AND CONCLUSIONS
text: In this work, we propose a novel method that leverages the perceptually aligned gradients phenomenon for enhancing the visual quality of synthesized images. Our approach does not require additional training of the generative model and it is completely model agnostic. Thus, it can be applied during the inference phase of any model. Clearly, training a robust classifier is a simpler task than training a generative model. Due to the core ability of such a robust classifier to better capture significant visual features, it is capable of effectively and efficiently improving the output of generative models. In a line of experiments, we confirm this rationale and show that this method is highly effective and capable of substantially enhancing both the qualitative and quantitative results of a wide range of generative models. In addition, we develop an image interpolation method that stems from a similar origin – adversarial attacks on a robust classifier.

heading: 7 REPRODUCIBILITY STATEMENT
text: In this work we have proposed a post-processing algorithm for enhancing the visual quality of generated images synthesized by generative models. We implemented our approach using the official software or using implementations from Lee & Town (2020) in each of the papers we have referred to, and using the same hyperparameters as stated in their experiments. In addition, in all of our tests,
we evaluated the performance multiple times using different seeds, so as to factor out randomness in the attained improvement. A code for running the conducted experiments for the unconditional generation case is provided in the supplementary materials. A complete code package will be published upon acceptance of the paper.

Review summaries:
This paper presents a post-processing method to improve the GAN results. The post-processing consists of a projected gradient descent step to update the generated image to fit its target class, which works with both conditional/unconditional generative models. For the unconditional generative model, the authors design a de-bias method to force the model to uniformly generate images in each class. The results show this model can significantly improve the FID and IS scores on CIFAR-10 dataset. Additionally, the authors show this post-processing also works for image interpolation. 

The paper proposes BIGRoC (Boosting Image Generation via a Robust Classifier), a method to refine samples from a base generative model using a robust classifier. In this context, a robust classifier means one that has been trained to be robust to adversarial samples. Given a robust classifier and a base generative model (both trained on the same dataset), the key idea involves using the gradients of the robust classifier to update samples generated from the base generative model in the direction that maximizes the conditional probability of the samples' class. Experiments on the CIFAR-10 dataset have been presented that demonstrate that BIGRoC improves the sample quality from several base generative models in terms of the FID and IS metrics.

This paper proposes a model-agnostic method for improving the quality of images produced by generative models. The method requires a robust classifier trained on the same data source as the generative models. Based on the perceptually aligned gradients phenomenon, the proposed method improves the quality of a generated image by the targeted projected gradient descent method with the help of the robust classifier. Experiments on CIFAR-10 show that the proposed method does improve several generative models both quantitively and qualitatively.