BIGRoC: Boosting Image Generation via a Robust Classifier

The interest of the machine learning community in image synthesis has grown significantly in recent years , with the introduction of a wide range of deep generative models and means for training them . Such machines ’ ultimate goal is to match the distributions of the given training images and the synthesized ones . In this work , we propose a general model-agnostic technique for improving the image quality and the distribution fidelity of generated images , obtained by any generative model . Our method , termed BIGRoC ( boosting image generation via a robust classifier ) , is based on a post-processing procedure via the guidance of a given robust classifier and without a need for additional training of the generative model . Given a synthesized image , we propose to update it through projected gradient steps over the robust classifier , in an attempt to refine its recognition . We demonstrate this post-processing algorithm on various image synthesis methods and show a significant improvement of the generated images , both quantitatively and qualitatively . 1 INTRODUCTION . Deep generative models ( DGMs ) are a class of deep neural networks trained to model complicated high-dimensional data ( Bond-Taylor et al. , 2021 ) . Such models receive a large number of samples that follow a certain data distribution , x ∼ PD ( x ) , and aim to produce samples from the same statistics . One of the most fascinating generative tasks is image synthesis , which is notoriously hard , due to the complexity of the natural images ’ manifold . Nevertheless , deep generative models for image synthesis have gained tremendous popularity in recent years , revolutionized the field and became state-of-the-art in various tasks ( Brock et al. , 2019 ; Karras et al. , 2020b ; Zhu et al. , 2017 ) . Energybased models , variational autoencoders , generative adversarial networks ( GANs ) , autoregressive likelihood models , normalization flows , diffusion-based algorithms and more , all aim to synthesize natural-looking images , ranging from relatively simple to extremely complicated generators , often containing millions of parameters ( Kingma & Welling , 2014 ; Goodfellow et al. , 2014 ; Oord et al. , 2016 ; Rezende & Mohamed , 2015 ; Ho et al. , 2020 ) . When operating on a multiclass labeled dataset , as considered in this paper , image synthesis can be either conditional or unconditional . In the unconditional setup , the generative model aims to produce samples from the target data distribution without receiving any information regarding the target class of the synthesized images , i.e. , sample from PD ( x ) . In contrast , in the conditional setup , the generator goal is to synthesize images from a designated class , i.e. , sample from PD ( x|y ) where y is the label . As such , conditional generative models receive additional class-related information . Most of the work in the deep generative models ’ field has been focusing on improving the quality and the variety of the images produced by such models , tackled by seeking novel architectures and training procedures . In this work , while still aiming to improve the performance of trained generative models , we place a different emphasis than in most of these studies and propose a method for boosting generative models without any re-training or fine-tuning . More specifically , our method improves the perceptual quality of the images synthesized by any given model via an iterative postprocessing procedure driven by a robust classifier . With the introduction of learning-based machines into “ real-world ” applications , the interest in the robustness of such models has become a central concern . While there are abundant of definitions for robustness , the most common and studied is the adversarial one . This definition upholds if a classifier is robust to a small perturbation of its input , made by an adversary in order to fool it . Previous work ( Szegedy et al. , 2014 ) has demonstrated that deep neural networks are not robust at all and can be easily fooled by an adversary . In light of this observation , many robustification methods were proposed , but the most popular among these is adversarial training ( Goodfellow et al. , 2015 ) . According to this method , in order to train a robust classifier , one should generate adversarial examples and incorporate them into the training process . While examining the properties of such classifiers , researchers have revealed a fascinating phenomenon , called perceptually aligned gradients ( Tsipras et al. , 2019 ) . According to this tendency , a modification of an image that sharpens such a classifier ’ s decision yields visual features that are perceptually aligned with the target class . In other words , when drifting an image content to be better classified , the changes obtained are visually pleasing and faithful to natural image content . In this work we harness and utilize the above described phenomenon – we propose to iteratively modify the images created by a trained generative model , so as to maximize the conditional probability of a certain target class , approximated by a given robust classifier . This modification can potentially improve the quality of the synthesized images , since it emphasizes visual features that are aligned with images of the target class , thus boosting the generation process both in terms of perceptual quality and distribution faithfulness . We hypothesize that given an image dataset , the supervised training of a robust classifier is much simpler and effective than the unsupervised training of a generative model , thus enabling an indirect yet powerful improvement of generative models . We term this method “ BIGRoC ” – Boosting Image Generation via a Robust Classifier . The method presented in this article is general and model-agnostic , and it can be applied to any image generator , both conditional or unconditional . In the unconditional case , since we do not have a target class to guide the boosting process , we propose to estimate it via the trained robust classifier . The marked performance improvement achieved by our proposed method is demonstrated in a series of experiments on a wide range of image generators . We show that this approach enables us to significantly improve the quality of images synthesized by relatively simple models , boosting them to a level of more sophisticated and complex generators . Furthermore , we demonstrate the ability of our method to enhance the performance of higher-quality generative architectures , both qualitatively and quantitatively . In addition to our contribution on enhancing image generation performance , we leverage the same robust classifier and its perceptually aligned gradient property for proposing an adversarial technique for image interpolation . Our approach is inspired by two main works : Santurkar et al . ( 2019 ) and Turner et al . ( 2019 ) . Santurkar et al . ( 2019 ) have shown that a single robust classifier is capable of tackling various computer vision problems , such as generation , inpainting and image-to-image translation , all achieved by utilizing the perceptually aligned gradients phenomenon . Turner et al . ( 2019 ) aims at improving the generation quality of GANs , by discarding low quality images , identified by the GAN ’ s discriminator . While our work is related to the above two papers , it differs substantially in the following main characteristics : ( i ) While our boosting technique relies on the perceptually aligned gradient property as in Santurkar et al . ( 2019 ) , it builds on the outcome of pre-trained generative models , thus getting much higher quality images ; ( ii ) Our proposed boosting can operate on any image synthesizer ; and ( iii ) All images produced by the generator are taken into account , none being discarded . 2 BACKGROUND . 2.1 ADVERSARIAL EXAMPLES . Adversarial examples are instances that are intentionally designed by an attacker to cause a false prediction by a machine learning-based classifier ( Szegedy et al. , 2014 ) . The generation procedure of such examples is based on applying modifications to given training examples , while restricting the allowed perturbations ∆ . Ideally , the “ threat model ” ∆ should include all the possible perturbations that are unnoticeable to a human observer . As it is impossible to rigorously define such a set , in practice a simple subset of the ideal threat model is used , where the most common choices are the ` 2 and the ` ∞ balls : ∆ = { δ : ‖δ‖2/∞ ≤ } . Given ∆ , the attacker receives an instance x and generates x̂ = x + δ s.t . δ ∈ ∆ , while aiming to fool the classifier . Adversarial attacks can be both untargeted or targeted : An untargeted attack perturbs the input in a way that minimizes p ( y|x̂ ) with respect to δ . In contrast , a targeted attack receives in addition the target class ŷ , and perturbs x to maximize p ( ŷ|x̂ ) . There are diverse techniques for generating adversarial examples , yet , in this work , we focus on targeted attacks using the Projected Gradient Descent ( PGD ) method ( Madry et al. , 2018 ) – an iterative method for creating adversarial examples that operates as shown in Algorithm 1 . Algorithm 1 : Targeted Projected Gradient Descent ( PGD ) for adversarial targeted attacks Input : classifier fθ , input x , target class ŷ , , step size α , number of iterations T δ0 ← 0 for t from 0 to T do δt+1 = Π ( δt − α∇δ ` ( fθ ( x+ δt ) , ŷ ) ) ; end xadv = x+ δT Output : xadv The operation Π stands for a projection operator onto ∆ , and ` ( · ) is the classification loss . 2.2 ADVERSARIAL ROBUSTNESS . Adversarial robustness is a property of classifiers , according to which , applying small perturbation on a classifier ’ s input in order to fool it does not affect its prediction ( Goodfellow et al. , 2015 ) . To attain such classifiers , one should solve the following optimization problem : min θ ∑ x , y∈D max δ∈∆ ` ( fθ ( x+ δ ) , y ) ( 1 ) Namely , train the classifier to accurately predict the class labels of the ” toughest ” perturbed images , allowed by the threat model ∆ . In practice , solving this optimization problem is challenging , and there are several ways to attain an approximated solution . The most simple yet effective method is based on approximating the solution of the inner-maximization via adversarial attacks , such as PGD . According to this strategy , the above optimization is performed iteratively , fixing the classifier ’ s parameters θ and optimizing the attacks δ for each example via PGD , and then fixing these and updating θ . Repeating these steps results in a robust classifier , as we use in this work . 2.3 PERCEPTUALLY ALIGNED GRADIENTS . Perceptually aligned gradients is a phenomenon that occurs in adversarially trained models when modifying an image to maximize the probability assigned to a target class . Tsipras et al . ( 2019 ) show that performing the above PGD process on such models yields meaningful visual features that are perceptually aligned to the target class . It is important to note that this phenomenon does not occur in non-robust models . The perceptually aligned gradients property indicates that the features learned by robust models are more aligned with human perception . Figure 2 presents a visual demonstration of this fascinating phenomenon .

Summary: This paper presents a post-processing method to improve the GAN results. The post-processing consists of a projected gradient descent step to update the generated image to fit its target class, which works with both conditional/unconditional generative models. For the unconditional generative model, the authors design a de-bias method to force the model to uniformly generate images in each class. The results show this model can significantly improve the FID and IS scores on CIFAR-10 dataset. Additionally, the authors show this post-processing also works for image interpolation.