BIGRoC: Boosting Image Generation via a Robust Classifier

The interest of the machine learning community in image synthesis has grown significantly in recent years , with the introduction of a wide range of deep generative models and means for training them . Such machines ’ ultimate goal is to match the distributions of the given training images and the synthesized ones . In this work , we propose a general model-agnostic technique for improving the image quality and the distribution fidelity of generated images , obtained by any generative model . Our method , termed BIGRoC ( boosting image generation via a robust classifier ) , is based on a post-processing procedure via the guidance of a given robust classifier and without a need for additional training of the generative model . Given a synthesized image , we propose to update it through projected gradient steps over the robust classifier , in an attempt to refine its recognition . We demonstrate this post-processing algorithm on various image synthesis methods and show a significant improvement of the generated images , both quantitatively and qualitatively . 1 INTRODUCTION . Deep generative models ( DGMs ) are a class of deep neural networks trained to model complicated high-dimensional data ( Bond-Taylor et al. , 2021 ) . Such models receive a large number of samples that follow a certain data distribution , x ∼ PD ( x ) , and aim to produce samples from the same statistics . One of the most fascinating generative tasks is image synthesis , which is notoriously hard , due to the complexity of the natural images ’ manifold . Nevertheless , deep generative models for image synthesis have gained tremendous popularity in recent years , revolutionized the field and became state-of-the-art in various tasks ( Brock et al. , 2019 ; Karras et al. , 2020b ; Zhu et al. , 2017 ) . Energybased models , variational autoencoders , generative adversarial networks ( GANs ) , autoregressive likelihood models , normalization flows , diffusion-based algorithms and more , all aim to synthesize natural-looking images , ranging from relatively simple to extremely complicated generators , often containing millions of parameters ( Kingma & Welling , 2014 ; Goodfellow et al. , 2014 ; Oord et al. , 2016 ; Rezende & Mohamed , 2015 ; Ho et al. , 2020 ) . When operating on a multiclass labeled dataset , as considered in this paper , image synthesis can be either conditional or unconditional . In the unconditional setup , the generative model aims to produce samples from the target data distribution without receiving any information regarding the target class of the synthesized images , i.e. , sample from PD ( x ) . In contrast , in the conditional setup , the generator goal is to synthesize images from a designated class , i.e. , sample from PD ( x|y ) where y is the label . As such , conditional generative models receive additional class-related information . Most of the work in the deep generative models ’ field has been focusing on improving the quality and the variety of the images produced by such models , tackled by seeking novel architectures and training procedures . In this work , while still aiming to improve the performance of trained generative models , we place a different emphasis than in most of these studies and propose a method for boosting generative models without any re-training or fine-tuning . More specifically , our method improves the perceptual quality of the images synthesized by any given model via an iterative postprocessing procedure driven by a robust classifier . With the introduction of learning-based machines into “ real-world ” applications , the interest in the robustness of such models has become a central concern . While there are abundant of definitions for robustness , the most common and studied is the adversarial one . This definition upholds if a classifier is robust to a small perturbation of its input , made by an adversary in order to fool it . Previous work ( Szegedy et al. , 2014 ) has demonstrated that deep neural networks are not robust at all and can be easily fooled by an adversary . In light of this observation , many robustification methods were proposed , but the most popular among these is adversarial training ( Goodfellow et al. , 2015 ) . According to this method , in order to train a robust classifier , one should generate adversarial examples and incorporate them into the training process . While examining the properties of such classifiers , researchers have revealed a fascinating phenomenon , called perceptually aligned gradients ( Tsipras et al. , 2019 ) . According to this tendency , a modification of an image that sharpens such a classifier ’ s decision yields visual features that are perceptually aligned with the target class . In other words , when drifting an image content to be better classified , the changes obtained are visually pleasing and faithful to natural image content . In this work we harness and utilize the above described phenomenon – we propose to iteratively modify the images created by a trained generative model , so as to maximize the conditional probability of a certain target class , approximated by a given robust classifier . This modification can potentially improve the quality of the synthesized images , since it emphasizes visual features that are aligned with images of the target class , thus boosting the generation process both in terms of perceptual quality and distribution faithfulness . We hypothesize that given an image dataset , the supervised training of a robust classifier is much simpler and effective than the unsupervised training of a generative model , thus enabling an indirect yet powerful improvement of generative models . We term this method “ BIGRoC ” – Boosting Image Generation via a Robust Classifier . The method presented in this article is general and model-agnostic , and it can be applied to any image generator , both conditional or unconditional . In the unconditional case , since we do not have a target class to guide the boosting process , we propose to estimate it via the trained robust classifier . The marked performance improvement achieved by our proposed method is demonstrated in a series of experiments on a wide range of image generators . We show that this approach enables us to significantly improve the quality of images synthesized by relatively simple models , boosting them to a level of more sophisticated and complex generators . Furthermore , we demonstrate the ability of our method to enhance the performance of higher-quality generative architectures , both qualitatively and quantitatively . In addition to our contribution on enhancing image generation performance , we leverage the same robust classifier and its perceptually aligned gradient property for proposing an adversarial technique for image interpolation . Our approach is inspired by two main works : Santurkar et al . ( 2019 ) and Turner et al . ( 2019 ) . Santurkar et al . ( 2019 ) have shown that a single robust classifier is capable of tackling various computer vision problems , such as generation , inpainting and image-to-image translation , all achieved by utilizing the perceptually aligned gradients phenomenon . Turner et al . ( 2019 ) aims at improving the generation quality of GANs , by discarding low quality images , identified by the GAN ’ s discriminator . While our work is related to the above two papers , it differs substantially in the following main characteristics : ( i ) While our boosting technique relies on the perceptually aligned gradient property as in Santurkar et al . ( 2019 ) , it builds on the outcome of pre-trained generative models , thus getting much higher quality images ; ( ii ) Our proposed boosting can operate on any image synthesizer ; and ( iii ) All images produced by the generator are taken into account , none being discarded . 2 BACKGROUND . 2.1 ADVERSARIAL EXAMPLES . Adversarial examples are instances that are intentionally designed by an attacker to cause a false prediction by a machine learning-based classifier ( Szegedy et al. , 2014 ) . The generation procedure of such examples is based on applying modifications to given training examples , while restricting the allowed perturbations ∆ . Ideally , the “ threat model ” ∆ should include all the possible perturbations that are unnoticeable to a human observer . As it is impossible to rigorously define such a set , in practice a simple subset of the ideal threat model is used , where the most common choices are the ` 2 and the ` ∞ balls : ∆ = { δ : ‖δ‖2/∞ ≤ } . Given ∆ , the attacker receives an instance x and generates x̂ = x + δ s.t . δ ∈ ∆ , while aiming to fool the classifier . Adversarial attacks can be both untargeted or targeted : An untargeted attack perturbs the input in a way that minimizes p ( y|x̂ ) with respect to δ . In contrast , a targeted attack receives in addition the target class ŷ , and perturbs x to maximize p ( ŷ|x̂ ) . There are diverse techniques for generating adversarial examples , yet , in this work , we focus on targeted attacks using the Projected Gradient Descent ( PGD ) method ( Madry et al. , 2018 ) – an iterative method for creating adversarial examples that operates as shown in Algorithm 1 . Algorithm 1 : Targeted Projected Gradient Descent ( PGD ) for adversarial targeted attacks Input : classifier fθ , input x , target class ŷ , , step size α , number of iterations T δ0 ← 0 for t from 0 to T do δt+1 = Π ( δt − α∇δ ` ( fθ ( x+ δt ) , ŷ ) ) ; end xadv = x+ δT Output : xadv The operation Π stands for a projection operator onto ∆ , and ` ( · ) is the classification loss . 2.2 ADVERSARIAL ROBUSTNESS . Adversarial robustness is a property of classifiers , according to which , applying small perturbation on a classifier ’ s input in order to fool it does not affect its prediction ( Goodfellow et al. , 2015 ) . To attain such classifiers , one should solve the following optimization problem : min θ ∑ x , y∈D max δ∈∆ ` ( fθ ( x+ δ ) , y ) ( 1 ) Namely , train the classifier to accurately predict the class labels of the ” toughest ” perturbed images , allowed by the threat model ∆ . In practice , solving this optimization problem is challenging , and there are several ways to attain an approximated solution . The most simple yet effective method is based on approximating the solution of the inner-maximization via adversarial attacks , such as PGD . According to this strategy , the above optimization is performed iteratively , fixing the classifier ’ s parameters θ and optimizing the attacks δ for each example via PGD , and then fixing these and updating θ . Repeating these steps results in a robust classifier , as we use in this work . 2.3 PERCEPTUALLY ALIGNED GRADIENTS . Perceptually aligned gradients is a phenomenon that occurs in adversarially trained models when modifying an image to maximize the probability assigned to a target class . Tsipras et al . ( 2019 ) show that performing the above PGD process on such models yields meaningful visual features that are perceptually aligned to the target class . It is important to note that this phenomenon does not occur in non-robust models . The perceptually aligned gradients property indicates that the features learned by robust models are more aligned with human perception . Figure 2 presents a visual demonstration of this fascinating phenomenon .

Summary: The paper proposes BIGRoC (Boosting Image Generation via a Robust Classifier), a method to refine samples from a base generative model using a robust classifier. In this context, a robust classifier means one that has been trained to be robust to adversarial samples. Given a robust classifier and a base generative model (both trained on the same dataset), the key idea involves using the gradients of the robust classifier to update samples generated from the base generative model in the direction that maximizes the conditional probability of the samples' class. Experiments on the CIFAR-10 dataset have been presented that demonstrate that BIGRoC improves the sample quality from several base generative models in terms of the FID and IS metrics.