Designing Counterfactual Generators using Deep Model Inversion

1 Introduction . With the growing need for deploying deep black-box models into critical decision-making , there is an increased emphasis on explainability methods that can reveal intricate relationships between data signatures ( e.g. , image features ) and predictions . In this context , the so-called counterfactual ( CF ) explanations [ 1 ] that synthesize small , interpretable changes to a given image while producing desired changes in model predictions to support user-specified hypotheses ( e.g. , progressive change in predictions ) have become popular . Though counterfactual explanations provide more flexibility over conventional techniques , such as feature importance estimation [ 2 , 3 , 4 , 5 , 6 ] , by exploring the vicinity of a query image , an important requirement to produce meaningful counterfactuals is to produce discernible local perturbations ( for easy interpretability ) while being realistic ( close to the underlying data manifold ) . Consequently , existing approaches rely extensively on pre-trained generative models to synthesize plausible counterfactuals [ 1 , 7 , 8 , 9 , 10 ] . By design , this ultimately restricts their utility to scenarios where one can not access the training data or pre-trained generative models , for example , due to privacy requirements commonly encountered in many practical applications . In this paper , we focus on the problem where we have access only to trained deep classifiers and not the actual training data or generative models . Synthesizing images from the underlying data distribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . by inverting a deep model , while not requiring access to training data , is a well investigated topic of research . For e.g. , Deep Dream [ 11 ] synthesizes class-conditioned images by manipulating a noisy image directly in the space of pixels ( or more formally Image Space Optimization ( ISO ) ) constrained by image priors such as total variation [ 12 ] to regularize this ill-posed inversion . However , Deep Dream is known to produce images that look unrealistic , often very different from the training images , thus limiting their use in practice . Consequently , Yin et al. , proposed DeepInversion [ 13 ] that performs image synthesis in the latent space of a pre-trained classifier ( Latent Space Optimization ( LSO ) ) and leverages layer-specific statistics ( from batchnorm [ 14 ] ) to constrain the images to be consistent with the training data distribution . This was showed to produce higher-quality images , particularly in the context of performing knowledge distillation [ 15 ] using the synthesized images . Proposed Work . In contrast , this work aims to develop a deep model inversion approach that generates counterfactual explanations by exploring the vicinity of a given query image , instead of synthesizing an arbitrary realization from the entire image distribution . As illustrated in the example in Figure 1 , existing deep inversion methods are ineffective when natively adopted for counterfactual generation . Due to use of weak priors , and the severely ill-posed nature of the problem , it introduces irrelevant pixel manipulations that easily satisfy the desired change in prediction . Hence , we propose DISC ( Deep Inversion for Synthesizing Counterfactuals ) that improves upon conventional deep model inversion by utilizing : ( i ) stronger image priors through the use of deep image priors [ 16 ] ( DIP ) and implicit neural representations [ 17 ] ( INR ) ; ( ii ) a novel manifold consistency objective that ensures the counterfactual remains close to the underlying manifold ; and ( iii ) a progressive optimization strategy to effectively introduce discernible , yet meaningful , changes to the query image . From Figure 1 , we find that our approach produces meaningful image manipulations , in order to change the prediction to the smiling class , while other deep inversion strategies can not . Using empirical studies , we show that DISC consistently produces visually meaningful explanations , and that the counterfactuals from DISC are effective at learning model decision boundaries and are robust to unknown test-time corruptions . Our Contributions . 1 . A general framework to produce counterfactuals on-the-fly using deep model inversion ; 2 . Novel objectives to ensure consistency to the data manifold . We explore two different strategies based on direct error prediction [ 18 , 19 ] and deterministic uncertainty estimation [ 20 ] ; 3 . A progressive optimization strategy to introduce discernible changes to a given query image , while satisfying the manifold consistency requirement ; 4 . A classifier discrepancy metric to evaluate the quality of counterfactuals ; 5 . Empirical studies using natural image and medical image classifiers to demonstrate the effectiveness of DISC over a variety of baselines and ablations . 2 Related Work . Image Synthesis from Classifiers . Inverting a pre-trained deep model is a popular strategy for image synthesis in scenarios where there is no access to the underlying training data . While deep model inversion-based methods such as Deep Dream [ 11 ] and DeepInversion [ 13 ] have been successful in generating class-conditioned images , there have been other extensions to such approaches . For example , Dosovitskiy et al . [ 21 ] proposed to invert representations of a pre-trained CNN to obtain insights about what a deep classifier has learned . On similar lines , Mahendran et al . [ 12 , 22 ] addressed the problem of pre-image recovery , which in essence attempts to recover an arbitrarily encoded representation ( in the latent space of a classifier ) to a realization on the ( unknown ) image manifold and to enable model diagnosis . Ulyanov et al . [ 16 ] improved upon this ill-posed inversion by utilizing a strong image prior in the form of deep image priors ( DIP ) . They also explored the related problem of activation-maximization [ 16 ] , where the goal is to generate an image that maximizes the activation of a given output neuron in a pre-trained classifier , and demonstrated the effectiveness of DIP . Despite the effectiveness of the deep model inversion methods for image synthesis , we find that such methods can not be natively adopted for CF generation and are insufficient for producing meaningful pixel manipulations to a given query image . Counterfactual Generation . Existing methods extensively rely on generative models to provide counterfactuals that explain the decisions of a black-box model . Examples including CounteRGAN [ 23 ] , Counterfactual Generative Networks ( CGNs ) [ 24 ] and the methods reported in [ 7 , 8 ] , have clearly demonstrated the use of generative models in synthesizing high-quality CFs for any user-specified hypothesis on the predictions . However , this requirement of access to training data or generative models can be infeasible in practical scenarios , for e.g. , restrictions arising due to privacy requirements . In contrast , our approach formulates the problem of counterfactual generation using deep model inversion , and can produce meaningful counterfactuals using only the trained classifier . 3 Proposed Approach . In this section , we describe our approach for counterfactual generation that improves upon deep model inversion , and introduce a new classifier discrepancy metric for evaluating CFs . There are four key components that are critical to designing classifier-based counterfactual generators : ( i ) choice of metric for semantics preservation ; ( ii ) choice of image priors to regularize image synthesis ; ( iii ) manifold consistency to ensure that the synthesized counterfactual lies close to the true data manifold ; and ( iv ) progressive optimization strategy to introduce gradual meaningful changes to a query image . Figures 2a and 2b provide overview of DISC implemented with deep image prior and INR-based for regularizing the counterfactual optimization process . In its simplest form , for a given query x , a counterfactual explanation can be obtained as follows : arg min x̄ d ( x̄ , x ) s.t . F ( x̄ ) = ȳ , x̄ ∈M ( x ) , ( 1 ) where x̄ = C ( x ) is a counterfactual explanation for x , F is a pre-trained classifier model , M denotes the data manifold and ȳ = y + δ is the desired change in the prediction . The metric d ( . , . ) measures the discrepancy between the query image and the counterfactual ( i.e. , semantics preservation ) . 3.1 Choice of Metric for Semantics Preservation . We can measure the discrepancy between a query and its CF explanation , d ( x̄ , x ) , in the pixel space or in the latent space of the classifier . We now describe the high-level formulation of deep model inversion-based CF generation . a . Image Space Optimization ( ISO ) . ISO for counterfactual generation involves the ill-posed optimization of an input x̄ directly in the pixel space to generate an image semantically similar to the query x , while being consistent with a user-hypothesis on the prediction , ȳ . Strategies such as Deep Dream [ 11 ] perform ISO to synthesize artistic variations of images . Mathematically , arg min x̄ d ( x̄ , x ) +R ( x̄ ) s.t . F ( x̄ ) = ȳ . ( 2 ) whereR ( x̄ ) is a suitable image prior to regularize the optimization . b . Latent Space Optimization ( LSO ) . LSO refers to the ill-posed problem of inverting an arbitrarily encoded representation from the latent space of a deep classifier to a realization on the ( unknown ) image manifold . Let Ψl ( . ) denote the lth differentiable layer of the deep classifier . Then , counterfactual generation using LSO can be mathematically formulated as arg min x̄ ∑ l d ( Ψl ( x̄ ) , Ψl ( x ) ) +R ( x̄ ) s.t . F ( x̄ ) = ȳ . ( 3 ) Approaches such as DeepInversion [ 13 ] perform LSO for conditional image synthesis ( though with distribution-level comparison instead of our sample-level comparison ) and achieve visually superior images when compared to ISO approaches . 3.2 Choice of Image Priors . As observed from ( 2 ) and ( 3 ) , the choice of the regularizer or image prior R ( . ) is central towards regularizing and tractably solve this challenging inverse problem . A variety of image priors have been proposed in the literature , and we investigate the following in this work . a . Total Variation + ` 2 . Total variation [ 22 ] ( TV ) is a popular regularizer that encourages images to contain piece-wise constant patches while the ` 2 norm regularizes the range and energy of the image to remain within a given interval . The TV norm and the ` 2 norm are given by : RTV ( x̄ ) = ∑ i , j √ ( x̄i , j+1 − x̄i , j ) 2 + ( x̄i+1 , j − x̄i , j ) 2 ; R ` 2 ( x̄ ) = √∑ i , j ‖x̄i , j‖2 ( 4 ) b . Deep Image Priors ( DIP ) . A Deep Image Prior ( DIP ) [ 16 ] leverages the structure of an untrained , carefully tailored convolutional neural network ( e.g. , U-Net [ 25 ] ) to generate images and solve a variety of ill-posed restoration tasks in computer vision . DIP has been found to produce high-quality reconstructions , based on the key insight that the structure of the network itself can act as a regularizer . Consequently , the synthesized image is re-parameterized in terms of the weights θ of the prior model fθ i.e x̄ = fθ ( z ) . c. Implicit Neural Representations ( INR ) . We also considered an alternative approach based on INR , which provide a cheap and convenient way to learn a continuous mapping from the image coordinates to the pixel values ( RGB ) . While they have been found to be effective for image/volume rendering and designing generative models [ 26 ] , we explore their use in deep model inversion ( details in appendix ) . We build upon two key results to design our INR-based counterfactual generators : ( i ) Fourier mapping : Based on NTK ( neural tangent kernel ) theory , [ 27 ] showed that using Fourier mapping can recover high-frequency features in low-dimensional coordinate-based image reconstruction . Hence , we use a Fourier feature mapping z to featurize 2−D input coordinates v ∈ [ 0 , 1 ] 2 before passing them through a coordinate-based MLP : z ( v ) = [ a1 cos ( 2πb T 1 v ) , a1 sin ( 2πb T 1 v ) , · · · ] . Using a set of randomly chosen sinusoids , this maps the input points to the surface of a highdimensional hypersphere . Training the MLP network on these embedded points corresponds to kernel regression with the stationary composed NTK hNTK ◦ hz , where hNTK denotes the neural tangent kernel corresponding to the MLP ; ( ii ) SIREN activation : In [ 17 ] , Sitzmann et al . showed that periodic activation functions are better suited for recovering natural images and their derivatives , when compared to standard activation functions . More specifically , SIREN uses a sinusoid activation Φ ( x ) = sin ( Wx + b ) . We find that using both a Fourier mapping coupled with SIREN activation leads to a very strong image prior .

Summary: Summary: The paper proposed a novel approach to achieve deep model inversion for a pre-trained deep classifier using the classifier alone, without having access to the training dataset or generative models. Given a query image, the proposed deep model inversion generates a counterfactual explanation that is realistic looking and produces the desired classification outcome.    Earlier work on image synthesis by inverting deep classifiers either creates unrealistic images or creates perturbation of query image that successfully changes the classification decision but resulting pixel-level manipulations are not interpretable/meaningful (similar to an adversarial attack).   The proposed approach has four essential components: 1.	Counterfactual image is synthesized using an image prior. The authors experimented with an un-trained UNET-based model as a deep image prior (DIP) and Fourier mapping with SIREN activation as implicit neural representation (INR) for mapping noise to the counterfactual image. 2.	To ensure high similarity between counterfactual and query images, the authors used ISO to minimize discrepancies over the entire image and LSO to minimize differences between embeddings extracted from different layers of the deep classifier. 3.	To ensure that the generated counterfactual image is realistic and lies on the data manifold, the proposed model uses a manifold consistency (MC) loss. The MC-loss is quantified using an auxiliary loss predictor function in DEP or the radial basis kernel similarity in DUQ. Both methods provide ways to capture epistemic uncertainty due to out-of-distribution (OOD) samples. 4.	The model uses a functional consistency loss to ensure the counterfactual image produces a desired outcome from the deep classifier. Finally, the model uses progressive optimization to learn the image prior layer-by-layer.