Counterfactual Generative Networks

1 INTRODUCTION . Deep neural networks ( DNNs ) are the main building blocks of many state-of-the-art machine learning systems that address diverse tasks such as image classification ( He et al. , 2016 ) , natural language processing ( Brown et al. , 2020 ) , and autonomous driving ( Ohn-Bar et al. , 2020 ) . Despite the considerable successes of DNNs , they still struggle in many situations , e.g. , classifying images perturbed by an adversary ( Szegedy et al. , 2013 ) , or failing to recognize known objects in unfamiliar contexts ( Rosenfeld et al. , 2018 ) or from unseen poses ( Alcorn et al. , 2019 ) . Many of these failures can be attributed to dataset biases ( Torralba & Efros , 2011 ) or shortcut learning ( Geirhos et al. , 2020 ) . The DNN learns the simplest correlations and tends to ignore more complex ones . This characteristic becomes problematic when the simple correlation is spurious , i.e. , not present during inference . The motivational example of ( Beery et al. , 2018 ) considers the setting of a DNN that is trained to recognize cows in images . A real-world dataset will typically depict cows on green pastures in most images . The most straightforward correlation a classifier can learn to predict the label ” cow ” is hence the connection to a green , grass-textured background . Generally , this is not a problem during inference as long as the test data follows the same distribution . However , if we provide the classifier an image depicting a purple cow on the moon , the classifier should still confidently assign the label ” cow. ” Thus , if we want to achieve robust generalization beyond the training data , we need to disentangle possibly spurious correlations from causal relationships . Distinguishing between spurious and causal correlations is one of the core questions in causality research ( Pearl , 2009 ; Peters et al. , 2017 ; Schölkopf , 2019 ) . One central concept in causality is the assumption of independent mechanisms ( IM ) , which states that a causal generative process is composed of autonomous modules that do not influence each other . In the context of image classification ( e.g. , on ImageNet ) , we can interpret the generation of an image as a causal process ( Kocaoglu et al. , 2018 ; Goyal et al. , 2019 ; Suter et al. , 2019 ) . We decompose this process into separate IMs , each controlling one factor of variation ( FoV ) of the image . Concretely , we consider three IMs : one generates the object ’ s shape , the second generates the object ’ s texture , and the third generates the background . With access to these IMs , we can produce counterfactual images , i.e. , images of unseen combinations of FoVs . We can then train an ensemble of invariant classifiers on the generated coun- terfactual images , such that every classifier relies on only a single one of those factors . The main idea is illustrated in Figure 1 . By exploiting concepts from causality , this paper links two previously distinct domains : disentangled generative models and robust classification . This allows us to scale our experiments beyond small toy datasets typically used in either domain . The main contributions of our work are as follows : • We present an approach for generating high-quality counterfactual images with direct control over shape , texture , and background . Supervision is only provided by the class label and certain inductive biases we impose on the learning problem . • We demonstrate the usefulness of the generated counterfactual images for the downstream task of image classification on both MNIST and ImageNet . Our model improves the classifier ’ s out-of-domain robustness while only marginally degrading its overall accuracy . • We show that our generative model demonstrates interesting emerging properties , such as generating high-quality binary object masks and unsupervised image inpainting . We release our code at https : //github.com/autonomousvision/counterfactual generative networks 2 STRUCTURAL CAUSAL MODELS FOR IMAGE GENERATION . In this section , we first introduce our ideas on a conceptual level . Concretely , we form a connection between the areas of causality , disentangled representation learning , and invariant classifiers , and highlight that domain randomization ( Tobin et al. , 2017 ) is a particular instance of these ideas . In section 3 , we will then formulate a concrete model that implements these ideas for image classification . Our goals are two-fold : ( i ) We aim at generating counterfactual images with previously unseen combinations like a cat with elephant texture or the proverbial ” bull in a china shop. ” ( ii ) We utilize these images to train a classifier invariant to chosen factors of variation . In the following , we first formalize the problem setting we address . Second , we describe how we can address this setting by structuring a generator network as a structural causal model ( SCM ) . Third , we show how to use the SCM for training robust classifiers . 2.1 PROBLEM SETTING . Consider a dataset comprised of ( high-dimensional ) observations x ( e.g . images ) , and corresponding labels y ( e.g . classes ) . A common assumption is that each x can be described by lower-dimensional , semantically meaningful factors of variation z ( e.g. , color or shape of objects in the image ) . If we can disentangle these factors , we are able to control their influence on the classifier ’ s decision . In the disentanglement literature , the factors are often assumed to be statistically independent , i.e. , z is distributed according to p ( z ) = Πni=1 ( zi ) ( Locatello et al. , 2018 ) . However , assuming independence is problematic because certain factors might be correlated in the training data , or the combination of some factors may not exist . Consider the colored MNIST dataset ( Kim et al. , 2019 ) , where both the digit ’ s color and its shape correspond to the label . The simplest decision rule a classifier can learn is to count the number of pixels of a specific color value ; no notion of the digit ’ s shape is required . This kind of correlation is not limited to constructed datasets – classifiers trained on ImageNet ( Deng et al. , 2009 ) strongly rely on texture for classification , significantly more than on the object ’ s shape ( Geirhos et al. , 2018 ) . While texture or color is a powerful classification cue , we do not want the classifier to ignore shape information completely . Therefore , we advocate a generative viewpoint . However , simply training , e.g. , a disentangled VAE ( Higgins et al. , 2017 ) on this dataset , does not allow for generating data points of unseen combinations – the VAE can not generate green zeros if all zeros in the training data are red ( see Appendix A for a visualization ) . We therefore propose a novel generative model which enables full control over several FoVs relevant for classification . We then train a classifier on these images while randomizing all factors but one . The classifier focuses on the non-randomized factor and becomes invariant wrt . the randomized ones . 2.2 STRUCTURAL CAUSAL MODELS . In representation learning , it is commonly assumed that a potentially complex function f generates images from a small set of high-level semantic variables ( e.g. , position or color of objects ) ( Bengio et al. , 2013 ) . Most previous work ( Goyal et al. , 2019 ; Suter et al. , 2019 ) imposes no restrictions on f , i.e. , a neural network is trained to map directly from a low-dimensional latent space to images . We follow the argument that rather than training a monolithic network to map from a latent space to images , the mapping should be decomposed into several functions . Each of these functions is autonomous , e.g. , we can modify the background of an image while keeping all other aspects of the image unchanged . These demands coincide with the concept of structural causal models ( SCMs ) and independent mechanisms ( IMs ) . An SCM C is defined as a collection of d ( structural ) assignments Sj : = fj ( PAj , Uj ) , j = 1 , . . . , d ( 1 ) where each random variable Sj is a function of its parents PAj ⊆ { S1 , . . . , Sd } \ { Sj } and a noise variable Uj . The noise variables U1 , . . . , Ud are jointly independent . The functions fi are independent mechanisms , intervening on one mechanism fj does not change the other mechanisms { f1 , . . . , fd } \ { fj } . The SCM C defines a unique distribution over the variables S = ( S1 , . . . , Sd ) which is referred to as the entailed distribution PCS . If one or more structural assignments are replaced , i.e. , Sk : = f̃ ( P̃Ak , Ũk ) , this is called an intervention . We consider the case of atomic interventions , when f̃ ( P̃Ak , Ũk ) puts a point mass on a real value a . The entailed distribution then changes to the intervention distribution PC ; do ( Sk : =a ) S , where the do refers to the intervention . A thorough review of these concepts can be found in ( Peters et al. , 2017 ) . Our goal is to represent the image generation process with an SCM . If we learn a sensible set of IMs , we can intervene on a subset of them and generate interventional images xIV . These images were not part of the training data x as they are generated from the intervention distribution PC ; do ( Sk : =a ) S . To generate a set of counterfactual images xCF , we fix the noise u and randomly draw a , hence answering counterfactual questions such as ” How would this image look like with a different background ? ” . In our case , a corresponds to a class label that we provide as input , denoted as yCF in the following . 2.3 TRAINING AN INVARIANT CLASSIFIER . To train an invariant classifier , we generate counterfactual images xCF , by intervening on all fj simultaneously . Towards this goal , we draw labels uniformly from the set of possible labels Y for each fj , i.e. , each IM is conditioned on a different label . We denote the domain of images generated by all possible label permutations as XCF . The task of the invariant classifier r : XCF → YCF , k is then to predict the label yCF , k that was provided to one specific IM fk – rendering r invariant wrt . all other IMs . This type of invariance is reminiscent of the idea of domain randomization ( Tobin et al. , 2017 ) . Here , the goal is to solve a robotics task while randomizing all task-irrelevant attributes . The randomization improves the performance of the learned policy in the real-world . In domain randomization , we commonly assume access to the true generative model ( the simulator ) . This assumption is not feasible if we do not have access to this model . Similar connections of causality and data augmentation have been made in ( Ilse et al. , 2020 ) . It is also possible to train on interventional images xIV , i.e. , generating a single image per sampled noise vector . Empirically , we find that counterfactual images improve performance over interventional ones . We hypothesize that counterfactuals provide a more stable signal .

Summary: The main idea of the paper, i.e., using independent causal mechanisms to generate interventional images, has already been explored by Kocaoglu et al. in Causalgan: Learning causal implicit generative models with adversarial training, ICLR'18. Same as here, the authors there also "view image generation as a causal process" and "structure a generator network as a structural causal model (SCM)" and use a conditional gan to generate the image from the labels. The generation used here based on three variables, i.e., shape, texture and background seem to be a special case. Therefore, the authors should definitely cite this work.