Counterfactual Generative Networks

1 INTRODUCTION . Deep neural networks ( DNNs ) are the main building blocks of many state-of-the-art machine learning systems that address diverse tasks such as image classification ( He et al. , 2016 ) , natural language processing ( Brown et al. , 2020 ) , and autonomous driving ( Ohn-Bar et al. , 2020 ) . Despite the considerable successes of DNNs , they still struggle in many situations , e.g. , classifying images perturbed by an adversary ( Szegedy et al. , 2013 ) , or failing to recognize known objects in unfamiliar contexts ( Rosenfeld et al. , 2018 ) or from unseen poses ( Alcorn et al. , 2019 ) . Many of these failures can be attributed to dataset biases ( Torralba & Efros , 2011 ) or shortcut learning ( Geirhos et al. , 2020 ) . The DNN learns the simplest correlations and tends to ignore more complex ones . This characteristic becomes problematic when the simple correlation is spurious , i.e. , not present during inference . The motivational example of ( Beery et al. , 2018 ) considers the setting of a DNN that is trained to recognize cows in images . A real-world dataset will typically depict cows on green pastures in most images . The most straightforward correlation a classifier can learn to predict the label ” cow ” is hence the connection to a green , grass-textured background . Generally , this is not a problem during inference as long as the test data follows the same distribution . However , if we provide the classifier an image depicting a purple cow on the moon , the classifier should still confidently assign the label ” cow. ” Thus , if we want to achieve robust generalization beyond the training data , we need to disentangle possibly spurious correlations from causal relationships . Distinguishing between spurious and causal correlations is one of the core questions in causality research ( Pearl , 2009 ; Peters et al. , 2017 ; Schölkopf , 2019 ) . One central concept in causality is the assumption of independent mechanisms ( IM ) , which states that a causal generative process is composed of autonomous modules that do not influence each other . In the context of image classification ( e.g. , on ImageNet ) , we can interpret the generation of an image as a causal process ( Kocaoglu et al. , 2018 ; Goyal et al. , 2019 ; Suter et al. , 2019 ) . We decompose this process into separate IMs , each controlling one factor of variation ( FoV ) of the image . Concretely , we consider three IMs : one generates the object ’ s shape , the second generates the object ’ s texture , and the third generates the background . With access to these IMs , we can produce counterfactual images , i.e. , images of unseen combinations of FoVs . We can then train an ensemble of invariant classifiers on the generated coun- terfactual images , such that every classifier relies on only a single one of those factors . The main idea is illustrated in Figure 1 . By exploiting concepts from causality , this paper links two previously distinct domains : disentangled generative models and robust classification . This allows us to scale our experiments beyond small toy datasets typically used in either domain . The main contributions of our work are as follows : • We present an approach for generating high-quality counterfactual images with direct control over shape , texture , and background . Supervision is only provided by the class label and certain inductive biases we impose on the learning problem . • We demonstrate the usefulness of the generated counterfactual images for the downstream task of image classification on both MNIST and ImageNet . Our model improves the classifier ’ s out-of-domain robustness while only marginally degrading its overall accuracy . • We show that our generative model demonstrates interesting emerging properties , such as generating high-quality binary object masks and unsupervised image inpainting . We release our code at https : //github.com/autonomousvision/counterfactual generative networks 2 STRUCTURAL CAUSAL MODELS FOR IMAGE GENERATION . In this section , we first introduce our ideas on a conceptual level . Concretely , we form a connection between the areas of causality , disentangled representation learning , and invariant classifiers , and highlight that domain randomization ( Tobin et al. , 2017 ) is a particular instance of these ideas . In section 3 , we will then formulate a concrete model that implements these ideas for image classification . Our goals are two-fold : ( i ) We aim at generating counterfactual images with previously unseen combinations like a cat with elephant texture or the proverbial ” bull in a china shop. ” ( ii ) We utilize these images to train a classifier invariant to chosen factors of variation . In the following , we first formalize the problem setting we address . Second , we describe how we can address this setting by structuring a generator network as a structural causal model ( SCM ) . Third , we show how to use the SCM for training robust classifiers . 2.1 PROBLEM SETTING . Consider a dataset comprised of ( high-dimensional ) observations x ( e.g . images ) , and corresponding labels y ( e.g . classes ) . A common assumption is that each x can be described by lower-dimensional , semantically meaningful factors of variation z ( e.g. , color or shape of objects in the image ) . If we can disentangle these factors , we are able to control their influence on the classifier ’ s decision . In the disentanglement literature , the factors are often assumed to be statistically independent , i.e. , z is distributed according to p ( z ) = Πni=1 ( zi ) ( Locatello et al. , 2018 ) . However , assuming independence is problematic because certain factors might be correlated in the training data , or the combination of some factors may not exist . Consider the colored MNIST dataset ( Kim et al. , 2019 ) , where both the digit ’ s color and its shape correspond to the label . The simplest decision rule a classifier can learn is to count the number of pixels of a specific color value ; no notion of the digit ’ s shape is required . This kind of correlation is not limited to constructed datasets – classifiers trained on ImageNet ( Deng et al. , 2009 ) strongly rely on texture for classification , significantly more than on the object ’ s shape ( Geirhos et al. , 2018 ) . While texture or color is a powerful classification cue , we do not want the classifier to ignore shape information completely . Therefore , we advocate a generative viewpoint . However , simply training , e.g. , a disentangled VAE ( Higgins et al. , 2017 ) on this dataset , does not allow for generating data points of unseen combinations – the VAE can not generate green zeros if all zeros in the training data are red ( see Appendix A for a visualization ) . We therefore propose a novel generative model which enables full control over several FoVs relevant for classification . We then train a classifier on these images while randomizing all factors but one . The classifier focuses on the non-randomized factor and becomes invariant wrt . the randomized ones . 2.2 STRUCTURAL CAUSAL MODELS . In representation learning , it is commonly assumed that a potentially complex function f generates images from a small set of high-level semantic variables ( e.g. , position or color of objects ) ( Bengio et al. , 2013 ) . Most previous work ( Goyal et al. , 2019 ; Suter et al. , 2019 ) imposes no restrictions on f , i.e. , a neural network is trained to map directly from a low-dimensional latent space to images . We follow the argument that rather than training a monolithic network to map from a latent space to images , the mapping should be decomposed into several functions . Each of these functions is autonomous , e.g. , we can modify the background of an image while keeping all other aspects of the image unchanged . These demands coincide with the concept of structural causal models ( SCMs ) and independent mechanisms ( IMs ) . An SCM C is defined as a collection of d ( structural ) assignments Sj : = fj ( PAj , Uj ) , j = 1 , . . . , d ( 1 ) where each random variable Sj is a function of its parents PAj ⊆ { S1 , . . . , Sd } \ { Sj } and a noise variable Uj . The noise variables U1 , . . . , Ud are jointly independent . The functions fi are independent mechanisms , intervening on one mechanism fj does not change the other mechanisms { f1 , . . . , fd } \ { fj } . The SCM C defines a unique distribution over the variables S = ( S1 , . . . , Sd ) which is referred to as the entailed distribution PCS . If one or more structural assignments are replaced , i.e. , Sk : = f̃ ( P̃Ak , Ũk ) , this is called an intervention . We consider the case of atomic interventions , when f̃ ( P̃Ak , Ũk ) puts a point mass on a real value a . The entailed distribution then changes to the intervention distribution PC ; do ( Sk : =a ) S , where the do refers to the intervention . A thorough review of these concepts can be found in ( Peters et al. , 2017 ) . Our goal is to represent the image generation process with an SCM . If we learn a sensible set of IMs , we can intervene on a subset of them and generate interventional images xIV . These images were not part of the training data x as they are generated from the intervention distribution PC ; do ( Sk : =a ) S . To generate a set of counterfactual images xCF , we fix the noise u and randomly draw a , hence answering counterfactual questions such as ” How would this image look like with a different background ? ” . In our case , a corresponds to a class label that we provide as input , denoted as yCF in the following . 2.3 TRAINING AN INVARIANT CLASSIFIER . To train an invariant classifier , we generate counterfactual images xCF , by intervening on all fj simultaneously . Towards this goal , we draw labels uniformly from the set of possible labels Y for each fj , i.e. , each IM is conditioned on a different label . We denote the domain of images generated by all possible label permutations as XCF . The task of the invariant classifier r : XCF → YCF , k is then to predict the label yCF , k that was provided to one specific IM fk – rendering r invariant wrt . all other IMs . This type of invariance is reminiscent of the idea of domain randomization ( Tobin et al. , 2017 ) . Here , the goal is to solve a robotics task while randomizing all task-irrelevant attributes . The randomization improves the performance of the learned policy in the real-world . In domain randomization , we commonly assume access to the true generative model ( the simulator ) . This assumption is not feasible if we do not have access to this model . Similar connections of causality and data augmentation have been made in ( Ilse et al. , 2020 ) . It is also possible to train on interventional images xIV , i.e. , generating a single image per sampled noise vector . Empirically , we find that counterfactual images improve performance over interventional ones . We hypothesize that counterfactuals provide a more stable signal .

Summary: This paper proposes a new generative model that generate images from 3 seperate aspects: foreground masks (shapes), forground texture, and backgrounds. Then they convexly mix these 3 aspects into one image. By doing so, they can vary each aspect individually without changing other aspects, enabling the model to generate counterfactual images. For example, we can generate a cat shape with telephene texture and sea background, which would not exist in natural images. They show that in several colorful MNISTs datasets their methods can generate new combinations of images. In ImageNet, by using the pre-trained BigNet GAN as backbones and pretrained U-Net for foreground object masks, they can distill the knowledge in BigNet into these 3 seperate aspects and generate counterfactual natural images.