Counterfactual Generative Networks
Authors: Axel Sauer
Andreas Geiger
Venue: ICLR 2021 Poster (2021)
Keywords: Causality
Counterfactuals
Generative Models
Robustness
Image Classification
Data Augmentation

Abstract: Neural networks are prone to learning shortcuts -- they often model simple correlations, ignoring more complex ones that potentially generalize better. Prior works on image classification show that instead of learning a connection to object shape, deep classifiers tend to exploit spurious correlations with low-level texture or the background for solving the classification task. In this work, we take a step towards more robust and interpretable classifiers that explicitly expose the task's causal structure. Building on current advances in deep generative modeling, we propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision. By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background; hence, they allow for generating counterfactual images. We demonstrate the ability of our model to generate such images on MNIST and ImageNet. Further, we show that the counterfactual images can improve out-of-distribution robustness with a marginal drop in performance on the original classification task, despite being synthetic. Lastly, our generative model can be trained efficiently on a single GPU, exploiting common pre-trained models as inductive biases.

Published as a conference paper at ICLR 2021
COUNTERFACTUAL GENERATIVE NETWORKS
Axel Sauer & Andreas Geiger1.2 Autonomous Vision Group
Max Planck Institute for Intelligent Systems, Tübingen University of Tübingen (fi (sitneme.. lastname (esttamm)@ttue.... www.teeee....e....eeee.......... wwg.ppp....... de
ABSTRACT
Neural networks are prone to learning shortcuts they often model simple cor- relations, ignoring more complex ones that potentially generalize better. Prior works on image classification show that instead of learning connection to object shape, deep classifiers tend to exploit spurious correlations with low-level tex- ture the background for solving the classification task. In this work, take a step towards more robust and interpretable classifiers that explicitly expose the task's causal structure. Building current advances in deep generative modeling, we propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision. By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background; hence, they allow for generating counterfactual images. We demon- strate the ability of our model to generate such images on MNIST and ImageNet. Further, we show that the counterfactual images can improve distiiiiiitttniionnn robustness with marginal drop in performance on the original classification task, despite being synthetic. Lastly, our generative model can be trained efficiently on single GPU, exploiting common pre-trained models as inductive biases.
INTRODUCTION
Deep neural networks (DNNs) the main building blocks of many state-of-the-ar machine learn- ing systems that address diverse tasks such as image classification (He et al. 2016), natural language
processing (Brown al. 2020), and autonomous driving (Ohn- Bar et al. (020). Despite the consid- erable successes of DNNs, they still struggle in many situations, e.g., classifying images perturbed by an adversary (Szegedy al. 2013), or failing recognize known objects in unfamiliar contexts (Rosenfeld et al. 2018) or from unseen poses (Alcorn et al. 2019).
shortcut learn- ing (Geirhos et al. 2020). The DNN learns the simplest correlations and tends to ignore more com- plex ones. This characteristic becomes problematic when the simple correlation spurious, i.c., not present during inference. The motivational example of (Beery et al. 2018) considers the setting of a DNN that is trained to recognize cows images. real-world dataset will typically depict cows on green pastures in most images. The most straightforward correlation a classifier can learn to predict the label "cow" is hence the connection to a green, grass-textured background. Generally, this is not problem during inference as long as the test data follows the same distribution. However, if we provide the classifier an image depicting a purple cow on the moon, the classifier should still confidently assign the label "cow." Thus, if we want to achieve robust generalization beyond the training data, we need to disentangle possibly spurious correlations from causal relationships.
Many of these failures can be attributed dataset biases (Torralba & Efros 2011
Distinguishing between spurious and causal correlations is one of the core questions in causality research (Pearl 2209) [Pteers et al. [2017] [Schökkopf] 2019)). One central concept in causality the assumption of independent mechanisms (IM), which states that causal generative process is com- posed of autonomous modules that do not influence each other. In the context of image classification (c.g., on ImageNet), we can interpret the generation an image as a causal process (Kocaoglu al. 2018 Goyal et al.) 2019 Suter et al. 2019). We decompose this process into separate IMs, each controlling one factor of variation (FoV) of the image. Concretely, we consider three IMs: one generates the object's 's shape, the second generates the object's 's texture, and the third generates the background. With access these IMs, we can produce counterfactual images, i.e., images of unseen combinations of Fs We can then train an ensemble of invariant classifiers on the generated coun-

---

Published as a conference paper at ICLR 2021
Real Image
Counterfactual Image
Claseller::
Classifier: A strawberry
An ostrcc.
Invariant Clavsifirr Ensemble:
Invariant Classifeer Ensemble:
Asoocci shape
An estrich shape
strwhbrry ona diving background
on oxtrich background.
Out-of-Domain (OOD) Classification. A classifier focuses all factors of variation (FoV) in an image. For OOD data, this can be problematic: FoV might be spurious correla- tion, hence, impairing the classifier's performance. An ensemble, e.g., a classifier with common backbone and multiple heads, each head invariant to all but one FoV, increases OOD robustness.
Figure I:
terfactual images, such that every classifier relies on only a single one of those factors. The main idea is illustrated in Figure[] By exploiting concepts from causality, this paper links two previously distinct domains: disentangled generative models and robust classification. This allows us to scale our experiments beyond small toy datasets typically used in either domain. The main contributions of our work are as follows:
We present an approach for generating high-quality counterfactual images with direct con- trol over shape, texture, and background. Supervision is only provided by the class label and certain inductive biases we impose on the learning problem.
We demonstrate the usefulness of the generated counterfactual images for the downstream task of image classification on both MNIST and ImageNet. Our model improves the clas- sifier's out-of-domain robustness while only marginally degrading its overall accuracy.
We show that generative model demonstrates interesting emerging properties, such as generating high-quality binary object masks and unsupervised image inpainting.
We release code at ahwps///ttlaooooooooooooooooooot
STRUCTURAL CAUSAL MODELS FOR IMAGE GENERATION
In this section, we first introduce ideas conceptual level. Concretely, we form connection between the areas of causality, disentangled representation learning, and invariant classifiers, and highlight that domain randomization (Tobin al. 2017) is a particular instance of these ideas. In section 3
we will then formulate a concrete model that implements these ideas for image classifica- tion. Our goals
two-fold: (i) We aim generating counterfactual images with previously unseen combinations like a cat with elephant texture or the proverbial "bull in a china shop." (ii) We utilize these images to train classifier invariant chosen factors of variation.
In the following, we first formalize the problem setting we address. Second, we describe how we can address this setting by structuring a generator network as a structural causal model (SCM). Third, we show how to use the SCM for training robust classifiers.
2.1 PROBLEM SETTING
Consider dataset comprised of (high- dimensional) observations x (c.g. images), and corresponding labels (c.g. classes). A common assumption is that each x can be described by lower-dimensional, semantically meaningful factors of variation z (c.g., color or shape of objects in the image). If we disentangle these factors, the disentanglement literature, the factors are often assumed to be statistically independent, i.e., z is distributed according p(z) m7 (zz) (Locatello et al. 2018). However, assuming independence is problematic because certain factors might be correlated in the training data, or the combination of some factors may not exist. Consider the colored MNIST dataset (Kim al. 2019), where both the digit's color and its shape correspond to the label. The simplest decision rule classifier can learn count the number of pixels of specific color value; no notion of the digit's 's shape is required. This kind of correlation is not limited to constructed datasets classifiers trained on ImageNet (Deng al. (209)) strongly rely on texture for classification, significantly more than on the object's ''s shape (Geirhos et al. 2018). While texture or color is powerful classification cue, we do not want the
able to control their influence
the classifier's decision. In

---

Published as a conference paper at ICLR 2021
classifier to ignore shape information completely. Therefore, we advocate a generative viewpoint. However, simply training, e-g., a disentangled VAE (Higgins al. 2017) on this dataset, does not allow for generating data points of unseen combinations the VAE cannot generate green zeros if all zeros the training data are red (see Appendix Aor a visualization). We therefore propose a novel generative model which enables full control over several FoVs relevant for classification. We then train a classifier on these images while randomizing all factors but one. The classifier focuses on the non non-randomized factor and becomes invariant wrt. the randomized ones.
2.2 STRUCTURAL CAUSAL MODELS
In representation learning, it is commonly assumed that a potentially complex function f generates images from small set of high-level semantic variables (e.g.. position color of objects) (Bengio ct al. (013). Most previous work (Goyal et al. 2019;; /utee et al.) 2019) imposes no restrictions on J. i.e., neural network is trained to map directly from low-dimensional latent space images. We follow the argument that rather than training monolithic network to map from a latent space to images, the mapping should be decomposed into several functions. Each of these functions is autonomous, e.g., we can modify the background of an image while keeping all other aspects of the image unchanged. These demands coincide with the concept of structural causal models (SCMs) and independent mechanisms (IMs). An SCM E is defined collection of d (structural) assignments
(1) where each random variable S, is a function its parents PA, S (S,...., Sa) (S and a noise variable Uz. The noise variables U1,...,U, Us are jointly independent. The functions Si are independent mechanisms, intervening on one mechanism fy does not change the other mechanisms {fi..... fu}' { The SCM E defines unique distribution over the variables S (S1, (Si,..., which is referred to the entailed distribution PS.. If one more structural assignments are replaced, i.e., Sy = f (PAAk Uk), this is called an intervention. We consider the case of atomic interventions, when fPPA,, U&) puts a point mass on a real value a. The entailed distribution then changes to the intervention distribution Pg (ido(S,mma) where the do refers to the intervention. A thorough review of these concepts can be found in (Peters et al. 2017). Our goal is to represent the image generation process with SCM. If we learn sensible set of IMs, we can intervene on subset of them and generate interventional images Xạv. These images were not part of the
Sy IJ (PAj, Ug); j=1,..., d
training data x as they are generated from the intervention distribution Pg ( To generate a set of counterfactual images XCF, we fix the noise u and randomly draw a, hence answering counterfactual questions such as "How would this image look like with different background?". In our case, a corresponds to a class label that we provide as input, denoted as yCF in the following.
2.3 TRAINING AN INVARIANT CLASSIFIER
To train an invariant classifier, we generate counterfactual images XCF, by intervening on all fj simultancously. Towards this goal, we draw labels uniformly from the set of possible labels y for each fj. i.e., each IM is conditioned on different label. We denote the domain of images generated by all possible label permutations as Xex. The task of the invariant classifier r Xex - Vera is then predict the label YCF, that provided wrt. all other IMs. This type of invariance is reminiscent of the idea of domain randomization (Tobin et al. 2017). Here, the goal is to solve robotics task while randomizing all task-inrelevant attributes. The randomization improves the performance of the learned policy in the real-world. In domain randomization, we commonly assume access to the true generative model (the simulator). This assumption is not feasible if we do not have access to this model. Similar connections of causality and data augmentati augmentation have been made in (Ilse et al. 20200). It is also possible to train on interventional images X/v, i.e., generating single image per sampled noise vector. Empirically, we find that counterfactual images improve performance over interventional ones. We hypothesize that counterfactuals provide more stable signal.
specific IM fs rendering invariant
3 COUNTERFACTUAL GENERATIVE NETWORKS
In this section, we apply our ideas outlined above to the particular problem of image classification. Our goal is to decompose the image generation process into several IMs. In image classification,
3

---

Published as a conference paper at ICLR 2021
Sslap
CON
ahage
DegGAN
frest
test
u~ ()) y~2()
Loo
Lrac
BigGAN
BigGAN
cGAN
Figure 2: Counterfactual Generative Network (CGN). Here, we illustrate the architecture used for the ImageNet experiments. The CGN is split into four mechanisms, the shape mechanism fahope. the texture mechanism ftent. the background mechanism fog. and the composer C. Components with trainable parameters are blue, components with fixed parameters are green. The primary supervision is provided by an unconstrained conditional GAN (cGAN) via the reconstruction loss Cree. The cGAN is only used for training, as indicated by the dotted lines. Fach mechanism takes as input the noise vector (sampled from spherical Gaussian) and the label y (drawn uniformly from the set of possible labels }) and minimizes its respective loss (Cshopes Lteat, and Ctg). To generate a set of counterfactual images, we sample and then independently sample y for each mechanism.
there is generally one principal object in the image. Hence, we assume three IMs for this specific task: object shape, object texture, and background. Our goal to train the generator consisting of these mechanisms in an end-to-end manner. The inherent structure of the model allows us to generate meaningful counterfactuals by construction. In the following, we describe the inductive biases we use (network architectures, losses, pre-trained models) and how to train the invariant classifier. We refer to the entire generative model using IMs as Counterfactual Generative Network (CGN).
3.1 INDEPENDENT MECHANISMS
We assume the causal structure be known, and consider three learned IMs for generating shape, texture, and background, respectively. The only difference between the MNIST variants and Ima- geNet is the background mechanism. For the MNIST variants, we can simplify the SCM to include second texture mechanism instead of dedicated background mechanism. There is no need for globally coherent background in the MNIST setting. An explicit formulation of both SCM is shown in Appendix B In both cases, the learned IMs feed into another, fixed, IM: the composer. An overview of our CGN is shown in Figure 2 All IM-specific losses are optimized jointly end-to-end. For the experiments BigGAN-deep-256 (Brock et al.] 2018), the current state-of-the-art for conditional image genera- tion. BigGAN has been trained single monolithic function; hence, it cannot generate images of only texture or only background, since these would be outside of the training domain.
ImageNet.
initialize each IM backbone with weights from pre- trainrd
Composition Mechanism. The function of the composer is not learned but defined analytically. For this work, we build on common assumptions from compositional image synthesis (Yang et al. (017) and deploy a simple image formation model. Given the generated masks, textures and backgrounds, we composite the image Xgen using alpha blending, denoted as C:
Xgen C'(m, f, b) = m & f (1 m) b
where m is the mask (or alpha map), f is the foreground, and b is the background. The operator @ denotes elementwise multiplication. While, in general, IMs may be stochastic (Eq. [. we did not find this to be necessary for the composer; therefore, we leave this mechanism deterministic. This fixed composition is a strong inductive bias in itself the generator needs to generate realistic images through this bottleneck. To optimize the composite image, we could use an adversarial loss between real and composite images. While applicable to simple datasets such as MNIST, we found that an adversarial approach does not scale well to more complex datasets like ImageNet. To get stronger and more stable supervisory signal, we, therefore, use an unconstrained, conditional GAN

---

Published as a conference paper at ICLR 2021
(cGAN) to generate pseudo-ground-tnute images Xgt from noise u and label y. We feed the same u and y into the IMs to generate Xgen and minimize a reconstruction loss Lxec(Xyt; (ox:xxtt:e)) Xgen). We find a combination of L1 loss and perceptual loss (Johnson et al.) (016) to work well. Note that during training, we utilize the same noise u and label to reconstruct the image generated by the cGAN. However, at inference time, we generate counterfactual images by randomizing both u and y separately per mechanism.
Shape Mechanism. We model the shape using binary mask predicted by shape IM fahape, where 0 corresponds to the background and I to the object. Effectively, this mechanism implements fore- ground segmentation. The loss is comprised of two terms: Lbinary and Lmask. Loinary is the pixel- wise binary entropy of the mask; hence, minimizing it forces the output to be close to either 0 or 1. Lmask prohibits trivial solutions, i.e., masks with all O's or I's that are outside of a defined interval (see Appendix (irendixoooora details). As we utilize a BigGAN backbone for our need to extract a binary mask from the backbone's 's output. Therefore, we add pre-trained U2-Net (Qin et al., 2020) as head on top of the BigGAN backbone. The U2-Net was trained for salient object detection on DUTS-TR (10553 images) (Wang et al.) 2017). Hence, is class agnostic; generates an object mask for a salient object in the image. While the U2-Net presents a strong bias towards binary object masks, does not fully solve the task at hand as captures non-class specific parts (e.g., parts of trees in an elephant-class picture, see Figure 5) By fine-tuning the BigGAN backbone, we learn to generate images of the relevant part with exaggerated features to increase saliency. We refer to these pre-masks m.
we
Texture Mechanism. The texture mechanism fiest is responsible for generating the foreground object's appearance, while not capturing any object shape or background cues. For MNIST, use an architectural bias - an additional layer before the final output. This layer spatially divides its input into patches and randomly rearranges them, similar simple idea does not work on ImageNet, as we want to preserve local object structure, e.g., the position of an eye. We, therefore, sample patches from the full composite image and concatenate them into a grid. We denote this patch grid as pg. The patches are sampled from regions where the mask values are highest (hence, the object is likely located). We then minimize a perceptual loss between the foreground f (the output of fteat) and the patchgrid: Liezt (f, pg). Over training, the background gradually transforms into object texture, resulting in texture maps, as shown in Figure 5. More details can be found in Appendix C
shuffled sliding puzzle. This conceptually
Background Mechanism. The background mechanism foy needs to capture the background's global structure while the object must be removed and inpainted realistically. However, we found that we cannot use standard inpainting techniques because classical methods (Barnes et al. 2009) slow down training too much, and deep learning methods (Liu et al. 2018) do not work well on synthetic data because of the domain shift. Instead, we exploit the same U2-Net as used for the shape mechanism f fahape- Again, we feed the output of the BigGAN backbone through the U2-Net with fixed weights. However, this time, we minimize the predicted saliency. Over the progress of training, this leads to the object shrinking and finally disappearing, while the model learns inpaint the object region (see Figure ( and Appendix E}. We refer to this loss as Lóg. We attribute this loss's success mainly the powerful pre-trained backbone network. BigGAN is already able to generate objects on realistic backgrounds; it only needs to unlearn the object generation.
3.2 GENERATING COUNTERFACTUALS TO TRAIN CLASSIFIERS
After training our CGN, each IM network has learned a class-conditional distribution over shapes, textures, or backgrounds. By randomizing the label input y and noise of each network, we can generate counterfactual images. The number of possible combinations is the number of classes to the power of the number of IM's. For ImageNet, this is 10003, The amount of possible images even larger since we learn distributions, i.e., we can generate a nearly unlimited variety of shapes, textures, and backgrounds, per class. We train on both real and counterfactual images. For MNIST, more counterfactual images always increase the test domain results; see the ablation study in Appendix
A.3 On Imagenet, we provide evenly sized batches of real and counterfactual images: i.c.. we use ratio of 1. A ratio below leads to inferior performance; a ratio above leads to longer training times without an increase in performance. Similar results were reported for training on BigGAN samples in Ravuri & Vinyals (2019).
5

---

Published as a conference paper at ICLR 2021
Real Imagee
Generated Counterfactual Images
0
/
1
/
/
/
2
2 J
2
2
2
7
3
3
3
3
3
3
Dolole-Cooord
Colored
Double-Coored
Figure 3: MNISTs. Left: Samples of the different MNIST variations (for brevity, show only the
first four classes). Right: Counterfactual samples generated by our CGN. Note that the CGN learned class-conditional distributions, i.e., it generates varying shapes, colors, and textures.
4 EXPERIMENTS
Our experiments aim to answer the following questions: (i) Does our approach reliably learn the disentangled IMs on datasets of different complexity? (ii) Which inductive biases necessary to achieve this? (iii) Do counterfactual images enable training invariant classifiers? We first apply our approach different versions of MNIST: colored-, double-colored- and Wildlife-MNIST (details about their generation are in Appendix A.2). The label is encoded in the digit shape, foreground color or texture, and the background color or texture, see Figure 3 Our work focuses on a setting where the spurious signal is a strong predictor of the label; hence we assume correlation strength of at least % between signal and label in our simulated environments. This assumption is in line with latest related work on visual bias (Goyal et al.) 2019) Wang et al.) 2020), which considers strong correlation to be above 95 %. We then scale our approach to ImageNet and demonstrate that can improve the robustness of ImageNet classifiers. Implementation details about architectures, loss parameters, and hyperparameters can be found in Appendix C
4.1 DOES OUR APPROACH LEARN THE DISENTANGLED INDEPENDENT MECHANISMS?
Standard metrics like the Inception Score (IS) (Salimans et al., 2016) are not applicable since the counterfactual images are outside of the natural image domain. We thus focus on qualitative results in this section. For quantitative analysis, we refer the reader to Section 4.3 where we analyze the accuracy, robustness, and invariance of classifiers trained on the generated counterfactual data.
The generated counterfactual images are shown in Figure 3 (right). None of the counter- factual combinations were present in the training data. We can see that CGN successfully generates high-quality counterfactuals. The results on Wildlife MNIST are surprisingly good, considering that the object texture is only observable on the relatively thin digits. Nevertheless, the texture IM lears to generate realistic textures. All experiments on MÄIST are done without pre-training any network.
MNISTs.
ImageNet. As shown in Figure 4 our CGN generates counterfactuals of high visual fidelity. We train a single CGN for all 1000 classes. We also find an unexpected benefit of our approach. some instances, the composite images eliminate structural artifacts of the original BigGAN images, such as surplus legs, as shown in Figure 5] We hypothesize that fahape learns a general shape concept per class, resulting in outliers, like elephants with eight legs, being smoothed out. We show more samples, individual IM outputs, and interpolations in Appendix D The CGN can fail to produce high-quality texture maps for very small objects, e.g., for a bird high up the sky, the texture map will still show large portions of the sky. Also, in some instances, residue of the object is left on the background, e.g., a dog snout. For generating counterfactual images, this is not a problem as different object will cover the residue. Lastly, the enforced constraints can lead to reduction in realism of the composite images Xgen compared to the original BigGAN samples. We show examples and propose solutions for these problems in Appendix F
4.2 WHICH INDUCTIVE BLASES ARE NEEDED ACHIEVE DISENTANGLEMENT?
We employ two kinds of biases: pre-training of modules and IM-spcific losses. We find that pre- training is not necessary for our experiments on MNIST. However, when scaling to ImageNet, pow- erful pre-trained models are key for achieving good results. Furthermore, this allows to train the

---

Published as a conference paper at ICLR 2021
Stape
ralóit
truuppul arch
mudroven
Iyynna
callinge
basuu
cleplant
school
Background
Noseball
valley
bittenn (binc)
viaduct
grey whale
snorkel
Figure 4: ImageNet Counterfactuals. The CGN successfully learns the disentangled shape, texture, and background mechanisms, and enables the generation of numerous permutations thereof.
m
Xgen
Figure 5: Individual IM Outputs over Training. We show pre-mu ma-ks m, masks m, foregrounds f, and backgrounds b. The arrows indicate the beginning and end of the training. The initial output of the pre-trained models is gradually transformed while the composite image only marginally changes.
whole CGN on single NVIDIA GTX 1080Ti within 12 hours, in contrast to BigGAN, which was trained a Google TPU v3 Pod with 512 cores for up 48 hours. To investigate each loss' influ- ence, we disable one loss at a time and measure its influence on the quality of the composite images. The composite images are on the image manifold, hence, we can calculate their Inception score (IS). As we train with pseudo ground truth, the performance of the unconstrained BigGAN is natural upper bound. The used model reaches an IS of 202.9. To measure if the CGN collapsed during training, we monitor the mean value of the generated mask Pmask- A M'aak close to means that ftent is not training. Instead, it generates the output of the pre-trained BigGAN, hence, a mask of I's trivially minimizes the reconstruction loss Cree (xet(Xyt; Xgen). The same is true for jmask close to 0 and fay- The results in Table T indicate that each loss necessary, and jointly optimizing all of them end-to-end is needed for a high IS without collapse of Mmaakk- Removing Lshapes leads to bad quality masks (non-binary, only partially capturing the object). This results in a low IS since object texture and background get mixed in the composited image. Not using either Lesat or Lbg results in a high IS (as the output is close to the original BigGAN output), but a collapse of Pmask- The mechanisms do not disentangle their respective signal. Finally, disabling Cree leads very low IS, since the IMs can optimize their respective loss without any constraint on the composite image. We show the evolution and collapse of the masks over training in Appendix G
4.3 DO COUNTERFACTUAL IMAGES ENABLE TRAINING OF INVARIANT CLASSIFIERS?
The following experiments investigate if instill invariance into classifier. We perform experiments on the MNIST variants, a cue-conflict dataset, and an OOD version of ImageNet.
MNIST Classification. In the training domain, shapes, colors, and textures correlated with the class label. In the test domain, only the shapes correspond to the correct class. We compare to current approaches for training invariant classifiers: IRM (Arjovsky et al. 2019) and Learning-nof---llaamn (LNTL) (Kim et al. 2019). For a detailed description we refer to Appendix C Original CGN is additionally trained on counterfactual data to predict the input labels of the shape IM. Original GAN is bascline that is trained on real and generated, non-counterfactual samples. IRM considers signal be causal if is stable across several environments. We train IRM on environments (90 % and 100 % correlation) or 5 environments (90 %, 92.5 %, 95 %, 97.5 %, and 100% correlation). LNTL considers color to be spurious, whereas we assume (complementary) that shapes are causal. Alternatively, we can follow the same assumption as IRM with an additional causal identification

---

Published as a conference paper at ICLR 2021
Cskege
Lives
Liy
Ever
15 1
Plens
colored
MINIST Tell Aok.Noc
MINIST
/
85.9
0.22
Trais Ave
Ase
Inan
Aoeis.Ao 1
Test Aue
198.4
0.9+0..15
Original
99.5 16
35.9%
100.0 %
10,3 16
100,8
10.1 %
K
195.6
0.1 0:11
BRM (2 Erna)
38.39
0.3+0..2%
BRM (5 bons)
99.9
78.9
99.899
76,8%
130.2
0.:+-..25
ENTI.
96.3% 56
$188%
98.7%
69.0
99.9%
11.5%
Oniiiaa+
RigGAN (Upper Bound)
202.9
Orggnnal CON
99.7 1g
95.1
97.4%
89.0 16
99.2 96
65.7% 0g
Table 1: Loss Ablation Study. We turn off one loss at time. Values indicating mask collapse are red.
Table 2: MNISTs Classification. In the test set, colors and
textures are randomized, only the digit's s shape corresponds to the class label. Random performance is at 10%.
Trained
Shape Bias
top-1 I IN Ace
lop-5 IN
Top-1 Test Accuracies
IN
21.39
76.13 %
92.86 %
Trained
IN-9
Mixed-Same f
Mixed-Rand
BG-Gap U
SIN
81,37 %
60.18 %
82.62 %
IN
95.6%
86.2%
78.9%
7.3%
IN + SIN
34.65
74.59
90.03 %
SIN
89.2 %
73.1 %
63.7 %
9.4% %
IN + CGNSShupe
54.82
+ SIN
94.7 %
85.9 %
78.5 %
7.4 %
CGN/Text
16.67
91.71
Mixed-Rand
73.3%
71.5%
71.3%
OGN/Bg
22.89
+ CGN
94.2
83.4
3.3%
Table 3: Shape vs. Texture. We can control
Table Accuracies on IN-9. The reported accu- racies all obtained using Resnet-50.
the classifier's shape or texture preference.
step, see Appendix H The results in Table confirm that training on counterfactual data leads to classifiers that are invariant to the spurious signals. We hypothesize that the difference between environments may be hard to pick up for IRM, especially if only few are available. We find that we can further improve IRM's performance by adding more environments. However, continually increasing the number of environments is an unrealistic premise and only feasible in simulated environments. Our results indicate that LNTL and IRM have trouble scaling to more complex data.
Texture vs. Shape Bias. The Cue Conflict dataset consists of images generated using iterative style transfer (Gatys et al. 2015) between texture and a content image. A high shape bias corresponds to classification according to the content label and vice versa for texture. Their approach trained on stylized ImageNet (SIN), either as drop-in for ImageNet (IN) or augmentation. We use a classifier ensemble, i.e., a classifier with a common backbone and multiple heads, each head invariant to all but FoV. We average the predicted Dog-probabilies of each head for the final output of the ensemble. We conduct all experiments using a Resnet-50 architecture. As shown in Table 3 we can influence the individual bias of each classifier head without significant degradation in the ensemble's s performance.
Invariance over Backgrounds. Xiao al. (2020) propose the BG-Gap measure a classifier's dependence on the background signal. Based on ImageNet-9 (IN-9), a subset of ImageNet with 9 coarse-grained classes, they build synthetic datasets. For Mixed-Rand, the backgrounds are ran- domized, while the object remains unchanged, hence background an class are decorrelated. For Mixed-Same they sample class-consistent backgrounds. The BG-Gap the difference in perfor- mance between the two. Training on IN or SIN does not make it possible to disentangle and omit the background signal, as shown in Table 4 Directly training on Mixed-Ran Binded-Rand leads to a drop in per- formance on the original data which might be due to the smaller training dataset. We can generate unlimited data of this type, hence, we are able to reduce the gap while achieving high accuracy on IN-9. However, a gap to a fully invariant classifier remains. We partially attribute this to the general remaining domain gap between generated and real images.
RELATED WORK
Our work is related disentangled representation learning and the training of invariant classifiers.
Disentangled Representation Learning. A recent line of work in image synthesis aims to learn disentangled features for controlling the image generation process (Chen et al. 2016 Higgins al.) 2017 Liao et al. 2020). The challenge of the task is that the underlying factors can be highly correlated. Closely related to our work (Li et al. 2020). which aims disentangle background, shape, pose, and texture, using object bounding boxes for supervision. Their methods assumes
8

---

Published as a conference paper at ICLR 2021
images of a single object category (c.g. birds). We scale our approach to all classes of ImageNet which enables us to generate inter-class counterfactuals. A recent research direction explores the discovery of interpretable directions in GANs trained on ImageNet (Plumerault et al. 2020 Voynov & Babenko 2020; Peebles et al. 2020). These approaches do not allow for generating counterfactual images. Kocaoglu et al. (2018) train two separate generative models, one generating binary feature labels (mustache, young), the other generating images conditioned on these labels. Their model can create images of previously unseen combinations of attributes, e.g., women with mustaches. This approach assumes a data set with fine-grained labels; hence it would not be suited to our application since labels for high-level concepts like shape are hard to obtain. [Besserve et (2019) also leverage the idea of independent mechanisms to discover modularity in pre-trained generative models. Their approach does not allow for direct control of image attributes. Lastly, methods for causal generative modeling utilizing competing experts (von Kügelgen et al. 2020) have been demonstrated on toy datasets only. Further, none of the works above aim use the generated images to improve upon a downstream task such as image classification.
Invariant Classification. Current approaches do not take an underlying causal model into account. Instead, they rely on different assumptions. [Arjovsky et al. (2019) assume that the training data collected into separate environments (e.g. different measurement circumstances). Correlations that are stable across environments are considered to be causal. Kim et al. (011)) aim to learn fea- tures that are uninformative a given bias (spurious) signal. As mentioned above, attaining labels for shape or texture is expensive and not straight-forward. A recent strand of work is concerned with data augmentation for improving invariance against spurious correlations. (Shetty et al. (2020) propose train object detectors generated semantic adversarial data, effectively reducing the tex- ture dependency of their model. Their finding is in line with (Geirhos et al. (0118) that proposes to transfer the style of paintings onto images and them for data augmentation. These approaches, however, do not allow to choose the specific signal we want invariance for, e.g., the background.
The use of counterfactual data has been previously explored in natural language inference (Kaushik) et al. 2020 and visual question answering (Teney et al. 2020).
6 DISCUSSION
We assume that an image can be neatly distinguished into a class foreground and background throughout this work. This assumption breaks once we consider more complex scenes with differ- ent object instances or for tasks without a clear pregrruuddbbackrroun images. The composition mechanism is a powerful bias, and crucial to making our model work. In other domains, equally strong biases may need to be identified to enable learning the SCM. An exciting research direction is to explore different configurations of IMs to tackle these challenges.
distinction, e.g., in medical
The additional constraints that we enforce during the CGN training lead to reduced realism, as evidenced by the lower IS. We also find that our generated images can significantly influence a classifier's preference, but their quality is not high enough improve performance ImageNet. However, even state-of-the-as generative models (with higher IS) are not good enough yet to gen- erate data for training competitive ImageNet classifiers (Ravuri & Vinyals 2019).
Lastly, in our experiments, we assume the causal structure to be known. This assumption is sub- stantially stronger than the ones in more general standard disentanglement frameworks (Chen al., 2016 [Higinn et al.] (017). A possible extension to our work could leverage causal discovery to isolate IMs in domain-agnostic manner, e.g., via meta-learning (Bengio al.) 2020). On the other hand, the definition of a causal structure and the approximation through IMS may be principled way to integrate domain knowledge into machine learning system, The need for better interfaces to integrate domain knowledge has recently been highlighted in D''Aour et al. 2020).
7 CONCLUSION
In this work, we apply ideas from causality to generative modeling and the training of invariant clas- sifiers. We structure a generative network into independent mechanisms to generate counterfactual images useful for training classifiers. With the use of several inductive biases, we demonstrate our approach on various MNIST variants as well as ImageNet. Our ideas are orthogonal to advances in generative modeling with advances therein, our obtained results will further improve.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We acknowledge the financial support by the BMWi in the project KI Delta Learning (project num- ber 19A 190130). Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533). We would like to thank Yiyi Lao, Michael Niemeyer, and Elie Aljalbout for comments paper draft and Songyou Peng. Michael Oechsle, and Kashyap Chitta for last-minute proofreading. We would also like to thank Vanessa Sauer for her general support and constructive criticism on the generated counterfactuals in earlier stages.
earlier
REFERENCES
Michael A Alcorn, Li Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In CVPR, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Pazz Invariant risk minimization. arXiv preprinl urXiv:1907.02890 2019.
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan Goldman. Patchmatch: random- ized correspondence algorithm for structural image editing. In ACM TOG, 2009.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. TPAMI, 2013.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bila- niuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. In ICLR, 2020.
Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. In ICLR, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14 4655, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NeurlPS, 2016.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdi- ari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam. Mario Lucic. Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymy- rov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint MxXi:20111.03995 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. neural algorithm of artistic style. arXiv preprint 1rXiv:1508.06576 2015.
10

---

Published as a conference paper at ICLR 2021
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias im- proves accuracy and robustness. In ICLR, 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.07700 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (CACE). arXiv preprint arXiv:190..07165, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Maximilian Ilse, Jakub M Tomezak, and Patrick Forré. Designing data augmentation for simulating interventions. arXiv preprint arXiv:2005.01856, 2020.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. Learning the difference that makes difference with counterfacuuall--aggeeneee data. In ICLR, 2020.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In CVPR, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXhv:1412.6980, 2014.
Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causalgan: Learning causal implicit generative models with adversarial training. In ICLR, 2018.
Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Mixnmatch: Multifactor disen-
tanglement and encoding for conditional image generation. In CVPR, 2020.
Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In CVPR, 2020.
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In ECCV, 2018
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. In JCML, 2018.
Eshed Ohn-Bar, Aditya Prakash, Aseem Behl, Kashyap Chitta, and Andreas Geiger. Learning situ- ational driving. In CVPR, 2020.
Judea Pearl. Causality. Cambridge University Press, 2009.
William Peebles, John Peebles, Jun-Yan Zhu, Alexei Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In ECCV, 2020.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference. The MIT Press, 2017.
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. In ICLR, 2020.

---

Published as a conference paper at ICLR 2021
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recognition, 2020.
Suman Ravuri and Oriol Vinyals. Sceing is not necessarily believing: Limitations of biggans for data augmentation. ICLR Workshop LLD, 2019.
Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprini arXiv:18088 03305, 2018.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurlPS, 2016.
Bernhard Schölkopf. Causality for machine learning, arXiv preprint
2019.
Rakshith Shetty, Mario Fritz, and Bernt Schiele. Towards automated testing and robustification by semantic adversarial data generation. In ECCV, 2020.
Raphael Suter. Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In JCML, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1rXXv::312..699 2013.
Damien Teney, Ehsan Abbasnedjad, and Anton den Hengel. Learning what makes difference from counterfactual examples and gradient supervision. In ECCV, 2020.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In IROS, 2017.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, 2011.
Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition experts. In ICLR Workshop CLDM, 2020.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. arXiv preprint arXiv:2002.03754, 2020.
Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng. Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. CVPR, 2017.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In CVPR, 2020.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.0990 09994, 2020.
Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. Lr-gan: Layered recursive generative adversarial networks for image generation. arXiv preprint arXiv: 170..07030..000 2017.
12

---

Published as a conference paper at ICLR 2021
APPENDIX A MNIST VARIANTS
A.1
VARIATIONAL AUTOENCODERS ON COLORED MNIST
Figure 6 shows the latent space of 3-VAE Higgins et al. 2017) trained colored MNIST. The VAE disentangles the data into different clusters present in the data. However, the axes do not correspond to color and shape, i.e., color and shape vary when traversing one latent. We used only two latent dimensions for visualization purposes; the problem is not resolved by adding more dimensions. The same behaviour can be observed for unconstrained GANs (Goodfellow et al. 2014). as a GAN also approximates the training distribution.
O
I
1
3
01777
033777
2
2
2
3
3
3
3
3
3
3
3 4 5 6 7 8
3 4 5 6 7 8
4 5 C 7 8
8
2
9 9
2 2 2
Figure 6: Colored MNIST. (left) Examples of data points. (right) Training disentangled VAE with two latent dimensions on colored MNIST.
A.2 MNISTs GENERATION
Colored MNIST. This dataset was proposed by Kim et al. (2019). They select ten distinct colors and assign each of them to a class. For each training image, they sample a color from normal distribution with the class color as the mean. In the test set, the colors are randomly assigned. The variance a of the normal distribution can be used to control the amount of bias. We evaluate on the hardest, i.e., most biased, setting with = 0.02.
Double-Colored MNIST. We follow the same procedure as for colored MNIST. We additionally encode the class label in the background color.
Wildlife MNIST. To build a version MNIST closer to an ImageNet setting, we add a texture bias to the data. We follow the same procedure for double-colored MNIST. We take textures from (Cimpoi et al., 2014) and use ten images of the texture class "striped" to encode the label in the foreground. Similarly, we encode the label in the background with textures of the texture class "veiny." " We do not add noise to texture to add stochasticity. Instead, we sample a 32x32 patch from the larger texture image. The textures are multi-modal; hence, these patches can look quite different.
A.3 ABLATION STUDIES
we study the effects of the number of counterfactual data points on the test accuracy. We increase the amount of counterfactual data while the amount of real data is fixed (50k for MNIST). We also ablate the amount of counterfactual drawn per sampled noise u. We find that the higher the number of counterfactual data points, the better. Also, it is advantagcous to draw several coun- terfactuals per Our intuition is that several counterfactuals provide a more stable signal of the non-spurious factor.
In Figure 7
13

---

Published as a conference paper at ICLR 2021
Co ored MNIIST
Doubie-Colored MNIST
Wildlife MINIST
cf waa--3
Figure 7: MNIST Ablation Study. To improve visibility, start with 10 counterfactual data points, below the performance is marginally better than the fully biased baseline. The CF ratio indi- cates how many counterfactuals CF ratio is ten as there are only ten possible colors per shape.
generate per sampled noise. For colored MNIST, the maximum
APPENDIX B CAUSAL STRUCTURES
The two SCM's are as follows:
MNISTs
ImageNet
M shape (Y,, U1)
M fahape (Y1, U1)
F = ftext,1 (Y2, U2)
F fient (22 U2)
B Steat, (Y3; Uz)
Jog(Yss Uz)
X gen C(M, F. B)
X gen = C(M,
where M is the mask, is the foreground, B is the background, Uj is the exogenous noise, Yj is the class label, Xgen is the generated image, and ʃj and C are the independent mechanisms.
APPENDIX C IMPLEMENTATION DETAILS
C.1 SHAPE LOss DETAILS
The full shape loss as follows:
Lshape (fa) Lbinary (fe) + Lmask (fs)
= =p((uy)
Σ
m; logg (mg)(m;) (1- (1-m;) m4) logg (1m)))
N
mi + max (,
N N Σ m4 -7
+ Ep(n))
max
0,
0,
N
where Loinary is the wixel-wise binary entropy, Lmask is the mask loss, m Sahape (u, y) is the mask generated by fahape and is scalar threshold. Linnary enforces the output to be close to either or Lmask prohibits trivial solutions, i.., masks with all 0's or I's, that are outside the interval defined by т. We set 0.1 in all experiments. of 0.1 means that jmask is forced to be the interval of 10.1, 0.9] - the main object should occupy more than 10% and less than 90% of the image.
C.2 TEXTURE LOss DETAILS
The sampling procedure for Liert is as follows: we sample 36 patches of size 15 from the regions where the mask is closest to 1. Out of these 36 patches, we build a 6 x 6 patch grid. Finally, we upscale the grid the full 256 x 256 resolution; we denote this grid as pg. We then minimize perceptual loss between the foreground (the output of fteat) and the patchgrid: Liezt (f, pg).
14

---

Published as a conference paper at ICLR 2021
C.3 CGN TRAINING ON IMAGENET.
Training Settings. For training the CGN, we jointly optimize the following loss:
L = Lree + Lehope + Aghrezt + Aghon
(4)
= A1C + X2ppere + A3 A3Cosnary + A4 + A5 + Ab
We use the following lambdas: X 100, d 5. 300, A 500, As Ag = 2000. For the optimization we use Adam (Kingma & Ba 2014), and set the learning rate of fshape to 8 and for both Stent and Sbg to 1e-5. We not use real data for training the CGN so we do not need to any data loading. We sample single image from BigGAN and the CGN and accumulate the loss gradients for 4000 steps before taking gradient step. Accumulating large pseudo-batches proved crucial for high-quality gradients, confirming the observations of (Brock et al. 2018). Further, using a batch size of one makes possible to train on a single GPU. For the perceptual losses, i.c., pre-trained VGG16 with batch normalization layers. We calculate the style
Epere and Liext, reconstruction loss (Johnson et al. 2016) the features after the first four max-pooling layers.
We use the pre-trained BigGAN models from https: /tt:://githai com/ hwwginggggggge/////gooogg pyytorch-peetrainneeeeeeee-- ned-trrrinndBBiooo gGAN We experiment with different values for the truncation value of the truncated normal distribution used to sample the input noise. However, find it does not impact the performance of the CGN. Also, a low value leads to worse performance of the trained classifiers; hence, we leave the truncation value
Hyperparameter Search. We measure the Inception score (IS) during training and the mean value of the masks Pmosk to detect mask collapse. We also observe the generated images for a fixed noise vector; see the outputs in Figure 5. Our overall objective is high IS and a stable Mmask. Further, we aim for high-quality output of all IMs (Masks: binary, capture only class-specific parts; Textures: no background/llobal shape visible, Background: no trace of foreground objects visible). We observe these outputs for several classes during optimization. The hyperparameters can be tuned mostly independently from each other, i.e., better lambda for the mask loss does not influence the quality of the texture maps much.
C.4 CLASSIFIER TRAINING
We use the same CNN architecture for all experiments and approaches. For IRM, we produce versions of double-colored MNIST and Wildlife MNIST with different degree of correlation between the label and the foreground colors/txtures and background colors/textures. We then train IRM on 2 environments (90% and 100% correlation) or environments (90%. 92.5%, 95%, 97.5%, and 100% correlation). We schedule the gradient norm penalty weight, starting from O, then linearly increasing it over the training episodes. We find the scheduling to be crucial for IRM to converge and achieve good performance.
MNIST.
ImageNet. We use a ResNet-50 from PyTorch torchvision. We share the weight up to the last layer as a common backbone and add three fully-connected heads (shape, texture, background). Each of the heads is provided its respective label when training on the counterfactual images. On the real images, we average the logits of all heads. This approach allows the single classifiers to focus on its assigned FoV while the ensemble performs well overall. Similarly to the experiments by (Geirhos et al., 2018), we begin the training with pre-trained weights from ImageNet. We train for 70 episodes with Stochastic Gradient Descent using batch size of 512. Of the 512 images, 256 are real images, 256 are counterfactual images. We find that an even ratio between real and counterfactual images leads to the best results in terms of optimization stability and performance. We use momentum of 0.9, weight decay (Ie-4), and a learning rate of 0.1, multiplied by a factor of 0.001 after 30 and 60 epochs.
15

---

Published as a conference paper at ICLR 2021
APPENDIX D MORE SAMPLES AND INTERPOLATIONS
D.I INDIVIDUAL IM OUTPUTS FOR DIFFERENT CLASS TYPES
In the following we illustrate the individual outputs of each IM for different classes, In each figure, we show from top to bottom: pre-masks m, masks m, texture maps f, backgrounds b, and composite images Xgen- For all shown outputs, set the truncation parameter for the noise to 0.5.
Figure 8: IM Outputs for 'jay". From top bottom: m, m, f, b, Xgen-
Figure 9: IM Outputs for 'wallaby'. From top to bottom: m, m. f, b, Xgen-
Figure 10: IM Outputs for 'king penguin'. From top bottom: m, m, b, Xgen-
Figure 11: IM Outputs for "vizsla'. From top to bottom: m. m, f, b, Xgen-
16

---

Published as a conference paper at ICLR 2021
Figure 12: IM Outputs for 'barn'. From top bottom: m, m, f, b, Xgen-
Figure 13: IM Outputs for 'speedboat', From
top to bottom: m, m, f, b, Xgen-
Figure 14: IM Outputs for 'viaduct'. From top
Figure 15: IM Outputs for 'cauliftower'.
to bottom: m. m. f, b, x gen-
From top to bottom: m, m, f, b, ggen-

---

Published as a conference paper at ICLR 2021
Figure 16: IM Outputs for 'bell pepper'.
Figure 17: IM Outputs for 'strawberry'.
From top to bottom: m. m, f, b, Xgen.
From top to bottom: m. m, f, b, Xgen.
Figure 18: IM Outputs for 'geyser'. From top
Figure 19: IM Outputs for 'agaric'. From top to bottom: m, m. f, b, Xgen-
to bottom: m. m. f, b, x gen-
18

---

Published as a conference paper at ICLR 2021
D.2 LATENT INTERPOLATIONS
Indian cobra
ostrich
submarine
commode
triumphal arch
mud turtle
unicycle
American cagle
wine bottle
tripud
forklift
aircraft carrier
Figure 20: Interpolating Shapes. We interpolate between and y pairs.
wine bottle
pill bottle
pome- granate
pizza
military uniform
cheetah
jack-o'- lantern
cauliflower
king penguin
trench coat
monarch butterfly
French
loaf
Figure 21: Interpolating Textures. We interpolate between and pairs.
19

---

Published as a conference paper at ICLR 2021
water ouzel
fig
sandbar
viaduct
Airedale terrier
axolotl
mushroom
valley
cliff
turtle
grey whale
coral reef
Figure 22: Interpolating Backgrounds, We interpolate between and y pairs.
D.3
MORE COUNTERFACTUAL SAMPLES
1
1
1
/
f
2
A
2
%
2
2
2
2
3
2
3
3
3
6
3 4 5 6
3
3
3
5
3
4
4
4 5
4
4
4
5
5
5
5
5
5
5
6
6
6
6
6
7
1
7 0
7 $
7
7
7
7
g
8
8
5 8
8
8
8
0
4
9
9
9
9
9
9
Figure 23: MNIST Counterfactuals. From Left Right: colored, double-colored-, and Wildlife
MNIST.
20

---

Published as a conference paper at ICLR 2021
Shape
Background
Stape
Background
school bus
penda
geyser
otTáore rig
anbulance
breakwater
hymaa
chello
stamberry
mushroom
sand
cetrich
lipstick
diving
stowmobile
Indian cobra
undlyy clock
green licand
4
katamaran
chorial
ganden
bank
bald tagle
striwberry
spike
liyea dog
breasiplanc
truumpal
poodle
ballfing
2
Genman shaphend
king penguis
bitters
Gekii
Beaierr cuttain
valley
beaker
busby
pincopple
whine bottle
bottle
orange
spider
2
teldy
bozt glass
rock crab
submasise
Face
lay
malinois
viaduct
wood nibiit
jackool Justten
ouzii
upright
teаpor
militry
houglins
lanoo
hunbill
Figure 24: ImageNet Counterfactuals, Top: Counterfactual Images. Bottom: ImageNet labels.
21

---

Published as a conference paper at ICLR 2021
APPENDIX E IM OUTPUTS OVER THE COURSE OF TRAINING
In the following we illustrate the individual outputs of each IM over the course of training. In each figure, we show from top to bottom: pre-masks m, masks m. texture maps f, backgrounds b, and composite images Xgen-
m
m
f
b
Xgen
Figure 25: IM Outputs over Training for 'dalmatian' The arrows indicate the beginning and end of the training.
ti
f
b
Xgen
Figure 26: IM Outputs over Training for 'cauliflower' The arrows indicate the beginning and end of the training.
m
b
Xgen
Figure 27: IM Outputs over Training for 'castle" The arrows indicate the beginning and end of
the training.

---

Published as a conference paper at ICLR 2021
m
f
b
100
Xgen
Figure 28: IM Outputs over Training for 'ringtailed lemur' The arrows indicate the beginning and end of the training.
m
m
f
b
Xgen
Figure 29: IM Outputs over Training for 'mushroom' The arrows indicate the beginning and end of the training.
23

---

Published as a conference paper at ICLR 2021
APPENDIX F
FAILURE CASES OF CGN
Xgt
f
Xgt
f
Xgt
f
Figure 30: Texture- Backur--a Entanglement. For relatively small objects, the texture maps can still show traces the background. possible remedy would be choose the patch size for Ltext dependent on the relative object size.
Xgt
b
Xgt
b
Xgt
b
Figure 31: Background Residues. Especially for large objects, i.e., where large regions need to be in-painted, there can be faint artifacts visible. For the composite images, this is not a problem as an object will cover the residue.
Xgt
Xgen
Xgt
X Xgen
Xgt
X Xgen
Figure 32: Reduced Realism. As evidence by the lower IS, the generated images Teen are generally lower in realism. This reduced realism is due to the constraints that we enforce and the simplified composition mechanism. A solution might be to add shallow refinement network after the com- poser.

---

Published as a conference paper at ICLR 2021
APPENDIX G COLLAPSING MASKS
Figure 33: Masks when disabling different losses. From left right: the beginning of training. training without Lshope, training without Liezt, training without Lög. training with all losses. The third and fourth columns show the collapse of the masks, as described in section 4.2
APPENDIX H CGN AUGMENTATION WITH IRM ASSUMPTIONS
We drop the assumption of priori knowledge of the causal signal and follow the assump- tion as IRM: several environments with varying correlations, an invariant signal is considered causal. In the following, we train CGN on double-colored MNIST. We then generate counterfactual data to train three classifiers:
Shape classifier (SC): invariant wrt. object color and background color
Object color classifier (OCC): invariant wrt. object shape and background color
Background color classifier (BCC): one invariant object shape and object color
We measure their respective performance in the environments used train IRM. In these envi- ronments, only the shape is stably correlated with the label; the foreground and background color vary in their degree of correlation. Based the results in Table the causal signal as only the test accuracy of SC is stable across environments.
determine the shape be
Degree of Correlation
Accuracy SC [%]
Accuracy OCC [%]
Accuracy BCC [%]
90 %
85.05 + 0.12
58.72 + 0.67
82.47 + 0.13
95 %
85.08 + 0.07
62.69 = 0.58
87.96 + 0.12
100 %
85.14 + 0.10
65.05 = 1.32
90.16 + 0.13
Table 5: Test Accuracy on double-colored MNIST. We report the mean and standard deviation over three random seeds.
25


Reviews:
[{"review_id": "smrkYgYnS5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Summary ---\n\nThis paper proposes a new generative model that generate images from 3 seperate aspects: foreground masks (shapes), forground texture, and backgrounds. Then they convexly mix these 3 aspects into one image. By doing so, they can vary each aspect individually without changing other aspects, enabling the model to generate counterfactual images. For example, we can generate a cat shape with telephene texture and sea background, which would not exist in natural images. They show that in several colorful MNISTs datasets their methods can generate new combinations of images. In ImageNet, by using the pre-trained BigNet GAN as backbones and pretrained U-Net for foreground object masks, they can distill the knowledge in BigNet into these 3 seperate aspects and generate counterfactual natural images.\n\n--- Pros ---\n\n1. Intersting ideas of combining pre-trained models with novel loss function that helps disentangle these 3 aspects.\n2. Plausible and intriguing counterfactual examples.\n3. Strong improvements in colorful MNIST datasets.\n\n--- Major comments ---\n\n1. The claim that \"we are able to reduce the gap while achieving hig accuracy on IN-9\" is not true. In Table 4, the method IN+CGN has lower accuracy than IN alone, and it reduces the gap by lowering the original performance in Mixed-same. So it does not actually improve the performance.\n\n2. The above makes me wonder the ImageNet experiment does not actually generate plausible enough images that help improve accuracy, probably due to its unnatural image generation as evidenced by relatively low IS scores (130). \nEspecially seeing there are 6 lambdas to tune listed in Appendix B.3 and other hyperparameters like learning rates etc, this method might not be very practical in high-dimensional natural images. Maybe authors can be honest about it. Or illustrate what's the best way to tune this method, what has been attempted etc.\n\n3. The current setting seems to a bit limited that requires a single foreground and background. For example, designing independent mechanisms for medical imaging might not be easy with no exact foreground/background boundaries.\n\n4. In Table 3, maybe authors can evaluate on some difficult ImageNet datasets like ImageNet-A to assess if the performance improves.\n\n--- Minor comments ---\n\n1. The loss descriptions in Appendix B should be moved to main text to help readers understand the method. Details like how to pick $\\tau$ for shape loss should be mentioned.\n\n2. The name \"pre-masks\" is confusing that originally I think it's a binary mask, but instead it's a colorful image.\n\n3. More failure examples in Appendix E will better help readers understand its limitations.\n\n4. The figure 5 and the Appendix D should also include the final image that combines m, f and b to better assess its improvement over the training stages. Also, having an arrow in Figure 5 or specifying epochs might better help readers understand it is showing the transitions.\n\n5. The caption in Figure 24 is wrong: should be columns instead of rows.\n\n--- Evaluations ---\n\nOverall I like this paper. The method is interesting to read and the examples are interesting. The experiments are thorough and do show some improvements in colorful MNISTs. The method, unfortunately, does not seem to work that well in ImageNet, and does not improve generalization performance. I encourage the authors to be upfront about the limitations of this method and write better descriptions of loss and hyperparameters tuning. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "jdVomZdsJ_1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting conceptual advance connecting causality, disentangled representation learning, invariant representations and robust classification.\nThe authors propose a Counterfactual Generative Network (CGN), which is basically \"modular\" generative adversarial network that can independently control the generation of independent factors of variations in the data corresponding to Independent Mechanism, i.e. independent factors in the structural causal model of the data. In the context of generating natural images like those comprising the ImageNet dataset, the CGN once trained can be used to generate high-quality counterfactual images with direct control over of factors of variations determining the content of an image shape, texture, and background. These generated samples obtained by independently and uniformly sampling over factors of variation can be used to train a classifier to achieve out-of-domain robustness. The authors show indeed show in simulation that this procedure works as a data augmentation procedure that increases out-of-domain robustness while only marginally degrades the overall accuracy. As the authors explain, this can be thought of as a generalization of \"domain randomization\".\nAdditionally, CGN can be used as a generative model of high-quality binary object masks and unsupervised image inpainting.\nThe authors also carry out extensive ablation studies that quantify the contribution of the different training costs for CGN to the overall quality (measures as Inception Score) of the generated counterfactual images. \nThis is first and foremost an \"idea paper\" putting forth a very interesting conceptual proposal. This is then empirically validated on out-of-distribution classification tasks in different versions of colored MNIST, and a coarse grained subsed of ImageNet.\nA natural question for the authors is whether and by how much this type of data augmentation might help in realistic large-scale classification scenarios. Another natural question would be to quantify the effect of the counterfactual data augmentation in terms of trade-off between performance on in-distribution and out-of-distribution samples.\nLastly, it would be interesting to know whether the authors have any thought on how to generalize their architecture to other setting, i.e. how to isolate Independent Mechanisms in a general domain and what type of domain knowledge is necessary to do that effectively.", "extra_scores": {}, "extra_text": {}}, {"review_id": "iiIQfxdodHJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main idea of the paper, i.e., using independent causal mechanisms to generate interventional images, has already been explored by Kocaoglu et al. in Causalgan: Learning causal implicit generative models with adversarial training, ICLR'18. Same as here, the authors there also \"view image generation as a causal process\" and \"structure a generator network as a structural causal model (SCM)\" and use a conditional gan to generate the image from the labels. The generation used here based on three variables, i.e., shape, texture and background seem to be a special case. Therefore, the authors should definitely cite this work. \n\nMy general remark is that there is very little causality in the approach. The causal structure that is used in the generation of data is not different than a conditional GAN. This makes the claims in the introduction very disconnected from the actual methodology and the experiments in my opinion.\n\n\" we can intervene on a subset of them and generate counterfactual images \"-> What the authors call counterfactual images are actually interventional images from a causal point of view. Please consider changing \"counterfactual\" to \"interventional\" throughout the paper. This will help clarify the distinction between interventional and counterfactual layers in Pearl's hierarchy. \n\n\"From a causal perspective, we maximize the average causal effect (ACE) of one IM on the classifier’s decision, while minimizing the ACE of all other IMs.\"\nCan you formalize this claim? This does not seem trivial.\n\nCould you explain \"alpha blending\"? This step is not motivated well and seems specific to the used dataset. \n\nEven though the ImageNet experiments look impressive, I believe the main factor for success is in the deterministic and manually defined composition mechanism. Furthermore, I believe this composition is doing most of the disentangling during training. \n\nForeground and background segmentation use an existing method U2-Net which is used to create masks, or values for the variables used in the graph. \n\nThe intuition on comparing with other methods is missing. For example, why do you think training a classifier on interventionally augmented data performs better than IRM? Shouldn't this depend on the number of environments and degree of correlation? These are not reported. \n\n\"We, therefore, follow an augmentation strategy, i.e., we augment ImageNet with additional counterfactual images.\"\nHow many samples are added to the original data? I believe the amount of augmentation relative to the original dataset size is important.\n\n \n%% AFTER REBUTTAL %%  \nThank you for all the updates.\n\nI would like to thank the authors for their humility in the rebuttal and for clarifying the paper's contributions. Accordingly, I will increase my score. However, I still believe Section 3.1's contribution, and the follow-up of using this to improve classifier robustness, is useful only for a very specific type of data and it is hard to assess its value from a practical point of view. \n\nThe fact that the authors were able to showcase that such counterfactual data augmentation improves classification is, although expected, useful in itself. However, performance improvement is only evident in colored MNIST, relative to GAN augmentation. Furthermore, R4 points out the important issue that the relevant causal feature is assumed to be known in the experiments. This information is normally not available and must be inferred by the classifier. The additional experiments provided by the authors during the rebuttal are welcome but they should be in the main paper rather than the appendix since this is the main setting where spurious correlations create problems. I believe the experimental section should put more weight on this setting.\n\nIn light of all this, I will provide a borderline score leaning towards rejection. \n\nI encourage the authors to expand section 3 to settings that do not restrict the images to have one foreground object and a single background. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "XFEnIrNbVF4", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Deep neural network brittleness can be attributed to their tendency to latch on to spurious correlations in the training dataset. The proposal in the paper is to learn to generate samples where these correlations can be eliminated. To this end, the authors, distill trained conditional big gan into a transformation with explicit modules to capture the shape, texture of the foreground object, and the background. The distilled network is called Counterfactual Generator Network (CGN). Thus, an image can be generated with a background of one class, the shape of another class, and the foreground texture of a different class. Then a classifier with multiple heads is learned where each head predicts a class based on only one of the factors among shape, texture, and background. \n\nThe proposed approach is motivated by the assumption of independent mechanisms where different modules of the causal data generating process are independent of each other. Once the decomposition of a training image into shape, texture, and background is obtained, any component can be swapped to generate counterfactual data.\n\nPros:\n+ The solutions provided to extract object masks, background and texture are interesting and scale to Imagenet dataset.\n+  Shows that augmenting the training dataset with the generated counterfactual images can help improve robustness. \n \nCons + Questions:\n- The presentation of the paper can be improved. It is not always clear if the causal structure is assumed to known. In Sec 2.2 SCM is defined, but the SCM for the MNIST or Imagenet is not provided. Do all the nodes in the CGN share the same noise or exogenous variables?\n- The proposed method appears to assume that the causal structure is known. In this, it assumes it is made up of three nodes shape, texture, and background, and thus can limit the counterfactual generation ability. Many semantic changes cannot be achieved as evidenced by the fact that the counterfactual images are not realistic. \n- in the invariant MNIST classification task it appears that the results are based on the assumption that the invariant feature - shape is known apriori. In practice, this information is not available. IRM does not assume this knowledge, so it does not seem comparison with IRM is fair in this case.\n- Some related work that seems to be missing [1][2]\n\n[1] Kocaoglu, Murat, et al. \"Causalgan: Learning causal implicit generative models with adversarial training.\" arXiv preprint arXiv:1709.02023 (2017).\n\n[2] Kaushik, Divyansh, Eduard Hovy, and Zachary C. Lipton. \"Learning the difference that makes a difference with counterfactually-augmented data.\" arXiv preprint arXiv:1909.12434 (2019).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a "conceptual  advance connecting causality, disentangled representation learning, invariant representations and robust classification". Authors propose to decompose the image generation process to independent mechanism that can be composed (foreground masks (shapes), forground texture, and backgrounds), allowing for a specific image to generate counterfactuals , by changing some variations factors, while keeping other fixed. One can use interventional data to augment classifiers, this can lead in certain cases to improvement in accuracy and in other in improving the robustness. 

There was concerns about the clarity of the paper regarding the structured causal model considered and its applicability beyond image generation, experimental protocol for choosing hyperparameters (loss scaling and ratios of real data and interventional samples ) and some missing references. The rebuttal of the authors and their updated paper reflected comprehensively all those concerns and addressed them, highlighting limitations of the method and adding more examples of its failures. 

I liked the ideas and concepts in  this paper , and it will be exciting to generalize such generative approach to other domains, this  work is a first step. I think it will be good addition to ICLR program 
Decision: Accept (Poster)