Counterfactual Generation Under Confounding
Authors: Abbavaram Gowtham Reddy
Saloni Dash
Amit Sharma
Vineeth N. Balasubramanian
Venue: Submitted to ICLR 2023 (2023)
Keywords: counterfactual
confounding
cycleGAN
classification

Abstract: A machine learning model, under the influence of observed or unobserved confounders in the training data, can learn spurious correlations and fail to generalize when deployed. For image classifiers, augmenting a training dataset using counterfactual examples has been empirically shown to break spurious correlations.  However, the counterfactual generation task itself becomes more difficult as the level of confounding increases. Existing methods for counterfactual generation under confounding consider a fixed set of interventions (e.g., texture, rotation) and are not flexible enough to capture diverse data-generating processes. Given a causal generative process, we formally characterize the adverse effects of confounding on any downstream tasks and show that the correlation between generative factors (attributes) can be used to quantitatively measure confounding between generative factors. To minimize such correlation, we propose a counterfactual generation method that learns to modify the value of any attribute in an image and generate new images given a set of observed attributes, even when the dataset is highly confounded. These counterfactual images are then used to regularize the downstream classifier such that the learned representations are the same across various generative factors conditioned on the class label. Our method is computationally efficient, simple to implement, and works well for any number of generative factors and confounding variables. Our experimental results on both synthetic (MNIST variants) and real-world (CelebA) datasets show the usefulness of our approach.

Under review as a conference paper at ICLR 2023
COUNTERFACTUAL GENERATION UNDER CONFOUNDING
Anonymous authors
Paper under double-blind review
ABSTRACT
A machine learning model, under the influence of observed or unobserved con- founders in the training data, can learn spurious correlations and fail to gener- alize when deployed. For image classifiers, augmenting a training dataset using counterfactual examples has been empirically shown to break spurious correla- tions. However, the counterfactual generation task itself becomes more difficult as the level of confounding increases. Existing methods for counterfactual gen- eration under confounding consider a fixed set of interventions (e.g., texture, ro- tation) and are not flexible enough to capture diverse data-generating processes. Given a causal generative process, we formally characterize the adverse effects of confounding any downstream tasks and show that the correlation between generative factors (attributes) can be used to quantitatively measure confounding between generative factors. To minimize such correlation, we propose counter- factual generation method that learns to modify the value of any attribute in an image and generate new images given set of observed attributes, even when the dataset is highly confounded. These counterfactual images are then used to regu- larize the downstream classifier such that the learned representations are the same across various generative factors conditioned on the class label. Our method is computationally efficient, simple to implement, and works well for any number of generative factors and confounding variables. Our experimental results on both synthetic (MNIST variants) and real-world (CelebA) datasets show the usefulness of our approach.
INTRODUCTION
confounder variable that causally influences two or more variables that are not necessarily directly causally dependent (Pearl 2001). Often, the presence of confounders in data-generane getarannnn process is the reason for spurious correlations among variables in the observational data. The bias caused by such confounders is inevitable in observational data, making it challenging to identify invariant features representative of a target variable (Rothenhäusler et al. 202T Meinshausen & Bühlmann 2015: Wang et al. 2022). For example, the demographic area individual resides in often confounds the race and perhaps the level of education that individual receives. Using such observational data, if the goal is to predict an individual's 's salary, machine learning model may exploit the spurious correlation between education and race even though those two variables should ideally be treated independent variables. Removing the effects of confounding in trained machine learning models has shown to be helpful in various applications such as zero or few-shot learning, disentanglement, domain generalization, counterfactual generation, algorithmic fairness, healthcare,
etc. (Steer et al. 2019 Kilbertus et al. 2020 Atzmon et al. 2020 2hao et al 2020 Yue et al. 2021
Sauer & Geiger
1 Goel al. 2021 Dash et 2022 Reddy al. 2022: Dinga et al. 2020).
In observational data, confounding may be observed or unobserved and can pose various challenges
in learning models depending on the task. For example, disentangling spuriously correlated features using generative modeling when there are confounders is challenging (Sauer & Geiger 2021 Reddy cet al., 2022; Funke al. 2022). As stated earlier, a classifier may rely on non-causal features to make predictions in the presence of confounders (Schölkopf al. 2021). Recent years have seen a few efforts to handle the spurious correlations caused by confounding effects in observational data (Träuble et al.,, |021] Sauer& & Geiger 2021 Goel et al.) 2021 Reddy et al,, 2022). However, these methods either make strong assumptions on the underlying causal generative process or require strong supervision. In this paper, study the adversarial effect of confounding in observational data

---

Under review as a conference paper at ICLR 2023
on a classifier's performance and propose mechanism to marginalize such effects when performing data augmentation using counterfactual data. Counterfactual data generation provides a mechanism to address such issues arising from confounding and building robust learning models without the additional task of building complex generative models.
The causal generative processes considered throughout this paper are shown in Figure Aa). We assume that a set of generative factors (attributes) Z₁. Zz,..., Zn (e.g., background, shape, texture) and label Y (e-g., cow) cause real-world observation X (e.g., an image of cow particular background) through an unknown causal mechanism (Peters et al. 20176). To study the effects of confounding, we consider Y. Z1, Zz..... ., Zn, be confounded by set of confounding variables Ci..... Cm (c.g., certain breeds of cows appear only in certain shapes or colors and appear only in certain countries). Such causal generative processes have been considered earlier for other kinds of tasks such as disentanglement [Suter et al. (2019): Von Kügelgen et al. (202T): Reddy et al. (2022).
The presence of confounding variables results in spurious correlations among generative factors in the observed data, whose effect we aim to remove using counterfactual data augmentation.
A related recent effort by (Sauer & Geiger 2021] proposes Counterfactual Generative Net- works (CGN) to address this problem using data augmentation approach. This work as- sumes each image to be composed of three Independent Causal Mechanisms (ICMs) (Pe- ters et al.] (017aa) responsible for three fixed factors of variations: shape, texture, and back- ground (as represented by Z1, Zz. and Z3 in Figure bb). This work then trains generative model that learns three ICMs for shape, texture, and background separately, and combines them in deterministic fashion to generate observa- tions. Once the ICMs are learned, sampling im- these mecha-
C1
mm
Z1
Ze
Y
Z1
Z2
Z3
X
X
X g(Y, Zu Z...) (0) CONIC
x= s0Y, 2, (1- (Z)2) (6) CGN
(a) causal data generating process con- sidered in this paper (CONIC Ours): (b) causal data generating process considered in CGN & Geiger 2021).
Figure 1:
Sauer
ages by making interventions nisms give counterfactual data that can be used along with training data to improve classification results. However, fixing the architecture to specific number and types of mechanisms (shape, tex- ture, background) is not generalizable, and may not directly be applicable to settings where the number of underlying generative factors is unknown. It is also computationally expensive to train different generative models for each aspect of an image such as texture, shape or background.
In this work, we begin with quantifying confounding in observational data that is generated by an underlying causal graph (more general than considered by CGN) of the form shown in Figure ([a). We then provide a counterfactual data augmentation methodology called CONIC (COunterfactual geNeratlon under Confounding). We hypothesize that the counterfactual images generated using the proposed CONIC method provide a mechanism to marginalize the causal mechanisms responsible for spurious correlations (i.e., causal arrows from C, to Zj for some i, i..). We take a generative mod- eling approach and propose a neural network architecture based on conditional CycleGAN (Zhul et al. (017) to generate counterfactual images. The proposed architecture improves CycleGAN's ability to generate quality counterfactual images under confounded data by adding additional con- trastive losses to distinguish between fixed and modified features, while learning the cross domain translations. To demonstrate the usefulness of such counterfactual images, we consider classification as downstream task and study the performance of various models unconfounded test set. Our
key contributions include:
We formally quantify confounding in causal generative processes of the form in Fig a),, and study the relationship between correlation and confounding between any pair of generative factors.
We present a counterfactual data augmentation methodology to generate counterfactual instances of observed data, that can work even under highly confounded data (~ 95% confounding) and provides a mechanism to marginalize the causal mechanisms responsible for confounding.
We modify conditional CycleGAN to improve the quality of generated counterfactuals. Our method is computationally efficient and easy to implement.
Following previous work, we perform extensive experiments well-known benchmarks three MNIST variants and CelebA datasets - to showcase the usefulness of our proposed methodology in improving the accuracy of downstream classifier.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Counterfactual Inference: (Pearl 2009). in his seminal text on causality, provided three-step pro- cedure for generation of counterfactual data instance, given an observed instance: (i) Abduction: abduct/recover the values of exogenous noise variables; (ii) Action: perform the required interven- tion; and (iii) Prediction: generate the counterfactual instance. One however needs access to the underlying structural causal model (SCM) to perform the above steps for counterfactual genera- tion. Since real-world data do not come with an underlying SCM, many recent efforts have focused on modeling the underlying causal mechanisms generating data under various assumptions. These methods then perform the required intervention on specific variables in the learned model to generate counterfactual instances that can be used for various downstream tasks such as classification, fair- ness, explanations etc. (Kusner al. 2017, Joo & Kärkkäinen 2020 Denton et al., 2019; Zmigrod
et al. 2019 Pitis et al.) 2020 Yoon et al. 2018 Bica et al. 2020; Pawlowski et al. 2020
Generating Counterfactuals by Learning ICMs: In more recent effort, assuming any real-world image is generated with three independent causal mechanisms for shape, texture, background, and compovition mechanism of the first three, (Sauer & Geiger 2021) developed Counterfactual Gen- erative Networks (CGN) that generate counterfactual images of a given image. CGN trains three Generative Adversarial Networks (GANs) (Goodfellow et al. 2014b) to learn shape, texture, back- ground mechanisms and combine these three mechanisms using a composition mechanism as g( ((shape, texture, background) = shape @ texture + (1 shape) @ background where @ the Hadamard product. Each of these independent mechanisms is given an input of noise vector u and a label y specific to that independent mechanism while training. Once the independent mechanisms are trained, counterfactual images are generated by sampling label and a noise vector correspond- ing to each mechanism and then feeding the input to CGN. Finally, a classifier is trained with both original and counterfactual images to achieve better test time accuracy, showing the usefulness of CGN. However, such deterministic nature of the architecture is not generalizable to the case where the number of underlying generative factors are unknown and it is computationally infeasible train generative models for specific aspect of an image such as texture/background.
The spurious correlations among generative factors have been considered in disentangiement (Funke et al. 2022 von Kügelgen et al. (021). The gen- eral idea in these efforts is to separate the causal predictive features from non-causal/spurious pre- dictive features to predict an outcome. Our goal is different from disentanglement and we focus on the performance of a downstream classifier instead of separating the sources of generative factors. Traditional data augmentation methods such as rotation, scaling, corruption, etc. (Hendrycks et al.) 2020 Devries& & Taylor 2017 Zhang et al. 2018 Yun et al.) 2019) do not consider the causal gen- erative process and hence they can not remove the confounding in the images via data augmentation (e.g., color and shape of an object can not be separated using simple augmentations). We hence focus on counterfactual data augmentations that is focused on marginalizing the confounding effect caused by confounders.
Disentanglement and Data Augmentation:
A similar effort to paper is by (Goel et al. 2021) who use CycleGAN to generate counterfactual data points. However, they focus on the performance of a subgroup (a subset of data with specific properties) which is different from our goal of controlling confounding in the entire dataset. Another recent work by (Wang et al., 2022) considers spurious correlations among generative factors and uses CycleGAN to generate counterfactual images. Compared to these efforts, rather than using CycleGAN directly, we propose a CycleGAN-based architecture that is optimized for controlled generation using contrastive losses.
Applications of Counterfactuals: Augmenting the training data with appropriate counterfactual data has shown to be helpful in many applications ranging from vision to natural language tasks (Joo) & Kärkkänen, 2020) Lample et al. [2077: Kusner et al. 2017: Kaushik et al.) 2019; Dash et al. 2022). (Joo & Kärkkäinen (0220) identified existing biases in computer vision APIs deployed in the real world by Amazon, Google, IBM, and Clarifai by looking at the differences made by those APIs on counterfactual images that differ by protected/sensitive attributes (e.g., race and gender). Using locally independent causal mechanisms, (Pitis et al.) 2020) augmented training data with counterfactual data points in a model-free reinforcement learning setting. Here, the idea is to use any two factual trajectories of an episode and combine the two trajectories at particular point in time to generate the counterfactual data point, which will then be added to the replay buffer. Independently factored samples are essential to get plausible and realistic counterfactual instances.
3

---

Under review as a conference paper at ICLR 2023
3 INFORMATION THEORETIC MEASURE OF CONFOUNDING
Background and Problem Formulation: Let 1 .2..... Znn be set of random variables denot- ing the generative factors of an observed data point X, and Y be the label of the observation X. Each generative factor Z, (e.g., color) can take on a value form a discrete set of values { =; (c.g., red, green etc.). Let the set S {Y, Zi..... Znd generates N real-world observations { through an unknown causal mechanism g. Each X; can be thought of as observation generated using the causal mechanism g with certain intervention on the variables in the set S. Variables in S may potentially be confounded by set of confounders C = {C1..... Cm that denote real-world confounding such as selection bias. Let D be the dataset of real-world observations along with cor- responding values taken by (Y, Z1, ..... Zn Causal graph in Figure aa) shows the general form of this setting. From a causal effect perspective, each variable in S has a direct causal influence on the observation X (e.g., the causal edge Z; + X) and also has non-causal influence on X via the confounding variables Ci...., C. (e.g., Z, + C + Zz + X for some C3 and Zz). These paths via the confounding variables, in which there is an incoming arrow to the variables in S, are also referred to as backdoor paths (Pearl 2001]. Due to the presence of backdoor paths, we may observe spurious correlations among the variables in S in the observational data D.
In any downstream application where D is used to train a model (c.g., classification, disentanglement etc.), it is desirable to minimize or remove the effect of confounding variable to ensure that a model is not exploiting the spurious correlations in the data to arrive at decision. In this paper, we present a method to remove the effect of such confounding variables using counterfactual data augmenta- tion. We start by studying the relationship between the amount of confounding and the correlation between any pair of generative factors in causal processes of the form shown in Figure [a).
Definition 3.1. No Confounding (Pearl 2009). In a causal directed acyclic graph (DAG) G EV.E), where V denotes the set of variables and & denotes the set of directed edges denoting the direction of causal influence among the variables in V. an ordered pair (Z,, Z,); Zi, Zj E V is unconfounded if and only if p'Z; = zz|do(Z, Z3 = 2,)) = Zi = 25 Z5; = 2,), Vzi, Zj. Where
do(Z, 24) denotes intervention to the variable Z, with the value 2, This definition can also be extended to disjoint sets of random variables.
provides the notion of no confounding, however, to quantify the notion of confound- ing between a pair of variables, we consider the following definition that relates the interventional distribution p( (: and the conditional distribution pl 7g (z,).
Definition 3.1
Definition 3.2. (Directed Information (Raginsky) 2011 Wieczorek & Roth 2019). In a causal directed acyclic graph (DAG) G = (V,E) where V denotes the set of variables and & denotes the set of directed edges denoting the direction of causal influence among the variables in V, the directed information from a variable Z₁ E V to another variable Z,- E V is denoted by I(Z + Zz). It is defined as follows.
= Bp(2,2,) log ,(Z,)Z,) 22))
T(Z + = = DKL (22
(1)
Using Definitions 3.1 and
3.2 it is easy to see that the variables Z, and Zj are unconfounded
O. Non zero directed information I(Z, - Zx) entails that, ZZ, * p(Z, do( (2)) and hence the presence of confounding (if there is no confounder, pl (Z, should be
and only if TZ, Zx)
equal to pl [Z |o( (d((()) Z,))). Also, it is important to note that the directed information is not symmetric (i.e., I(2 + 2₁) + I(2; + Z()) (Jiao et al. 2013). We use this fact in defining the measure of confounding below. Since we need to quantity the notion of confounding (as opposed no confounding), we use directed information to quantify confounding as defined below.
Definition 3.3. (An Information Theoretic Measure of Confounding.) ) In causal directed acyclic graph (DAG) g = (V,E), where V denotes the set of variables and & denotes the set of directed edges denoting the direction of causal influence among the variables in V, the amount of confound- ing between a pair of variables Z, E V and Zj E V is equal to I(2, + 2z) + I(2; + Zz).
Since directed information is not symmetric, we define the measure of confounding to include the directed information from one variable to the other for a given pair of variables Z,. Z,. We now relate the quantity I(Z, + Z,) + I(Z, + Zz) with the correlation between generative factors so that it is easy to quantify the amount of confounding in observational data. Before that, we present the following proposition which will be used in the proof of the subsequent proposition.
4

---

Under review as a conference paper at ICLR 2023
Proposition 3.1. In causal processes the form [a), the interventional distribution p(Zzldo(Z,)) same as the marginal distribution p(z)).
Proof. In causal processes of the form [a), let C" denote the set of all confounding variables that
are part of some backdoor path from Zi to Zz. That is C" = [CIZ, + C - Zz} for some i, j. Then we can evaluate the quantity pZZ; |do (2,))
((2)do((2)) = Σ (")11,,
"xpl""")=<<ll)))ppp = Σ
p(Z.)
Where the first equality is because of the adjustment formula (Pearl 2001) and the second equality because of the fact that Y collider causal graph 1 a) and hence conditioned on C', Zí is independent of Zj.
O
Proposition 3.2. For causal generative processes of the form A the correlation between a pair of generative factors (Z,,2) Zz) proportional the amount of confounding between Zu and Z,.
Proof. Expanding the quantity I(Z; Zz) + 1(Z) + Z:), we get the following,
[(z + Σ₁) + I(2 + 2,) = Ez, log( 2,Z
p 2Z,)
+ Ez,,, log(
P(Z:
Z ddo dd(())))
log( ,((,)22))((2,)11)
log (( Z, (z)()
= E2.22
E2,2,
p [(2,(o0(22))(((2]1(((23)))
p(Z:)p(Z
(2)
Toog( p( [(2212,))(222))((2222))))))
p(Z,, Zz)p((z;, 2i)
= EZ,,,
= B2
log(
p(2))((2))((2)p((33
p(Zz)2p(Z,)²
log( p(Z,,2,) = (z) p(Zi)p(Z,)
= 2 x IZ,
Where T(Z4; Zz) is the mutual information between Z; and Zj- The third equality is due Propo- sition 3.1 Since non-zero mutual information implies positive correlation, we see that the amount of confounding between Z, and Zj is directly proportional to the correlation between Z; and Zj. Hence, we use the correlation as measure of confounding between generative factors in the causal processes of the form a).
Using the connection between the confounding and correlation in causal graph [a), objective is to generate counterfactual data such that the resultant dataset after augmentation looks similar to the data obtained from causal process where there is no confounding between generative factors (i.e., no paths of the from Z; + Cj + Vi. k). Equivalently, our counterfactual data generation algorithm removes the spurious correlations between generative factors by marginalizing the causal arrows C₁ + Zj for some j. j. To understand how counterfactual instances break the correlations, consider the following definition.
Definition 3.4. (Counterfactual (Pearl 2009)). Given an observed instance X whose generative the counterfactual instance X' of X (generated using the 3-step counterfuctual inference procedure) disfered from X w.r.l. the generative factor Zi, is an instance whose generative factors Zi....; Zz...., Zn take on the val- Zn. 21.....21 2/ ......, Zn. Here Z, 's value is changed from 2; to 2 through external intervention do(2, 100%2=2..
factors Zi..... Ze...., Zn take on the values 21,...;2...... ...... 2i.....
If observe spurious correlation between two generative factors Z,, Z, when they take the values 2; and 2j respectively, generating counterfactual instances w.r.t. Z, with the intervention do(Z, 2/} and adding the counterfactual instances original data breaks the correlation be- tween Z,, Zy. With this idea, we now present our algorithm to generate counterfactual images in a systematic manner remove confounding from observational data.
4 CONIC: METHODOL.OGY
Our goal is to remove the effect of confounding in the observational data downstream task such as classification. To this end, we propose a way to systematically generate counterfactual data that marginalize the effect of any confounding edge C₁ + Z,j in Fig. (a) explained below.
5

---

Under review as a conference paper at ICLR 2023
Removing The Confounding Effect of C. Zy: In the causal graphs of the form T for paths of the form Z3 < C + Zi, we call the edges C, + Z₁ and C + Zi as confounding edges since together, their existence is the reason for confounding in the data. Also, let (2f, =f) is one pair of attribute values taken by the variable pair (Z Z1) under extreme confounding (e.g., in the training set of colored MNIST dataset, correlation coefficient of 0.99 between color and digit is observed such that whenever color is red, digit is etc.). To remove the effect of the confounding edge C, + Zj w.r.t the another confounding edge C; + Z (recall that confounding between Zj. Zi present if and only if there exists a pair of causal arrows C, + Zj and C, + Zi for some i: due to this reason we consider the confounding effect of the confounding edge C; - Zjj w.r.t. another confounding edge C; > 71). we consider two subsets Ti, T2 of the observational data D which constructed follows. Ti consists of the set of instances for which Zj * 2 and Zi = 2/. T2 consists of the set of instances for which Zj- 2 and Zi = 2/. The size of T is usually much smaller than the size of T2 because of high correlation between Zj and Zi ge.g.. there are more red 7's than non-red 7's).
Now, we learn a mapping M from the set Ti to the set T2 that changes the attribute Z, while fixing the value of Zi at z/. That is, for any given instance X E T, for which Zj + x4.. M maps X to a different instance X' which the value of the generative factor Z is changed to z (e.g.. M takes red 9 as input and returns red 7 as output). This mapping .M can be thought of as a function per- forming the 3-step counterfactual inference: learning the underlying generative factors, performing the intervention do(Z; z)} ) and then generating the counterfactual instance X'. Now, given an instance X for which Zj + 2/ and Zi + z/, using M,, we can generate counterfactual instance X' in which Z; = = and Z₁ # z/- These counterfactual instances, when augmented with the original observed dataset 'D removes the effect of the confounding edge C; + Zj w.r.t. the edge C; - Zi- That is, the counterfactual instances, when augmented with original data, breaks the correlation be- tween Zj and Zi. This process can now be repeated systematically for each confounding edge to generate counterfactual instances that remove the spurious correlations. Such augmented data points which differ from original data points w.r.t. only one feature (e.g., if original image is male with blond hair color, augmented image is same male with black hair color) are referred as coupled sets by (Goel et al. 2021 , images generated by causal essential transformations by (Wang al. 2022 The overall procedure to generate counterfactual instances is summarized in Algorithm |
Earlier works use CycleGAN gen- erate counterfactual images that dif- fer from original image by single attribute/feature (Wang et al. [022] Goel al., [021). Given two do- mains/sets of images that differ w.r.t. only one generative factor Z,. a Cy- cleGAN can learn to translate be- tween the two domains by chang- ing the attribute value of Zy. In this case, one can think of CycleGAN as function performing the required in- tervention Z, and generating coun- terfactual instance without modeling the underlying causal process. Con- cretely, CycleGAN is an architecture used to perform unsupervised domain translation using unpaired images. In CycleGAN, generator G1 first transforms a given image X from a domain/set Ti into X' so that X' appears to come from another domain/set T2 such that certain features from input X are preserved in the output X'. A discriminator Dr, then classifies whether the translated image X' is original (i.e., sampled from T2) fake (i.c.. generated by G1). A second generator G2 transforms the image X' back to original image X to ensure that G1 is using the con- tents of X to generate X'. The same procedure is repeated to translate images from domain T2 into domain T,. The loss function of CycleGAN can be written as follows.
L
L2
DT,
G2
T
Target Attribute X" (e.g., blond True)
Target Attribute (c.g., blond False)
Figure 2: Architecture of the proposed modified conditional CycleGAN generate counterfactual images. Pre-trained modules are shown in green color and target attribute is shown in blue color. Note that, for simplicity, we only show one pass of conditional CycleGAN (translation from Ti to T2) in this figure.
LCydeGaAN LGAN(G1, Dr,, X, X') + LGAN (G2,(G2, Dr, X', x) + Eeyete(Gr G2)
(3)
6

---

Under review as a conference paper at ICLR 2023
Algorithm I: Counterfactual Generation to Remove the Effect of Confounding Edge C; + Zz
Result: Counterfactual images that remove the confounding effect caused by the edge C₁ - Zz Input: D {Xi ) Nodes [Z,[c, Zj&C, + Zt}
Initialize: cf www.geseesss =
for each Z: Nodes do
T = [ E D 2 + kZ = 21 \ divide data into two domains using attribute values
T- [X E DI = &Z, )
M = conditioonl CyclecAN(T, T2)
\" Learn M to translate T1 1o T2 *
Fact factuallInne
= (X DIZ, + 2 + 2/} >* Pick factual images from train set *
CFs = M((FactallIIggs)
Generate counterfactuals using M **!
Append CFs to cf images end for return cf cf_images
Where LGAN is simple Generative Adversarial Network (GAN) (Goodfellow et al. 2014a) loss and Layele is cycle consistency loss measuring how well the output of G2 is matching with the original input X. For example, Leyete G2) = Ex~D (xxD]]])(([((X))
X|]1] can ensure that G2 (X)) = X. In this work, to learn the mapping function M, we use conditional variant of CycleGAN to leverage the supervision in terms of attribute values, For each generator, along with
input, also feed desired target attribute as shown in the Figure 2
To improve the quality of counterfactual images generated by conditional CycleGAN under extreme confounding, propose modification to conditional CycleGAN as detailed below. As discussed earlier, X', the output of G1. can be thought of as a counterfactual image of X. When changing the feature Z, of X, we keep the feature Zi fixed. That is, the representation for Zj in both X and X' should be different and the representation for Z in both X and X' should be same. To ensure this, shown in Figure 2 along with two generators G1, G2 and a discriminator Dr, that are part of conditional CycleGAN, we add two pre-trained discriminators L1, L2 (shown in green color in Fig. 2). L1 takes two images X, X' as input and returns high penalty the representation of Zj is similar in X' and small penalty otherwise. L2 takes two images X, X' as input and returns high penalty the representation of Z₁ different and small penalty otherwise. Thus, our overall objective to generate good quality counterfactual images is to train the modified conditional CycleGAN by minimizing the following objective.
Leonic = LCydeGAN + (o-ooootrsttineeee (L1(X), (G) + (la) Loontrattioe (Li (X'), Li (G2(X')) + Lenntrastive (La (X'),,,, La (2(('))))
(4)
Where a is a hyperparameter and Ceontrastive is the contrastive loss (Hadsell et al. 2006). For a pair of images (X, X'), Lcontraetive defined as follows.
Ccontrastive (X,X') = AD2 + (1 (1- A) max(e - -
(5)
Where A = 1 Xf, belong to same class (or have same attribute values), A 0 if Of X, to different classes (or have different attribute values). D is the distance between the representations of X, X' (e.g.. Euclidean distance). e is the margin of error allowed between two representations of the images of different classes. L, and L2 are pre-trained models and the parameters of L1 and L2 are fixed. That is. the loss values returned by Lcontrastive are only used to update the trainable parameters of conditional CycleGAN.
X' belong
To measure the goodness of counterfactual generation under confounding using Algorithm [ we consider the classification task on the unconfounded test set as a downstream task. Let Dang (X,, Y.) M be the dataset consisting of original data points from D and corresponding counterfactual data points. Usual empirical risk minimizer minimizes the following loss over D.
A Downstream Task Image Classification:
Larm == E(x,)= E(x,)) (Xf((X)))))
(6)
Where is cross entropy loss. Using Doug, we minimize the following loss Laug:
Laug B(X,a))Duus
(()a)))))(((((())....)))()())))) ((f((X))))
7

---

Under review as a conference paper at ICLR 2023
Model
CM-MNIST
DCM-MNIST
WLM-MNIST
CelebA
ERM
46,41 0.81%
43.31 + 2.30%
28.28 + 0.70%
70.64 + 6.93% 6.93%
CGAN
41.86 1.79%
30.66 + 3.86%
17.50 + 0.85%
70.99 2.35%
CVAE
49.58 + 1.50%
41.99 + 1.10%
34.19 + 1.58%
71.50 + 1.82%
C-B-VAE
51.22 1.00%
33.90 + 1.87%
74.29 - 0.65%
AugMix
47.36 0.01%
44.85 0.02%
26.30 + 1.30%
71.93 + 4.64%
CutMix
20.44 + 1.22%
23.10 2.98%
12.08 + 1.59%
73.66 + 0.76%
IRM
55.25 + 0.89%
49.71
+ 0.71%
50.26 + 0.48%
72.30 + 2.71%
CGN
42.15 3.89%
47.50
2.18%
43.84 + 0.25%
69.25 + 0.29%
CONIC
65.57 0.34%
92,41 + 0.26%
77.72++ 1.00%
79.56
1.28%
Table 1: Test set accuracy results on MNIST variants and CelebA
To further improve the performance classifier using Daug for each pair of images X,, X, we minimize the contrastive loss Leontrastive (X,, X2) on the logits in the final layer. Now, the final objective to optimize for classification task to minimize the following loss.
Laug + XB ) DDus))
( [Ccontrastive (X, X3)]
(8)
Where À 0 a regularization hyperparameter.
EXPERIMENTS AND RESULTS
In this section, we present the experimental results on both synthetic (MNIST variants) and real world (CelebA) datasets. Having access to the ground truth generative factors (i.e., Zı...., Zn) of images,we artificially create confounding in the training data and we leave test data to be uncon- founded (i.e., no correlation among generative factors). We compare CONIC with various baselines including traditional Empirical Risk Minimizer (ERM), Conditional GAN (CGAN) (Goodfellow et al. 2014a), Conditional VAE (CVAE) (Kingma & Welling 2013), Conditional- VAE (C-3- VAE) Higgins et al.] 2017), AugMix (Hendrycks et al., 2020), CutMix (Yun et al. 2019), In- variant Risk Minimization (IRM) (Arjovsky al. 2019). and Counterfactual Generative Networks (CGN) Sauer & Geiger 202T). To check the goodness of each of these methods, we check how well the performance of the downstream classifier on the test improved using the augmented
images.
MNIST Variants: We construct the following three synthetic datasets based on MNIST dataset (Lecun et al. 1998) and its colored, texture, and morpho (where the digit
thickness is controlled; Fig. 3) variants (Arjovsky
al.] 2019 Castro et al.] 2019
Sauer & Geiger) (021): (i) colored morpho MNIST (CM-MNIST), (ii) double col-
ored morpho MNIST (DCM-MNIST), and (iii) wildlife morpho MNIST (WLM-
MNIST). We consider extreme confounding among generative factors as explained below. For the experimental results shown in Table D in the training set of CM- MNIST dataset, the correlation coeffi cient between digit label and digit color r(label, color) is 0.95 and the digits from O to 9 are thick (see Figure B). That is, r(label, thin) [0,1,2.3,4] else r(label, thick) the training set of DCM-MNIST dataset, digit label, digit color, and background color jointly take a fixed set of values 95% of the time. That is, rllabel, color) rccolor, background) = r(label, background) digits from 5 to 9 thick. In the training set of WLM-MNIST dataset digit shape, digit texture, and background texture jointly take a fixed set attribute values 95% of the time and the digits from O to 4 are thin and digits from to 9 are thick.
3
6
7
/
3
4
1
6
7
7
5
3
3
3
/
8
7
8
6
9
thin and digits from 5
1
1
8
7
8
9
6
6 6
8 8
8
7
if the digit in 1. In
2
6
Figure 3: Left: sample thin morpho MNIST images and corresponding labels. Right: Sample thick morpho MNIST images and corresponding labels.
0.95 and the digits from 0 to are thin and
8

---

Under review as a conference paper at ICLR 2023
In all of these MNIST variants, test set images unconfounded (e.g., in the test set of DCM- MNIST. any digit can be thin or think, can be in any background color, can be in any foreground color). In these experiments, under extreme confounding, ages that break the confounding among generative factors. We evaluate models on this grounds by training classifier using the augmented data and testing it the unconfounded test data. Table [ shows the results in which CONIC outperforms all the baselines. See Appendix for comparison of augmented images by various baselines. Coninc uses only 10000, 15000, 15000 counterfactual images in CM-MNIST, DCM-MNIST, and WLM-MNIST experiments as augmented images re- spectively to get improved performance. The regularization hyperparameter À in Equation set to 0.5 for all MNIST experiments.
goal is to generate counterfactual im-
CelebA:
Unlike MNIST variants, CelebA (Liu al. 2015) dataset implicitly contains confounding
(e.g., the percentage of males with blond hair is different from the percentage of females with blond hair, in addition to the difference the total number of males and females in the dataset). In this experiment, consider the performance of a classifier trained on the augmented data that predicts hair color given an image. Our test set is the set of males with blond hair.
We train models the train set and test the performance on the set of males with blond hair. Since the number of males with blond hair is very low in the dataset (approxi- mately 4% of males have blond hair), show that the augmenting the train set with only 10000 images of males with blond hair improves the perfor- mance over baselines (see Table [ whereas other baselines require more than 50000 augmented images to get minor improvement over ERM. Given male image with non- blond hair, CONIC generates the counterfactual image with blond hair without changing the male attribute (see Figure 4 for sample counterfactual images). We also note that the deterministic models such as CGN fail when they are applied to different task where the number and type of generative factors are not fixed and are difficult to separate (e.g., CelebA). CGN results in table T are obtained with only 1000 counterfactual images as augmented data points. When we increase the number of counterfactual instances, performance of CGN reduces further.
Figure 4: Top: CelebA original images of males with non- blond hair color. Bottom: Counterfactual images of males with blond hair generated using Algorithm
Time Complexity Analysis: Apart from its simple
methodology,
CONIC
brings
additional advantages in terms of computing time quired to train the model that generates counterfac- tual images. As shown in Table 2 the time required to run our method to generate counterfactual images w.r.t. a generative factor Z, is significantly less than CGN that learns deterministic causal mechanisms as discussed in Section 2 Even though we used Cycle- GAN in this work, for the cases where the number of generative factors are more, StarGAN (Choi et al.) 2018) can be used to minimize the time required to learn the mappings from one domain to another
Dataset
CONIC
CGN
CM-MNIST
2.76 + 0.19
103 + 1.50
DCM-MNIST
2.22 + 0.01
103 + 2.04
WLM-MNIST
1.22 = 0.01
111 2.50
Table 2: Run time (in minutes) of CONIC compared CGN on MNIST variants
domain (Wang et al. 2022 Goel et al. 2021)
CONCLUSIONS
We studied the adverse effects of confounding in observational data on the performance of clas- sifier. We showed the relationship between confounding and correlation in the causal processes considered, and we proposed methodology to remove the correlation between the target vari- able and generative factors that works even when the dataset is highly confounded. Specifically, we proposed a counterfactual data augmentation method that systematically removes the confounding effect rather than addressing the confounding problem through random augmentations. Using the generated counterfactuals leads substantial increase in downstream classifier's 's accuracy. That said, we observed that the counterfactual quality can still be improved, which will be interesting future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Diffusion causal models for counterfactual estimation, 2022.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization, 2019.
Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik. A causal view of compositional zero-shot recognition. In NeurlPS, 2020.
loana Bica, James Jordon, and Mihaela van der Schaar. Estimating the effects of continuous- valued interventions using generative adversarial networks. In H. Larochelle, M. Ran- zato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), NewrIPS, volume 33, pp. 16434 16445, Curran Associates, Inc., 2020. URL http:://wcceedigss..erriisccoopp bea5955b30836 25b100077555442225555ppapppp
Daniel C. Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker. Morpho- MNIST: Quantitative assessment and diagnostics for representation learning. JMLR, 20(178), 2019.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. CVPR, 2018.
Saloni Dash, Vineeth N Balasubramanian, and Amit Sharma. Evaluating and mitigating bias in image classifiers: A causal perspective using counterfactuals. In WACV, 2022.
Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru. Detecting bias with genera- tive counterfactual face attribute augmentation, 2019.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. ArXiv, abss/100.0445 04552, 2017.
Richard Dinga, Lianne Schmaal, Brenda W.J.H. Penninx, Dick J. Veltman, and Andre F. Marquand. Controlling for effects of confounding variables on machine learning predictions. bioRxiv, 2020.
Christina M Funke, Paul Vicol, Kuan-Chieh Wang, Matthias Kuemmerer, Richard 7emel, and Matthias Bethge. Disentanglement and generalization under correlation shifts. ICIR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022. URL https:
Karan Goel, Albert Gu, Yixuan Li, and Christopher Re Model patching: Closing the subgroup performance gap with data augmentation. In ICLR, 2021.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David FardeeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014a.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014b. URL
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In ICLR, 2020. URL
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017. URL huts://oneeeeewwwwweroommmmnssss
10

---

Under review as a conference paper at ICLR 2023
Zhiting Hu and Li Erran Li. A causal lens for controllable text generation. In A. Beygelzimer, Y. Dauphin, P. Liang, J.d Wortmam Vaughan (eds.), Adiances in Neural Information Processing Systems, 2021. URL https///wppuuwoewweee//////eo/tp
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-aceuraa In Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, 2022.
Jiantao Jiao, Haim H Permuter, Lei Zhao, Young-Hap Kim, and Tsachy Weissman. Universal es- timation of directed information. IEEE Transactions on Information Theory, 59(10):622--6242, 2013.
Jungseock Joo and Kimmo Kärkkäinen. Gender slopes: Counterfactual faimesss for computer vi- sion models by attribute manipulation. Proceedings of the 2nd International Workshop on Fairness, Accountability, Transparency and Ethics in Multimedia, FATE/MM '20, pp. 1-5, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 97814503814 482.. doi: 10.1145/3422841 1.3233332221133333333. URL httpp::/doi..rrg/1011115555 145/342284 3,3333333 3423533
Nitish Joshi and He He. An investigation of the (in)effectiveness of counterfactually augmented data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Pupers), pp. 3668-3681, Dublin, Ireland, May 2022. Association for Computational Linguisties.
Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. Learning the difference that makes difference with snanterfactually-aggmmeett data. arXiv preprint 19X9 2019.
Niki Kilbertus, Philip J Ball, Matt J Kusner, Adrian Weller, and Ricardo Silva. The sensitivity of counterfactual fairness to unmeasured confounding. In UAI, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 2013.
Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In
I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett
(eds.), NeurlPS, volume 30. Curran Associates, Inc., 2017. URL https//prooeedngg..errii..
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic DENOYER, and Mare' Aurelio Ranzato. Fader networks:manipulating images by sliding attributes. In NeurIPS, 2017.
Y. Lecun, L. Bottou, Y. Bengio. and P. Haffner. Gradient-based learning applied document recog- nition. Proceedings of the IEEE, 96(11):227--233 1998. doi: 10. 10.1109/5.72699
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Nicolai Meinshausen and Peter Bühlmann. Maximin effects in inhomogeneous large-scale data. The Annals of Statistics, 43(4):1801 1830, 2015.
Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and
H. Lin (eds.), NeurIPS, volume 33, pp. 857-869. Curran Associates, Inc., 2020. URL https:
Judea Pearl. Direct and indirect effects. In UAI, pp. 411-420. 2001.
Judea Pearl, Causality. Cambridge university press, 2009.
J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017a
Peters, D. Janzing, and B. Scholkopf. Elements of Causal Inference: Foundations and Learning Algorithms. Adaptive Computation and Machine Learning series. MIT Press, 2017b.
11

---

Under review as a conference paper at ICLR 2023
Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), NeurlPS, volume 33, pp.
Curran Associates, Inc.. 2020. URL https:/proceedings. 134a7e68a8 Paapeee
Maxim Raginsky. Directed information and pearl's causal calculus. In 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 958-965, 2011. doi: http9/AAlletto.2211.610000
Abhavaram Gowtham Reddy, Benin I. Godfrey, and Vineeth N Balasubramanian. On causally dis- entangled representations. In AAAI, 2022.
O. Ronneherger, P.Fischer, and T. Brox. U-net: Convolutional networks for biomedical image seg- mentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), volume 9351 of LNCS, pp. 234-241. Springer, 2015.
Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, Jonas Peters, et al. Anchor regres- sion: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B, 83(2): 215-246, 2021.
Axel Sauer and Andreas Geiger. Counterfactual generative networks. In ICLR, 2021.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke. Nal Kalchbren- ner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. CoRR, abs/2102.11107, 2021,
URL https//arri.orr//b2211....111
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In ICML, 2019.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In ICML., 2021.
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. sulf-supe learning with data augmentations provably isolates content from style. In NeurlPS, 2021.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 34:16451-16467, 2021.
Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. Out-of-distribution generalization with causal invariant transformations. CVPR, 2022.
Aleksander Wieczorek and Volker Roth. Information theoretic causal effect quantification. Entropy, 21(10), 2019.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. GANITE: estimation of individualized treatment effects using generative adversarial nets. 6th ICLR, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
https://ppenrevie..e//ffommm ByKWBWAA
Zhonggi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Counterfactual zero- shot and open-set visual recognition. CVPR, 2021.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp. 6022-6031, 2019. doi: 00.100//CCV.2019.0066
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In ICLR, 2018. URL http::///pppmmeeee...ee//mmmtt/d
12

---

Under review as a conference paper at ICLR 2023
Qingyu Zhao, Ehsan Adeli, and Kilian M Pohl. Training confounder-frees deep learning models for medical applications. Nature communications, 11(1):1-9, 2020.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data augmen- tation for mitigating gender stereotypes in languages with rich morphology, 2019.
13


Reviews:
[{"review_id": "aqX7bxdBPwx", "reviewer": "Reviewer_YnUp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on building robust models using counterfactual data. The main idea is to modify the cycle-GAN architecture to use group-annotations in the training data as additional supervision to learn to modify specific features. This method seems to yield useful improvements to the task of robust modeling when using generative models. The proposed method seems to be able to handle confounding better than other methods just condition on feature labels.", "summary_of_the_review": "While I think the methodological contributions to generating counterfactual data are useful, I'm not convinced about the value of the method for the larger ML goal of building robust models.", "strength_and_weaknesses": "Strengths:\n1. The experiments show sizeable improvements to other counterfactual generation type methods.\n2. The method looks complex but it is intuitively simple and I imagine the implementation to not be hard.\n\nWeaknesses:\n\n1. The paper does not discuss the value of counterfactual data for robust modeling, compared to other methods that do not build generative models using group-annotations in training. The authors should discuss this papers like https://arxiv.org/pdf/2110.14503.pdf and explain the value of their work within the larger ML goal of building robust models, especially without the additional and complex task of building a generative model.\n2. The paper also does not discuss other work pointing out potential issues with counterfactually augmented data. Discussing those also would be worthwhile: https://arxiv.org/pdf/2107.00753.pdf\n\nI would be happy to revisit my score if the authors can address these.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_KxVl254bo", "reviewer": "Reviewer_jJ3z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the important confounding issue in image classification problems from the observational data. The authors built the connection between the confounding and the spurious correlation, by assuming that the generating factors are causally independent of each other. This result allows them to add counterfactual data augmentation in CycleGAN to remove the correlation between the target variable and generative factors. Numerical studies show their promising prospect over existing methods.", "summary_of_the_review": "Overall, this paper considers an interesting problem of confounding issues in image classification and is easy to follow. The theoretical results are sound but with a very strong assumption. The algorithm is built upon an existing approach (CycleGAN) while necessary comparison to show the uplifts is missing.", "strength_and_weaknesses": "*Strength*\n\n1. This paper considers an important confounding issue in image classification from observed data.\n\n2. The authors provide sound theoretical results to study the relationship between correlation and confounding.\n\n3. Numerical studies show their promising prospect on well-known benchmarks.\n\n4. The paper is well-written in general and easy to follow.\n\n*Weaknesses*\n\n1. My main concern is that this paper primarily assumed that $Z_i$ are causally independent of each other, i.e., there is no causal link among different $Z$s, in the DAG illustrated in Figure 1a. This means the only correlation among $Z$s is spurious owing to their common confounders $C$. In this way, they are able to utilize their proposed CONIC to remove the Effect of confounding edge. Yet, the causal independence among $Z$s is a quite strong assumption and needs further justification. The authors may also comment on the worse case when this assumption is violated.\n\n2. The implementation details are not clear. For example, I wonder how the authors weighted the five different loss functions in Equation 4. Or are they equally weighted or using the same strategy in Equation 8? More specifications are required.\n \n3. The main methodology is built upon CycleGAN. The authors may benefit from decoupling their counterfactual data augmentation part from CycleGAN and show their method is generally applicable to different image classification methods. A very natural question is: how much has the proposed method improved CycleGAN? Why not include CycleGAN for comparison in the experiments of benchmark?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Nx_qQ0foTuu", "reviewer": "Reviewer_ZkeM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an algorithm to augment the classifier training with counterfactual examples. The proposed method modifies the CycleGAN algorithm to do this augmentation. Different from previous methods which could only deal with texture / shape, this algorithm could process any confounding effects to capture data-generation process.", "summary_of_the_review": "The idea of this work makes sense in such a problem. However, this work suffers from some unspecified points discussed above. I would of course consider raising the score if they could be addressed properly.", "strength_and_weaknesses": "## Weaknesses\n\n1. How to get the two datasets for CycleGAN training? In my opinion, this requires knowing the attribute difference between the two datasets, which is related to the prior of data generating process (DGP). Does this mean one need to know about the DGP? If so, is it fair to compare it with algorithms such as IRM who does not assume such knowledge? If not, I think there are many larger scale datasets that should be taken into account (e.g. DomainBed). If I miss any point here, please feel free to correct me! \n\n2. The datasets used in experiments are very spurious, in the sense that 95% of the data are spurious correlated. Therefore, I am curious to see how ERM perform if only trained with the rest 5% unbiased data, which I think is a more proper baseline if one knows which part of data is biased / spurious correlated.\n\n3. What is the connection between the theoretical discussion in Sec 3 and the proposed method? I don't see any usage of mutual information term in the proposed algorithm, which is a bit confusing. \n\n4. In Fig. 1(a), CONIC has multiple confounding factors $C_i$, which is considered as a major difference with previous work CGN. However, is this reflected in the design of the algorithm? Which part of the algorithm utilizes such an assumption?\n\n### Minors\nSome of the citations should use \\citet{} instead of \\citep{}.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EKOVXjrIl07", "reviewer": "Reviewer_8VNe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method called CONIC for generating counterfactual data under the presence of confounders, using a variant of CycleGAN where the training objective is augmented with a contrastive loss. The high-level idea is to have the CycleGAN generate a counterfactual X’ with respect to X such that it only modifies the attribute value of interest. The training data is then augmented with their corresponding counterfactual images, and this augmented dataset is used to train a downstream classifier which is evaluated on a test set without confounding. CONIC helps boost performance over existing baselines.", "summary_of_the_review": "The paper proposes a method to generate counterfactuals under confounding using a modified version of CycleGAN, but does not provide enough empirical evidence to explain why/how the method is working. The baselines CONIC outperforms appear weak and are not well-explained in the paper, so it's hard to tell how strong they are. The clarity of the paper can also be improved. I am happy to raise my score if the authors can address these issues, though, since I think this work is interesting and addresses an important problem.", "strength_and_weaknesses": "Strengths:\n- Counterfactual generation is an important problem, and although there are works that try to address this in simple settings (e.g. no confounding), the setting in the presence of confounding is not as well-studied despite being more practically relevant and useful.\n- I liked the earlier exposition about how the particular structure of the underlying causal mechanism (Figure 1a) directly allows for measuring confounding via mutual information estimation between the generative (latent) factors. It would have been really nice if the authors could have incorporated this into the experiments somehow (e.g. by looking at how the performance of CONIC is affected by the level of confounding as measured by this metric).\n\nWeaknesses:\n- I wasn’t able to find any further references back to directed information and how it could be used to measure the amount of confounding in the dataset. \n- The empirical results left much to be desired, especially with the lack of detail regarding the experimental setups and ablations. For example, while CONIC seemed to improve the accuracy of the downstream classifier, I was surprised to see how poorly other approaches performed, such as the CGAN. \n- I actually have a lot of clarification questions for the authors:\n  - Was the discrepancy in performance a function of the architecture used (was it comparable in size to the CycleGAN)? \n  - Were the conditional samples generated just purely by conditioning on each of the attributes in an attempt to “balance out” the training set? \n  - Also, how much of the performance boost could be attributed to the additional contrastive loss in Eq. 8 of the downstream classifier rather than just standard ERM training using the real+counterfactual data as in Eq. 7? \n  - In a similar vein, how much did each of the additional contrastive loss terms help in the CONIC training objective? \n  - What do the samples from the vanilla conditional generators look like? Did those baselines also use any additional forms of supervision, such as a modified version of Eq. 8 (modified since they are not exactly counterfactuals)?\n- ^ These details would be important to address in the final version of the paper, as it’s hard to tell what exactly the authors did and where the gains are coming from. This is the primary reason why it’s hard for me to recommend acceptance of the paper in its current form.\n- It would have been nice to see whether the quality of the counterfactual images that were generated had any effect on the downstream results. For example, it is not obvious that there is a clear relationship between image quality and usefulness to the downstream task (https://arxiv.org/pdf/1905.10887.pdf), and the generated images in Figure 4 actually do not appear to be very high quality. Would the authors comment on this? I would have expected the downstream classifier trained on such images to perform worse on the test set, since the counterfactual images appear to be pretty out-of-distribution when compared to the images in the training set.\n- The paper is also missing some relevant references (though these do counterfactual generation without confounding): \n  - https://arxiv.org/pdf/2201.09119.pdf\n  - https://arxiv.org/pdf/2202.10166.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject