Confounder Identification-free Causal Visual Feature Learning
Authors: Xin Li
Zhizheng Zhang
Guoqiang Wei
Cuiling Lan
Wenjun Zeng
Xin Jin
Zhibo Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Domain Generalization
Casual Learning
Front-door criterion
Confounder identification-free

Abstract: Confounders in deep learning are in general detrimental to model's generalization where they infiltrate feature representations. Therefore, learning causal features that are free of interference from confounders is important. Most previous causal learning based approaches employ back-door criterion to mitigate the adverse effect of certain specific confounder, which require the explicit identification of confounder. However, in real scenarios, confounders are typically diverse and difficult to be identified. In this paper, we propose a novel Confounder Identification-free Causal Visual Feature Learning (CICF) method, which obviates the need for identifying confounders. CICF models the interventions among different samples based on front-door criterion, and then approximates the global-scope intervening effect upon the instance-level interventions from the perspective of optimization. In this way, we aim to find a reliable optimization direction, which avoids the intervening effects of confounders, to learn causal features. Furthermore, we uncover the relation between CICF and the popular meta-learning strategy MAML, and provide an interpretation of why MAML works from the theoretical perspective of causal learning for the first time. Thanks to the effective learning of causal features, our CICF enables models to have superior generalization capability. Extensive experiments on domain generalization benchmark datasets demonstrate the effectiveness of our CICF, which achieves the state-of-the-art performance.

Under review as a conference paper at ICLR 2023
CONFOUNDER IDENTIFICATION-RRE CAUSAL VISUAL FEATURE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Confounders in deep learning are in general detrimental to model's generaliza- tion where they infiltrate feature representations. Therefore, learning causal fea- tures that are free of interference from confounders is important. Most previous causal learning-based approaches employ back-door criterion to mitigate the ad- verse effect of certain specific confounders, which require the explicit identifi- cation of confounders. However, in real scenarios, confounders are typically di- verse and difficult to be identified. In this paper, we propose a novel Confounder Identification-fri frree Causal Visual Feature Learning (CICF) method, which obvi- ates the need for identifying confounders. CICF models the interventions among different samples based on the front-door criterion, and then approximates the global-scope intervening effect based on the instance-level intervention from the perspective of optimization. In this way, aim to find reliable optimization di- rection, which eliminates the confounding effects of confounders, to learn causal features. Furthermore, we uncover the relation between CICF and the popular meta-learning strategy MAML (Finn et al. (017)) and provide an interpretation of why MAMI. works from the theoretical perspective of causal learning for the first time. Thanks to the effective learning of causal features, our CICF enables models to have superior generalization capability. Extensive experiments on domain gen- eralization benchmark datasets demonstrate the effectiveness of our CICF, which achieves the state-of-the-an performance.
INTRODUCTION
Deep learning excels at capturing correlations between the inputs and labels in a data-driven man- ner, which has achieved remarkable successes on various tasks, such as image classification, object detection, and question answering (Liu et al., [20T He al., 2016 Redmon et al.) 2016]] (He al.] 1017|Anoo et al. 2015). Even so, in the field of statistics, correlation is in fact not equivalent to causation (Pearl et al. 2016). For example, when tree branches usually appear together with birds in the training data, deep neural networks (DNNs) are easy mistake features of tree branches as the features of birds. A close association between two variables does not imply that one of them causes the other. Capturing/modeling correlations instead of causation is high risk of allowing various confounders to infiltrate into the learned feature representations. When affected by intervening ef- fects of confounders, network may still make correct predictions when the testing and training data follow the same distribution, but fails when the testing data is out of distribution. This harms the generalization capability of learned feature representations. Thus, learning causal feature, where the interference of confounders is excluded, is important for achieving reliable results.
As shown in Fig. [ confounders C' bring a spurious (non-causal) connection X CC- Y between samples X and their corresponding labels Y. A classical example to shed light on this is that we can instantiate Y.Y.C C as the sales volume of ice cream, violent crime and hot weather. Seem- ingly, an increase in ice cream sales X is correlated with an increase in violent crime Y. However, the hot weather is the common cause of them, which makes an increase ice cream sales to misleading factor of analyzing violent crime. Analogically, in deep learning, once the misleading features/confounder are captured, the introduced biases may be mistakenly fitted by neural net- works, thus leading to the detriment of the generalization capability of learned features. In theory, we expect DNNs to model the causation between X and Y. Deviating from such expectation, the interventions of confounders C make the learned model implicitly condition on C. This makes that

---

Under review as a conference paper at ICLR 2023
Gu
C2
X
(b) Back-door criterion
X
Z
Background Brightness
Viewpoint
(a) Examples of some confounders
(c) Front-door criterion
Figure 1: (a) Examples of some confounders, which may lead to learning biased features. (b) Back- door criterion in causal inference, where the counfunders are accessible. (c) Front-door criterion in causal inference, where the confounders are inaccessible.
the regular feature learning does not approach the causal feature learning. To learn causal features, previous studies (Yue et al. 2020 Zhang et al.) 2020; Wang et al., 2020b) adopt the backdoor cri- terion (Pearl et al.] (016) explicitly identify confounders that should be adjusted for modeling intervening effects. However, they can only exploit the confounders that are accessible and can be estimated, leaving others still intervening the causation learning. Moreover, in many scenarios, confounders are unidentifiable or their distributions are hard to model (Pearl et al., 2016).
Theoretically, front-door criterion(Pearr et al. 2016) does not require ideniffinggeeppliitty modeling confounders. It introduces an intermediate variable Z and transfers the requirement of modeling the intervening effects of confounders
X - to modeling the intervening effects of on Z + Y. Without requiring explicitly modeling confounders, the front-door criterion is inherently suitable for wider scenarios. However, how to exploit the front-door criterion for causal visual feature learning is still under-explored.
In this paper, we design Confounder Identification-Fi Causal visual Feature learning method (CICF). Particularly, CICF models the interventions among different samples based the front- door criterion, and then approximates the global-scope intervening effect based on the instance-level interventions from the perspective of optimization. In this way, we aim to find reliable optimization direction, which eliminates the confounding effects of confounders, to learn causal features. There two challenges
will address for CICF, How model the intervening effects from other samples on a given sample in the training process. 2) How to estimate the global-scope intervening effect across all samples in the training set to find suitable optimization direction.
As we know, during training, each sample intervenes others through its effects on network parame- ters by means of gradient updating, Inspired by this, we propose gradient-based method model the intervening effects on a sample from all samples to learn causal visual features. However, it intractable to involve such modeled global-scope intervening effects in the network optimiza- tion, which requires traversal over the entire training set and is costly. To address this, we pro- pose an efficient cluster-then-sample algorithm to approximate the global-scope intervening effects for feasible optimization. Moreover, revisit the popular meta-learning method Model-Agmostiee Meta-Learning (MAML) (Finn et al. 2017). We surprisingly found that our CICF can provide an interpretation why MAML works well from the perspective of causal learning: MAML tends to leam causal features. We validate the effectiveness of our CICF on the Domain Generalization (DG) (Wang et al. 2021: Zhou et al.) 2021a) task and conduct extensive experiments the PACS, Digits- DG, Office- Home, and VLCS datasets. Our method achieves the state-of-the-ard performance.
RELATED WORK
Causal Inference aims at pursuing the causal effect of a particular phenomenon by removing the interventions from the confounders (Pearl al. 2016). Despite its success in economies (Rubin 1986). statistics (Rubin) 1986, Imbens & Rubin 2013) and social science (Murnane & Willett
2010 big challenges present when it meets machine learning, i.e., how model the intervention
2

---

Under review as a conference paper at ICLR 2023
from the confounders and how establish the causal model. A growing number of works have moved a step forward by taking advantage of the back-door criterion (Pearl et al. 2016) on various tasks, e.g., few-shot classification (Yue et al. 2020), vision-language task (Wang et al., 2020b), domain adaptation (Yue et al. 2021). class-incremental learning (Hu et al. 2021), and semantic segmentation (Zhang et al., 2020p. Limited by the back-door criterion, most of them are required to identify and model the distributions of the confounders, However, this may be challenging in the real world because confounders are typically diverse and usually appear implicitly. To get rid of the dependency on confounders, Yang et al. (021). D). for the first time, propose to utilize the front-door criterion establish a causal attention module for vision-language task. However, still requires the modeling of intervention in the testing stage, which is complicated.
In contrast, this work is the first attempt to apply the front-door criterion for learning causal visual features by considering the intervention among samples. Ours improves the generalization ability of DNNs from the optimization perspective and is confounder identification-fre
Model Generalization plays a prominent role for DNNs to be applied in real-world scenarios. To
improve the performance on the testing dataset which has distribution shift (Sun et al. (2016) with training data, various domain generalization (DG) (Muandet et al. 2013; Zhou et al.] 202 Wang
ct al. 2021 Shen et al., 2021, Wei et al., 2021) methods have been proposed. In general, these methods can be divided into three categories, i.e., domain-invariane representation learning, data or feature manipulation, and meta-learning. The first category intends to learn domain-invarian fea-
tures that follow the same distributions (Muandet et al. 2013 Li al., 2018b Taori et al. [2020] Li
iet al.) 2018d] Motiian et al. 2017; Mahajan et al.) 2021 Jin et al. 2020). The second category aims improve the generalization ability of models through enriching the diversity of source domains,
either in image space (e.g., CrossGrad (Shankar et al. 2018). DDAIG (Zhou et al., (020a) and M-
ADA QQao et al., 2020),, or feature space (MixStyle (hou et al. 2021c) and RSC (Huang et al., 2020 ). Another new line of DGs utilize meta-learnin learning as training strategy (Zhao et 20121
1 Liu ct al. 2020 Li al., 8018a 2020 Wei et al., 2021 Balaji et al.) 2018). MAMI. (Finn et al. 2017) takes the advantage of meta-learnn learning to find good parameters initialization for fast adaptation to new tasks. Following MAML, Li et al. 8a); Dou et al. (2019); Li et al. (2020); Balaji et al. parame- ters regularizer during training. Other variants of DGs exploit episodic training (Li et al. 2019) and
(2018) introduce meta-learning into DG to simulate domain shift or learn
ensemble learning (Zhou et al. 2021b Seo et al. 2020 Cha et al. 2021.
In this paper, from a new perspective, we propose a scheme for model generalization termed as Confounder Identification-free Causal Visual Feature Learning (CICF).
PROPOSED METHOD
In this section, we first depict a supervised learning process in a causal graph (Pearl 2009b), and uncover the stumbling effects of confounders which prevent the achievement of high generalization capability of models in Sec. 3.1 Then, in Sec. 3.2. based on the front-door criterion, we elaborate our Confounder Identification-fre Causal Visual Feature Learning CICF) from two perspectives, respectively as how to model mutual intervening effects between different instances and how to ap- proximate such intervening effects from the global scope. Furthermore, we describe our CICF which alleviates the intervening effects from the optimization perspective in Sec. 3.2 In Sec. 3.3 we un- cover the relation between CICF and the popular meta-learning strategy MAML (Finn et al. 2017), and provide an interpretation of why MAML works from the theoretical perspective of causal inference.
3.1 PROBLEM DEFINITION AND ANALYSIS
Given training dataset with input and label pairs Y), the goal of training Deep Neural Net- works is to learn/capture the causation between input samples and prediction labels Y, i.e., the conditional probability P(Y]do(X)), As shown in Fig. 2 (a), parameterize the network as ip and separate it into two successive parts, i.e., and
DNNs capture label-associated features which not necessarily the casual ones due to the in- tervening effects of confounders, such as background, brightness, and viewpoint. We denote the intermediate features and confounders Z and C, respectively. Fig. Pillustrates the relations in
3

---

Under review as a conference paper at ICLR 2023
Model
do operation
7.
Sub-madel
Sub-madel
Y
X
Z
Y
X
Y
h
R
(a)
(b)
Figure 2: Causal graph in supervised learning. (a) A network/mooel ip consists of sub-models and f. X and Y denote the input and prediction labels, respectively. (b) Causal graph where do operation aims remove the intervening from confounder C.
causal graph. Intervened by the confounders, the conditional probability P(Y|X) learned by the model ip actually involves two paths, i.e., X + Z Y and X +
Y. X + Z Y denotes the expected causal effect from the input samples X to their prediction labels Y. The path X <- C - denotes the non-causal correlation between X and Y due to their common cause C, which may introduce biases into the learning of P(Y|X) and thus affect the generalization capability of model yo.
Intuitively, it is crucial to get rid of the harmful bias from those confounders for causal feature learning. In the literature of causal inference (Pearl et al.) 2016). the confounding effects from con- founders can be removed through the do operation (Pearl et al. 2016) by cutting off the connection from C' to X, as illustrated in Fig. (b). With the definition of do operation, the real causation from X to Y can be formulated by P(Y)do(X)), The objective of CICF is learn features representation conforming to P(Y]do(X))
Back-door criterion. In previous works Yuue et al. 2020, Wang al. 2020b Hu al. 202
when C' is identifiable, the back-door criterion (Pearl et al. 2016 is typically utilized to achieve the do operation as:
= >
(1)
which acquires access to the distributions of all confounders E C. However, in many scenarios, the intervening effects caused by unobservable implicit factors. It is not feasible to identify the distribution of C during training, limiting the usage of the back-door criterion.
Front-door criterion. For unidentifiable confounders, the Front-door criterion PPaal et al. 2016) provides us with a more practical alternative to Estimate the Intervening Effect, called FEIE, es- chewing the identification of confounders C. Specifically, it introduces an intermediate variable Z to help assess the effect of X on i.e., P(Y ((Y)))(X), which can be formulated as:
do = xx)) Σ P = Σ 2())) ZEX
where = h(X), & E denotes sample from training data. Note that the effect X on Z is identifiable because they have no common causes. In other words, there is no backdoor path from X to Z. Thus, we have P(Z z|do(X x)) = 2| =x).
Front-door criterion attractive for eliminating interventions. However, it is still under-explored for visual feature learning, where there is a lack of a simple and practical mechanism to exploit this theory for enhancing the generalization capability of models.
3.2 CONFOUNDER IDENTIHICATION-FRE CAUSAL VISUAL FEATURE LEARNING
In this paper, we aim to achieve Confounder Identification-ffee Causal Visual Feature Learning, obviating the need for confounder identification. As indicated by Eq. (2). thanks to the front- door criterion, we do not need to identify and explicitly model confounders C. Despite this, still imposes challenge on how to accurately model the term Σ3 P(Y Z 2, in the network training process. We treat the first part of &p as the model h to obtain the intermediate variable 2, ie.,, z h(x). Because the parameters of h are fixed in the inference stage and h(x)) is known
4

---

Under review as a conference paper at ICLR 2023
given any X, P(Z x|X = I)) is equal to 1 Thus, re- wrrit Eq.
as:
P
Σ #EX
P(Y b(),
(3)
where Z = -( 3) denotes the mutual intervening effects to the causation path Z + Y from another sample J. With the summation operation, Eq. 3 represents the global intervening effects accumulated from all samples in the training data. We will describe the instantiations of P(Y 2 h(x), 3) and the accumulation in Eq. respectively as below.
A Gradient-basse based Instantiation of FEIE. Referring to the practices in prior works (Yue et al, 2020 Wang et al. 2020a) upon the back-door criterion, a straightforward method for modelling P(Y Z h(xx), 2) is to directly concatenate = h(x) and the feature of I E X before feeding them into f. However, this method would easily lead to a trivial solution once the information of T is ignored by the layers of the neural networks. In contrast, we propose to explicitly model the
intervening effects of I on Z + Y with a gradient-based instantiation. We notice that, in the training process, the influence of one instance others be reflected on the parameters updating with the gradient obtained based on this instance. Therefore, for given sample x, we propose to explicitly model the intervening effects P(Y Z h(x), .) of another sample è through f as:
(4) So and fez denote the model / before and after the parameters updating respectively, Ia denotes the calculated gradient with respect to the sample è and its label ÿ. L and a represent the loss of cross entropy and learning rate, respectively. Incorporating Eq. 4 into Eq. (3). we have Eq. (5) as below to explicitly eliminate the interventions of all samples on the sample x: as:
P - h(x), - foz (Z h())), where 03- 0 aga, ga (eL()
PoY|do(X = x)) = Σ (Z= REX
(5)
Global-scope Intervening Effects Approximation.
With the above introduced gradient-based in- stantiation, the globally accumulative intervening effects from all the training samples can be esti- mated by traversal X, which, however, is time- and memory-consuming practice. To achieve an efficient estimation in the global scope, we apply the first-order Taylor's expansion on Eq.
P(Y|do(X= = x)) =
agg Voll-oooWooolotttoooooollllllll
2EX
~ Sa(h(v)) - al Σ oa P()) BEX
(6)
Eq. (6) reveals that the key estimating P(Y)do(X)) lies in computing the global-scope gradient 01 Laex P(t) over all ā E X accumulated via weighted sum with P(ū) as the weight.
However, is intractable to directly compute the global-scope gradient gi by traversing over all the training samples. Alternatively, we can traverse over a sampled small subset that shares the similar data distribution to that of all the training data, As we know, when the training data are unbalanced and diverse, random sampling of small subset would result in bias that mismatches the distribution of the dataset, leading to an inaccurate estimation of the global-scope gradient. To better estimate the data distribution and thus approach the global-scope gradient, we propose sampling strategy dubbed as clustering siaeteerinntthen--eee sampling. More discuusson/anallsiis can be found in the Appendix [A.] Concretely, first cluster the training, samples of each class in the dataset into K clusters with K-means algorithms (Pelleg et al. 2000) and totally obtain K"t clusters for the whole training data.
It is noteworthy that found the samples in each cluster usually have similar gradient directions in optimization. Thus we represent each cluster with fewer samples randomly sampled from the same cluster, avoiding traversing over all the data. Then, the global-scope gradient gt can be approximated
with weighted sum over the sampled M =Kk Ne samples from Kt clusters:
-k=N
kt N
91
M
k=
We provide proof in Appendix A.3 that this satisfies the front-door criterion.
5

---

Under review as a conference paper at ICLR 2023
where N& is the number of instances sampled from the k-th cluster (being proportional to the size of this cluster), G denotes the gradients of the sample Jy,k E Kg. Combined with Eq. (7)
rewrite Eq 66)
P( = =)) Re { ogt oo
(8)
Causal Visual Feature Learning. Based the above theoretical analysis and the proposed inter- vention approximation strategy, the intractable causal conditional probability P(Y [do X)) can be approximated based on Eq. without requiring the identification of confounders C. Actually, the Eq. 8 can be viewed as the first-order Taylor's s expansion of fe-og, (h(z)). Thus, we have:
P(Y]do(X = x)) = se-agy (h(x)),
here let 0+ 0-a (the parameters of f). which are updated with the global-scope gradient gt- The output of a model is thus denoted as Bdo(x) = fe, (h(z)), which has been aware of the global-scope interventions from all other samples on the current sample æ based on such global-scope gradient updated model (i.e., fo,). Then, we can train an unbiased model f to learn the causal visual features with the loss of cross-entropy:
LCICF = Loe (fe(()())), v),
(10)
xEX
where is the corresponding ground-truth label for x. The overall algorithm of Confounder Identification-free Causal Visual Feature Learning described in Alg. of Appendix.
Note that the training dataset, thereby being capable of approximating the global-scope gradient more ac- curately. We have theoretically analyzed that
is better than random sampling to approach the distribution of
has a more minor stan- dard error (SE) for estimating the distribution of all the training data than random sampling, i.e. S Eours < SErandon in the Appendix [..]] This demonstrates that our clustering teeee-hee-ssee then-sampling is a more efficient and more accurate strategy to estimate the data distribution and then the global-scope gradient.
3.3 DISCUSSION
In this section, we will provide an analysis and comparison between our CICF and MAML Finn ct al. 2017). For the first time, we interpret why MAML works from a causal learning perspective, which is supported by our analysis in the previous subsections.
In the seminal work MAML (Finn et al., 2017) 7). Finn et al. propose a agnosti meta-learning strategy that treats batch of data as meta-train and another batch of data meta-test for optimiza- tion. Particularly, given sets of data [Di,- D' corresponding to T tasks {Ti)?= where D and Di denote meta-train and meta-test data respectively, the loss function of MAML for optimiza- tion can be represented as:
LMAMI. = (Joz, (fo,, (Xio), Y(),
(11)
where 0r, refers to the parameter virtually updated with the gradient gf,- calculated on D i.e. 8r +- f- agf,, which is treated as meta-train task. The optimization Dí, is treated meta-test task, where the parameters are updated as 8% + 8 - a aVDLMMML- They interpret why MAML works from the perspective that it can provide a good parameter initialization which robust for fast adaptation to new data. However, there is a lack of theoretical analysis and support in Finn et al. 2017). Based on our analysis in Section 3.2, for the first time, we have a new understanding of the previous uses of MAML (Finn et al.
2017 7 Li et al. 2018a) (see Eq. (17)) from the perspective of causal inference. (fer, (Xic), Yi.) Eq. 11I actually models the intervention from meta-train data D to meta-test data Dt_ within the task and endeavors to eliminate such local data modeled intervention. However, as revealed by theoretical analysis in Section 3.2 learning reliable causal features requires the capturing and modeling of interventions from all the samples (i.e., global inter- ventions). There is no such solution in the previous works while provide practical and efficient one to model and eliminate the global-scope interventions in this paper. This enables reliable causal feature learning and promotes the achievement of higher generalization capability of models.

---

Under review as a conference paper at ICLR 2023
350
Marrory
0L.
01.3
60.
Ea1th 6:0
dutter
Figure 3: Ablation study on the influence from (a) batch-size, (b) the number clusters, for the global-scope intervention estimation, and (c) different choices of Z (c.f. Fig. 7 iZ ..... Zs} denotes the features from the shallow layer to the deep layer. All results are averaged over four domains PACS.
Table Classification accuracy (%) of different DG methods on PACS with ResNet- and Digits- DG with a CNN backbone (Zhou et al. 2021c). f: our reimplemented results using the official code different from the ones reported in the original paper.
PACS
Digits-DD DG
Method
Avg.
MINIST
MINIST-M
SVHN
SYN
Avg.
MMD-AAE
75.2 72.7 96.0 64.2
77.0
96.5
58.4
65.0
78.4
74.6
CCSA
80.5 76.9 93.6 66.8
79.4
95.2
58.2
65.5
79.1
74.5
JiGen
79.4 75.3 96.0 71.6
80.5
96.5
61.4
63.7
74.0
73.9
CrossGrad
79.8 76.8 96.0 70.2
80.7
96.7
61.1
65.3
80.2
75.8
MLDG
79.5 77.3 94.3 71.5
80.7
94.7
60.3
61.5
75.4
72.6
MASF
80.3 77.2 95.0 71.7
81.1
MetaReg
83.7 77.2 95.5 70.3
81.7
RSC
83.4 80.3 96.0 80.9
85.2
MatchDG
81.3 80.7 96.5 79.7
84.6
MixStyle
83.0 78.6 96.3 71.2
82.3
96.5
63.5
64.7
81.2
76.5
FACT
85.4 78.4 95.2 79.2
84.5
97.9
65.6
72.4
90.3
81.5
ERM
77.0 75.9 96.0 69.2
79.5
95.8
58.8
61.7
78.6
73.7
ERM+MAML
77.0 74.5 94.8 72.1
79.6
96.0
63.1
65.0
81.1
76.5
ERM+CICF
80.7 76.9 95.6 74.5
81.9
95.8
63.7
65.8
80.7
76.5
ERM
82.5 74.2 95.4 76.5
82.1
96.1
65.0
73.0
84.6
79.7
ERM*+MAMI
81.8
73.2 94.8
75.7
81.4
96.2
67.0
74.0
84.1
80.3
ERM*
84,2 78.8 95.1
83.2
85.3
95.6
68.8
76.5
86.0
81.7
4 EXPERIMENTS
To validate the effectiveness of our CICF, we apply it on the Domain Generalization (DG) (Zhou al. 2021a Wang et al. 2021) task, where the models are expected learn the unbiased causal features to be generalized to different domains. We describe the datasets and implementation details Sec. 4.1 Then we clarify the effectiveness of each component our CICF in Sec. 4.2 and compare with previous methods in Sec. Finally, we qualitatively show that CICF captures the causal features Sec.
4.1 DATASETS AND IMPLEMENTATION DETAILS
Datasets. We evaluate our method on four commonly used benchmark datasets (i.e., PACS (Li et al.
2017), Digits-DG, Office-Home (Venkateswara et al. 2017) and VLCS (Torralba & Efros 2011 for Domain Generalization. 1) PACS (Li et al. (017)) contains images from four domains, i.e., Photo (P), Art painting (A), Cartoon (C), and Sketch (S). Each domain consists of images in seven object
categories. 2) Digits-DG is composed of four digit datasets, including MINIST (LeCun et al.) 1998
MINIST-M (Ganin & Lempitsky 2015), SVHN (Netzer et al., (011)) and SYN (Ganin & Lempitsky 2015). Each dataset is regarded
domain, which contains ten digit categories from zero to nine.
3) Office-Home divided into four domains, including Artistic, Clipart, Product and Real World.
There are 65 object categories related the scenes of office and home. 4) VL.CS (Torralba &
7

---

Under review as a conference paper at ICLR 2023
Hoose
Dog
Guitar
Person
Giralle
House Clephanl
Horse
Dog
Guilar
Person
Girafle
llouse
Clephant
Inputs
ERM
IRM
Ours
Visualization of the Grad-CAMs w.r.t. classification task for domain generalization. The first row of left/right panels shows the images from sketch/art painting (Art) domains in PACS. The second, third and fourth rows show the Grad-CAMe of ERM, IRM (Arjovsky et al. 2019) and our CICF.
Figure 4:
Efros T covers images of five object categories from four domains, i.e., PASCAL VOC 2007,
LabelMe, Caltech, and Sun datasets. Following previous DG methods (Li et al. 2019 Zhou et al.
2021c et al. 2018b a). we evaluate methods under the
protocol, where one
domain is used for testing while others for training.
Implementation Details. For PACS and Office- Home, we take ResNet18 (He et al. 2016) pre- trained on ImageNet (Deng et al. (2009) backbone, following Zhou al. (2021c): Carlucci al. 2019). We also take the ResNet-50 pretrained on ImageNet as the backbone for PACS, following Huang al. (2020). For VLCS, geNet (Deng et al. 2009) as our backbone, which is the same as (Matsuura & Harada 2020 Dou al.]|[1000 (2019. For Digits-DG, we adopt the model architecture used previous works (Zhou al., 20206] 2021c). We cluster three clusters within each class in training datasets. All reported results are averaged among six runs. More implementation details can be found in the Appendix A.5
take AlexNet (Krizhevsky
al.] 2012) pretrained Ima-
4.2 ABLATION STUDY
Effectiveness of CICF. Our proposed CICF enables models to have superior generaliza- tion ability by effectively causal feature learning. We compare our CICF with the popular meta-learning strategy MAML, which aims to explore
the commonly optimal optimization direction for all tasks (i.e., different domains), on two baselines. 1) ERM:
Method
A
C
$ Avg.
MatchDG
85.6 82.1 97.9
78.8
86.1
training models only on source domains with simple data augmentations including flip and translation. 2) ERM*:
RSC
87.9 82.2 97.9 83.4
87.8
FACT Fish
89.6 81.8 96.8 84.5
88.2
training models ment (Cubuk et al. (018). The results on PACS are shown in Table [ with baseline ERM, ERM+CICF out- performs ERM by 2.4 % in accuracy without known do- main labels, while MAML only achieves the improve- ment of 0.1% with known domain labels. Moreover, with another baseline, ERM* outperforms ERM" by 3.2% and 2.0% on PACS and Digits-DG respectively. However, MAMI. not robust for different baselines and does not work for ERM"
source domains with AutoAug-
85.5
ERM
88.0 78.8 98.2 81.7
86.7
ERM ERM'+CICP 89.7 82,2 97.9 86.2
89.0
Table 2: Classification accuracy (%) of different DG methods on PACS with ResNet-50.
Different ways to estimate global-scope intervening effects. To estimate the global-scope inter- vening effects efficiently and accurately, we propose a native is the naïve random mini-batch sampling. As shown in Fig, Bla, the classification accuracy increases along with the increased sampling batch-size. This is because increasing batch-size will result in better estimation of the global-scop scope intervention with lower SE. However, the memory overhead increases drastically simultaneously. In contrast, our clustering-thet tchen-sampling outper- forms the naive random sampling by 0.7% with lower memory utilization (the batch-size is fixed as
then-sampling strategy. An alter-
8

---

Under review as a conference paper at ICLR 2023
256 for PACS in our case). We also conduct experiments the influence of the number of clusters KT for all datasets. The results in Fig. (()) show that more clusters result in better performance, and the performance is saturated when K 21 for PACS (i.e., three clusters for each class). Because more clusters lead to lower intra-cluster variance d₁ of the htth cluster and more accurate estimation of the global-scope intervention with lower SE, which is derived in the Aopendix/A.1... And three clusters for each class are enough for global-scope intervention modeling.
Different choices of Z. We explore the effects of different choices of Z from shallow layer features to deep layer features. The experiments and analysis are shown in Appendix A.4, which reveals that shallow features are better.
4.3 COMPARISON WITH STATE-OF- ARTS
PACS. As shown in Table our proposed CICF on top of ERM* achieves the best per- formance on PACS, outperforming the SOTA works including MixStyle (Zhou et al. 2021c), RSC (Huang et al., (020)) MatchDG MMaajaan al. 2021a) and FACT Xu et al. (2021) More- over, ERM' +CICF is clearly better than previous gradient- based and meta-learning based methods, e-g., CrossGrad (Shankar et al. 2018), MLDG (Li et al., 2018a) MetaReg (Balaji al. 2018
and MAMI. (Finn et al. 2017). thanks to the more accurate global intervening effects modeling in CICF. On the most challenging domain sketch, our ERM*+CICE outperforms all previous methods by a large margin (2 2.3%), ) which demonstrates that CICF can learn the causal visual features by removing the influence from confounders efficiently. The experiments on PACS with ResNet-50 are shown in Table where our CICF is clearly superior to the SOTA methods, e.g., RSC (Huang et al.) MMahajan al. 2021a), FACT (Xu et al. 2021) and recent Fish (Shi et al. 2021).
2020 MatchDG
Digits-DG. As shown in Table. I our ERM*+CICF achieves the best performance, outperforming the MAML. (Finn et al. (017)) p by 1.4% and the recent MixStyle (Zhou et al.) (021c) by 5.2%. For the most challenging domains (i.e., MNIST-M and SVHN), ERM*+CICF improves the accuracy of ERM by 3.8% and 2.5% respectively.
Office-Home. As shown in Table 3 of Appendix, our ERM +CICF achieves the best performance of 66. 2%, exceeding the previous state-of-the-art method MixStyle (Zhou et al. 2021c) by 0.7%. Furthermore, +CICF improves ERM* on the challenging Clipart by margin of 3.2%.
VLCS. The results on VLCS are shown in Table of Appendix, where our ERM*+CICE achieves the state-of-ther ohe-tr-rrttttttt result, outperforming the second best MASF (Dou et al. 2019) by 0.6%, which is based on meta-learning. Moreover, CICF brings the improvement of 1.7% for ERM
4.4 FEATURE VISUALIZATION
To validate that CICF actually learns the causal visual features, we visualize and compare the Grad-CAM (Selvaraju et al., 2017) of ERM and ERM+CICF in Fig. Intervened by the con- founders (e.g., background), ERM easily focuses on the object-irrelevant regions (i.e., non-causal features), impeding the model's generalization ability. In contrast, thanks to the guidance of CICF, ERM+CICF prone to focus more on the foreground object regions (i.e., causal features). Further, we visualize the learned features by t-SNE (Saito et al. 2019) on Digits-DG in the Appendix A.6
5 CONCLUSION
In this paper, we propose a novel method dubbed Confounder Identification-fre Causal Visual Fea- ture Learning (CICF) for learning causal visual features without explicit identification and exploita- tion of confounders. Particularly, motivated and based the front-door criterion, we model the interventions among samples and approximate the global-scope intervening effects for causal vi- sual feature learning. Extensive experimental results domain generalization validate that CICF can help a model to achieve superior generalization capability by learning causal features, without the need of identifying confounders. Our method is generic which should be applicable to other fields such as NLP. We leave this as future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pp. 2425-2433, 2015.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprini arXiv: 1 2019.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta-regularization. In NeurlPS, volume 31, pp. 998-1008, 2018.
Fabio M Carlucci, Antonio D' Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In CVPR, pp. 2229-2238, 2019.
Junbum Cha, Hancheol Cho, Kyungjae Lee, Seunghyun Park, Yunsung Lee, and Sungrae Park. Domain generalization needs stochastic weight averaging for robustness domain shifts. arXiv preprint arXiv:2 102.08040 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, Icee, 2009.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generaliza- tion via model-agnostic learning of semantic features. In NeurlPS, volume 32, pp. 6450-6461, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pp. 1126-1135. PMLR, 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, pp. 1180-1189, PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domaan-adverssrial training of neural net- works. The journal of machine learning research, 17(1):2096-0300, 2016.
Subir Ghosh. Elements of sampling theory and methods, 2002.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In ICCV, pp. 2961- 2969, 2017.
Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in class-incremental learning. In CVPR, pp. 3957-3966, 2021.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV, pp. 124-140, 2020.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sci- ences. Cambridge University Press, 2015.
Xin Jin, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. Feature alignment and restoration for domain generalization and adaptation. arXiv preprint arXiv:2006. /2009, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, volume 25, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(1) 1):2788-2324, 1998.
10

---

Under review as a conference paper at ICLR 2023
Da Li, Yongxin Yang. Yi-Zhe Song. and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, pp.
2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales, Learning to generalize: Meta- learning for domain generalization. In AAAI, 2018a.
Li Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song. and Timothy M Hospedales. Episodic training for domain generalization. In ICCV, pp. 1446-1455, 2019.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Sequential learning for domain generalization. In ECCV, pp. 603-619, Springer, 2020.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, pp. 5400-5440 5009, 2018b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624 0144,,39,44 2018c.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, pp. 624 639, 2018d.
Quande Liu, Qi Dou, and Pheng-Ann Heng. Shape-aware meta-learning for generalizing prostate mri segmentation to unseen domains. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 475 475. Springer, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, pp. 7313-7324, PMLR, 2021a
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In ICML, pp. 7313-7324. PMLR, 2021b.
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains. In AAAI, volume 34, pp. 11749-1 2020.
Saeid Motiaa, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep super- vised domain adaptation and generalization. In ICCV, pp. 5715-5725, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML, pp. 10-18, 2013.
Richard J Murnane and John B Willett. Methods matter: Improving causal inference in educational and social science research. Oxford University Press, 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Judea Pearl. Causal inference in statistics: An overview. Statistics surveys, 3:96-146, 2009a.
Judea Pearl. Causality. Cambridge university press, 2009b.
Judea Pearl, Madelyn Glymour, and Nicholas Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
Dan Pelleg, Andrew W Moore, et al. X-means: Extending k-means with efficient estimation of the number of clusters. In ICML,, volume 1, pp. 727-734, 2000.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. CVPR, pp. 2020.
11

---

Under review as a conference paper at ICLR 2023
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, pp. 779-788, 2016.
Donald B Rubin. Statistics and causal inference: Comment: Which ifs have causal answers, Journal of the American Statistical Association, (1(6)))661-96, 1986.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza-
tion. arXiv preprint (tXXi::/11.0070000 2019.
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong-weak distribution align- ment for adaptive object detection. In CVPR, pp. 6956-6965, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, pp. 618-626, 2017.
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learn- ing to optimize domain specific normalization for domain generalization. In ECCV, pp. 68 833. Springer, 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via grosdegr training. ICLR, 2018.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards outtof-distribution generalization: survey. arXiv preprint arXiv:2108. 13624, 2021.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937, 2021.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, 2016.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In NeurIPS, 2020.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, pp. 1521-1528. IEEE, 2011.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR. pp. 5018-5027, 2017.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: A survey domain generalization. In I/CAI, 2021.
Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense r-enn. In CVPR. pp. 10760-10770, 2020a.
Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense representation learning via causal inference. In CVPR Workshops, pp. 378-379, 2020b.
Guoqiang Wei, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation. In CVPR, pp. 16643-16653, 2021.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pp. 6502-6509, 2020.
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A based framework for domain generalization. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Patlern Recognition, pp. 14383-14392, 2021.
12

---

Under review as a conference paper at ICLR 2023
Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for vision-language tasks. In CVPR, pp. 9847-9885 2021.
Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Interventional few-shot learning. In NeurIPS, 2020.
Zhongqi Yue, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Transporting causal mechanisms for unsupervised domain adaptation. In ICCV, pp. 8599-8608, 2021.
Dong Zhang, Hanwang Zhang, Jinhui Tang, Xiansheng Hua, and Qianru Sun. Causal intervention for weally-supervised semantic segmentation. In NeurIPS, 2020.
Yuyang Zhao, Zhun Zhong, Fengxiang Yang, Zhiming Luo, Yaojin Lin, Shaozi Li, and Nicu Sebe. Learning generalize unseen domains via memory-based multi-source meta-learning for person re-identification. In CVPR, pp. 6277-6286, 2021.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain adversarial im- age generation for domain generalisation. AAAI, volume 34, pp. 13025-13032, 2020a.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV, pp. 561-578. Springer, 2020b.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, arXiv preprint arXi:22103.02503 2021a.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. lEEE Transactions on Image Processing, 30:8008-800 2018-801,,40 2021b.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021c
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 MORE DETAILS ON CICF
A.1.1 STATISTICAL ANALYSIS ON SAMPLING ALGORITHMS
In this section, we provide the statistical analysis for the superiority our proposed sampling strat- egy, i.e,, clustering muustering-then-tee then-sampling, by comparing to the random mini-batch sampling strategy. As a result, we theoretically derive that using our clustering- tien-sammling has a significant smaller standard error (SE) in estimating the global-scope gradient, compared with using the random mini- batch sampling strategy.
Given a training dataset D = (oi, y1) the global-scope gradient can be computed as:
91
Σ 9
P(x,)
N
(12)
However, it is intractable to compute 9t with Eq. training data.
12 directly, which requires traversing over all
Our peastering-then-sampppe training dataset. Another alternative naive method is random mini-batch sampling.
aims to simulate the global-scope gradient with partial data from the
Random mini-batch sampling. With this strategy, the gradient Grandom of each iteration is com- puted over mini-batch data { 25 j=1" where the mini-batch data with size M is randomly sam- pled from the training data:
M
1
Grandom
9j P(x.), where P(x)))
(13)
M
According to the sampling theory GGosh
2002
the expectation of grandom can be represented as:
M
M
E(grandom) = E
1
Σ 9T) j=1
= M Σ B(g,)= = n.
1
(14)
M
where denotes the expectation of gradients of all samples in the training data. To measure this strategy's 's estimation accuracy of approximating the scopel-coope gradient Grandom in Eq. 12 we can derive its SE as:
M N
S
(15)
M
where 2 is the variance of the gradients of all training samples. Considering M < N, Eq. 15 can be further approximated as:
S Erandom
(16)
M
elustering-then--ammiing In contrast, in our clustering-then-samppig we first cluster the training samples of each class in the dataset into K cluster with K-means algorithm (Pelleg al.) 2000) and totally obtain Kt clusters for the whole training data. We denote the mean and variance of the gradients of the tth cluster Kş as μ4 and of respectively. Since the clustering, the variance of the sample distribution in each cluster is significantly smaller than the variance of the sample distribution in whole training data. Therefore, the gradient variance of the 4tth cluster of is smaller than o². Then we sample N₃, samples from the t cluster to form a mini-batch with the size of M (i.e., M and P(Kx) is ratio of the size of the bth cluster N the size of entire training data N (i.e., N P(Kx) * Then the global-scope gradient can be computed with:
Ng). Here, N₁ proportional to the ratio P(K)) (i.e., N₂ MP(s))
1 kt
Σ
Jours
gä KK₁
(17)
M
14

---

Under review as a conference paper at ICLR 2023
Algorithm Confounder Identification-fiee Causal Visual Feature Learning
1: Input: Training dataset {z ji)]i=1
2: Init: learning rate: a, 3; model f with parameter
3: Obtain Kt clusters from training data by clustering the samples of each class into K clusters.
4: while not converge do
5: Sample M samples from KJ clusters as batch.
6: Estimate global intervention with 9t-
Eq.
7: Update f with 9t 0 = agt.
8: Compute the loss LCICE.
Eq 10
9: Update f <- 0 VV
10: end while
Based on stratified sampling theory (Ghosh 2002). the expectation of gours can be computed as:
NR 1k
MP(K3)
Fggours)
Pk =
(18)
M
The standard error (SE) of clustering-theen-smmpiig can be derived
Kt
N₂ -
SEours
Σ P(K2)
NE (1
(19)
N
#=1
where NoK denotes the number of samples in the Ahth cluster. Considering No < N
Kt
P(K)) M
SEourS
P(K))
N O Σ
(20)
In clustering, the intra-cluster gradient variance of reduces with the increasing the number of clus-
ters. Thus, we can get of < o² when 1 s k < KT. We represent the maximum value of of as maz) = max < ts S K.J. The Eq. 20 can be rewritten as:
P(K+)
P(Ks)
Fourrs
M of (0 max
(o maz
0 = S
(21)
Based on the Eq. 21 random mini-batch sampling for simulating the global-scope gradient accurately. Furthermore, from the Eq. 21 we can let (o)) < o2 by increasing the number of clusters, and obtain the E S Erandom-
draw conclusion that
senterinngthen-sappppee is better than
A.1.2 EXPERIMENTAL. EVIDENCE FOR THE SIGNIFICANT DIFFERENCE BETWEEN TWO SAMPLING STRATEGIES.
To further demonstrate the difference between clustering saatttt--hhh--mmml-t and random mini-batch sampling, we introduce a metric E Σ - Rxl) to measure the difference degree between two sampling strategies, where N and R3 denote the number of sampled instances from the tth cluster using clustering-then-samgee stmilinng and random mini-batch sampling, respectively. We conduct two experiments on PACS with batch-size M = 256 follows. 1) We cluster three clusters for each of the seven classes in PACS and totally obtain 21 clusters. The average is 55, i.e., the difference ratio E E/M between clustering- ceetteriingtten--ee taaee--thh--mmaapp therripplttnnnn--llingnngg and random mini-batch sampling 55 35/256 21.5%. 2) We consider the class prior for random mini-batch sampling, and adopt the random mini-batch sampling weighted by the number of each class. We have E 46 and the difference ratio E/M 46/256 = 18.0%. Based on the above experiments, we can find that our clustering then then-sampling significantly different from random mini-batch sampling.
15

---

Under review as a conference paper at ICLR 2023
A.1.3 MORE DETAILS ON THE SETTING OF SAMPLING ALGORITHM
Our CICF aims to simulate the global intervening effects from the perspective of optimization with:
Kt
N₂ R Ek
9t
(22)
M
==
where a mini-batch sampling procedure is applied to 3J K (I.e., After obtaining the updated 0+ with 0- = - a gt. we can obtain the loss function LCICF and get the reliable optimization direction against the intervening effects of confounders:
LCIC =
(h(x)), (h()))
(23)
xEX
where another mini-batch sampling procedure is applied to the variable x.
As above, there are two mini-batch sampling our CICF adopted for computing global-scope gradient and loss function Leier respectively. They are denoted by {ē, ū) and {a, y - respec- tively. To clarify the composition of the samples in each mini-batch, we first define some notations, respectively as follows:
N: The number of all samples in the training dataset
N The number of all samples in the k cluster.
Ng: The number of sampled samples in the 8 cluster form mini-batch.
M: The number of all samples in mini-batch for computing global-scope gradient.
M The number of all samples in mini-batch for computing LCICF
P(Kx): The ratio of all samples N in the hh cluster to all training data N, which is
N*
represented as P(K₂) N
For computing global-scope gradient, we respectively sample N& samples from the hthh cluster to form mini-batch {ät, " N₂ can be set with two strategies, respectively N₂ (P and Mf No KT
responding to two scheme for computing LIICF
When we directly consider the unbalance question between different clusters in computing Laics XT samples from each cluster form mini-batch {a, (), and corresponding KT. On the contrary, when we randomly sample M' samples from all training data to
sample M'
NA N = M compute Lerer: the corresponding Nş is N₂ MPK))) which needs model the intervention caused by the unbalance question.
A.2 MORE DETAILS ON BACK-DOOR BACK-DOOR/FRONT-DOOG CRITERION
Back-door and front-door criteria have been proposed in Pearl al. (2016) Pearl (2009a) reveal the causality between two variables X and Y. We further clarify their basics mathematically in this section.
Back-door criterion. Fig. 5 shows the back-door criterion, which targets for removing the inter- vention effects with do operation.
operation denotes surgery to cut off the connection from C' to X. From Fig. a). the X() ) is associated with two paths, respectively as X +- C - Y and + Y. Here - C + Y a spurious path, which intervenes the estimation of causality between X and Y, denoted P(Y do(X)) (i.e., the Fig, (())). Following the Pearl et al. 2016
we can denote the conditional probability between and in Fig. 5 as P., P.. (PPX). Since C' has no parent-level variable, the (C) XC) in the Fig, 5lb) is equivalent
the P(C) in the X, C) in the Fig. 5(b) is equivalent to P(Y X, C) Fig. a). Furthermore, we can get the (Y in the Fig. a) since the same graph architecture. Based the above definition, the back-door
16

---

Under review as a conference paper at ICLR 2023
C
do operation
X
Y
X
(a)
(b)
Figure 5: Back-door criterion
C
C
do operation
?
X
Z
Y
X
Z
(a)
(b)
Figure 6: Front-door criterion
criterion can be derived as:
P(Y]do(X)) XY
(24)
Then we can get the formulation of the back-door criterion as:
P(Y]do(XX) -
(25)
Front-door criterion. From Eq. 24 we can draw conclusion that estimating the causality be- tween and (i.e., PoY)do (X) requires traversing the distribution of confounders C. However, confounders C' are in general diverse and not identifiable. To solve the above question, shown in Fig. 6 the front-door criterion introduces an intermediate variable and transfers the requirement of modeling the intervening effects of confounders C' on X + Y to modeling the intervening effects of on Z - Y (Pearl et al. 2016). Specifically, front-door criterion decompose the P(Y (o()) into two components, i.e., P(Y Tdoo Z)) and P( (()),,())), which can be represented as:
P(Y)do(X)) = Σ P(Z ((
2))
(26)
As shown in Fig. a), the variables and Z do not have common causes, which reveals the path X + Z not intervened by other variables. Therefore, the causality between X and is equivalent to its correlation as:
(27) From Fig. G(a), the path Z + Y is intervened by two variables, respectively as C, X, since the existing spurious path Z X C Y. Then off the path X - Z with back-door criterion (Pearl et al. 2016), which gets rid of identifying the confounders C.
P(Z =(do(X)) P(Z = x|X)
simply block this spurious path by cutting
P(Y do = 2))
(28)
FEX
Based on Eq. 27 and we can derive Eq. [66
P(Y
P(z== x|X) Σ
(29)
SEX
Then obtain the formulation of the front-door criterion Eq.
29
17

---

Under review as a conference paper at ICLR 2023
A.3
USING (xX) AS Z SATISFIES FRONT-DOOR CRITERION
In this section, we give the proof that using h(x) as Z satisfies the front-door criterion. As depicted in Pearl al. (2016), the definition of the front-door criterion is following:
Definition: if Z satisfies the front-door criterion relative to an ordered pair of variables (X, Y), must obey the following principles: 1) Z intercepts all directed paths from X to Y. 2) There is no unblocked back-door path from X to Z. 3) All back-door paths from Z to Y are blocked by X.
We prove that ours satisfy the above principles as:
As shown in Fig. 2 the direct path from X to Y is built from the model ip, that is composed two sub-models, and f. We can represent this path as Z = (X),, Y f(Z). We can observe that Z intercepts all directed paths from X to Y, which satisfies the Principle I).
The second principle requires there is no unblocked back-door path from X to Z, (i.e., P(Z]do(X))
vital factor in ensuring that the causal effects can be estimated. We interpret that ours satisfy this principle from two perspectives. 1) As shown in Fig. 2 the effect of confounders C to Z. is transmitted by X through a sub-model
X)Z which
f as C' + X 4 Z instead of X + C - Z. which indicates that there no back- door from X to Z and thus common confounders for X and Z. 2) We prove this from contradiction. From the theory of the back-door criterion, if there are common confounders C for X and Z, then P P(ZIdo(X)) deterministic mapping of X at the inference stage, we always have P(Z|X, C) P(Z(X) for any confounder C. Then we can derive that P(Z|do(X) (X) = Ec=e P(Z]X,c) Σ Σc=e ((c) = XX) This means there are no common confounders for X and Z (i.e., no unblocked back-door path for and Z). which satisfies the Principal 2).
X() In fact, when Z h(X) is obtained from
There
paths from Z to Y, i.e., Z Y and Z X + C The second path is called the back-door path between Z and Y. When we condition on X, the back-door path will be blocked. Therefore, all back-door paths from Z to Y blocked by X, which satisfies the Principal 3).
Consequently, exploiting Z = h(X) is consistent with the front-door criterion.
Another essential factor for estimating the causal effects by the front-door criterion is P(X, Z) 0. That means if P(X, 20, the P(Y|dol (o({))(()) in Eq. 2 equivalent and cannot be estimated. In our CICF, Z is deterministic related X with Z = s(X). Based the probability theory P(X, Z) P(X) X). Since Z = f(X), the P(Z X) = (X = is always holds when / is fixed inference stage. Therefore. PXX, Z) P(X)) P(X) holds in our method.
A.4 DIFFERENT CHOICES OF Z
As shown in Fig. (a), the model yp is separated into successive h and f. and Z = h(X) is the inter- mediate output of yp. To explore the effects of different choices Z, i.e., and different separations PACS. Fig. shows the different choices of Z based on
of s. we conduct ablation experiments ResNet and the corresponding results are shown in Fig. 3]c). It is observed that the shallower Z obtained, the better accuracy the results achieved. We reckon the reason is that selecting Z from shallower layers will result in larger model for f, which will have more capability to learn the conditional probability ((y]))())) In this paper, we set Zo to make have more parameters to learn causal features.
X
conv
layer1
layer2
lave
lavera
fc
+Y
Zg
Z
Zz
Z3
Z4
Zs
Figure 7: Different choices for the intermediate output Z.
18

---

Under review as a conference paper at ICLR 2023
Horse
Dog
Guitar
Person
Girafle
House Elephunt
Horse
Dog
Guitar
Girafle
House
Elephunt
Inputs
ERM
IRM
Ours
Figure 8: Visualization of the Grad-Cams w.r.t. classification task for domain generalization. The first row of left/right panels shows the images from cartoon/photo domains in PACS. The second, third and fourth rows show the Grad-Cams of ERM, IRM (Arjovsky et al.) 2019) and our CICF.
MNIST
MNIST-M
SVHN
SYN
1.0
10-
0.8
0.4
0.4-
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
022
0.4
0.6
0.8
10
ERM
Ours
Figure 9: t-SNE visualization of features learned by ERM (left) and our CICF (right).
A.5 MORE IMPLEMENTATION DETAILS
For PACS and Office-Home, we take ResNet18 (He et al., 2016) pretrained on ImageNet (Degg
al. 2009) as backbone, following Zhou al. (2021c); Carlucci al. (2019); et al.
(2018a) For
VLCS, we take AlexNet (Krizhevsky et al. 2012) pretrained on ImageNet (Deng et al.
2009) as our
backbone, which is the same |ou et 2019 Li et al. (2019) ; Matsuura & Harada (2020). For
Digits- DG, we adopt the model architecture used in previous works (Zhou et al. 2020b; Carlucci al.) 2019 Zhou al. 2021c), which composed of four convolution layers with inserted ReLU and max-pooling layers. We set the mini-bateh size M for computing the global-scope gradient as and the mini-batch size M for computing I-cIF as 84. We set the batch size to 84 and train the model using SGD optimizer for 60 epochs. For PACS, the learning rate and 3 are set to 0.05 and 0.01 respectively. For VLCS, the learning rate a and are 0.005 and 0.001 respectively. As for Office-Home
a, 33 0.001 and 0.001 respectively. For Digits-DG, we set a, 3 as 0.5 and 0.1 respectively. All reported results are averaged among six runs with different seeds.
The GPU memory cost of CICF is 7.151 GiB and the clustering for training data takes 164 seconds for PACS, which introduces only 4.8% extra time to the whole training process (i.e., 3432 seconds) while bringing the significant gain of 2.4%.
19

---

Under review as a conference paper at ICLR 2023
Table 3: Classification accuracy (%) of different DG methods on Office-Home with ResNet-18 as the backbone.
Home Product
Method
Art
Clipart
Realworld Avg.
JiGen
53,0
47.5
71.5
72.8
61.2
MMD-AAE
56.5
47.3
72.1
74.8
62.7
MLDG
57.8
50.3
70.6
73.0
63.0
CrossGrad
58.4
49.4
73.9
75.8
64.4
CCSA
59.9
49.9
74.1
75.7
64.9
MixStyle
58.7
53.4
74.2
75,9
65.5
ERM
58.1
48.7
74.0
75.6
64.2
BRM+MAML
56.8
52.5
74.0
74.7
64.5
ERM+CICF
57.1
52.0
74.1
75.6
64.7
ERM'
59.6
53.0
74.3
75.4
65.6
ERM*+MAME
56.2
56.1
72.6
73.2
64.5
ERM*+CICF
59.3 56.2
74.2
75.1
66.2
Table 4: Classification accuracy (%) of different DG methods VLCS with AlexNet the back- bone.
VLCS
Methods
Caltech
Labelme
Pascal
Sun
Avg.
MLDG
97.9
59.5
66.4
64.8
72.2
Epi-FCR
94.1
64.3
67.1
65.9 72.9
JiGen
96.93
60.9
70.6
64.3 73.2
MMLD
96.6
58.7
72.1
66.8 73.5
MASF
94.8
64.9
69.1
67.6 74.1
ERM
96.3
59.7
70.6
64.5 72.8
ERM+MAML
97.8
58.0
67.1
64.1 71.8
ERM+CICF
97.8
60.1
69.7
67.3 73.7
ERM'
96.4
60.7
68.6
66.2 73.0
+MAML
98.1
58.2
69.6
64.5 72.6
ERM +CICF
98.1
62.4
69.3
69.1 74.7
A.6 FEATURE VISUALIZATION
We visualize more Grad-CAM (Selvaraju al. 2017) of ERM and ERM+CICF in Fig. We can observe that our CICF focus more on foreground regions (i.e., the casual features), while ERM easily focuses on the misleading regions (e.g., the bone in the dog of the cartoon, background) when capturing causal features. As shown in Fig. we visualize the learned feature on Digits-DG by t- SNE (Saito et al., | (019)). We find that the distribution of features extracted from ERM+CICF is more compact across samples with the same category, compared to ERM. This validates the effectiveness of our algorithm for causal feature learning. We also visualize the t-SNE visualizations with different random seeds in the Fig. 10 and Fig.
A.7 MORE COMPARISONS WITH THE DOMAINBED BENCHMARKS.
To further demonstrate the effectiveness of our CICF, we implement our CICF on the baseline
FRM of DomainBed (Gulrajani & Lopez-Paz, 2020) and compare it with the benchmarks used in Dem (Gulrajani & Lopez- Paz, (az2) including IRM (Arjovsky et al. 2019), DRO (Sagawa
al.
2019p. Mixup et al. 2020), MLDG (Li et al. 2018a), CORAL (Sun et 2016). MMD LLi
2018c) used datasets Office- Home /Venkateswara et al. 2017 VLCS Torrala & Efros 2011). and the
al.
2018b), DANN (Ganin et al. 2016), and C-DANN ( al. 2018c). We utilize the commonly-
synthesized ColorMNIST (Arjovsky et al. 2019 As shown in the Table 5 Table 6 Table 7 our
CICF improves the ERM by an average gain of 2% on Office- Home, 1.7% on VLCS and 4.1% on ColorMNIST. Our CICF achieves the best performances these three datasets. It is noteworthy
20

---

Under review as a conference paper at ICLR 2023
that even in the most difficult domain in ColorMNIST, CICF achieve accuracy of 21.4%, which exceeds the second method MMD (Li et al. 2018b) by 10.9%.
Table 5: Classification accuracy (%) of different DG methods on Office-Home with ResNet-50 as the backbone.
Home: Model selection method:
training domain validation
Algorithm
A
C
P
R
Avg.
ERM
62.7 + 1.1
0.6
7
0.3
67.5 = 0.5
IRM
61.8 1.0
1.0
0.8
66.6 1.0 1.1
DRO
0.7
52.
0.5
7..
6.3
Mixup
6.7
5.67
0.3
0.3
6.10
MLDG
6.3
5.65
7..9
7..6
6.1
CORAL
6.3
0.5
7.5
77.9
6..6
MMD
6..0#0.1
53.7
7.31
6.1 7.5
DANN
1.1
5..7
7.81
6.64 0.6
C-DANN
6..0
51.1
7.3
65.6 7.70
ERM + CICF (Ours)
63.1 0.2
5.24
7.0
78.1 0.4
69.5 + 0.2
Table 6: Classification accuracy (%) of different DG methods VLCS with ResNet-50 the
backbone
VLCS: Model selection method: training domain validation set
Algorithm
Avg.
ERM
97.6 1.0
0.9
72.2 1.220.3
76.4
7
IRM
97.6
0.9
0.5
76.9 1.3
78.1
DRO
9.47
6.1
0.7
7.9
0.6
Mixup
0.3
6.6
7.9
1.3
0.4
MLDG
0.3
63.0
0.6
0.3
0.4
CORAL
9.1
6..6
7..7
7..
0.5
MMD
9.41
6.74
0.84
7..9
7.9
DANN
9.2
6..9
0.7
0.3
0.3
C-DANN
0.1
65.2
7.14+1.11
7.29
78.2
ERM + CICF (Ours)
99.1
64.0
0.5
79.2
79.1 0.3
Table 7: Classification accuracy (%) of different DG methods ColorMNIST with MNIST back-
bone used DomainBed
ColorMNIST: Model selection method: training domain validation set
Algorithm
0.1
0.2
0.9
Avg.
ERM
0.2
73.2 0.3
10.0
52.0 + 0.1
IRM
0.3
0.0
0.2
51.8 0.1
DRO
0.3
7.3
1..0
0.1
Mixup
72.4+
0.3
0.19 1..0
MLDG
7..4
7.0
10.0
5..6
CORAL
7..8
7.2
10.1
5..7
MMD
72.1
7..
10.5
5..8
ADA
0.30
0.54
0.3 10.0
CondADA
0.3
7.2
1..4
51.9
ERM CICF (Ours)
0.3
74.1 0.2
21.4
56.1 + 0.2
21

---

Under review as a conference paper at ICLR 2023
MNIST
MNIST-M
SVIIN
SYN
1.-
0.6
0.5
0.4
0.0
0.0
0.0
0.6
0.8
1.0
0.8
0.8
0.4
0.4
0.2
0.0
0.0
0.8
0.4
1.0
1.0
0.6
0.6
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
ERM
CICF (Ours)
Figure 10: t-SNE visualization of features learned by ERM (left) and our CICF (right) with different seeds.
22

---

Under review as a conference paper at ICLR 2023
MNIST
MNIST-M
SVHN
SYN
0.8
0.8
0.6
0.2
0.2-
0.0
0.2
0.4
0.6
0.0
1.0
0.0
022
0.4
0.6
0.8
1.0
1.0
1.0
0.5
0.6
0.2
0.2
0.0
1.0
0.0
0.2
0.A
0.6
0.8
1.0
0.8
0.8
0.6
0.2
0.2
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
ERM
CICF (Ours)
Figure 11: t-SNE visualization of features learned by ERM (left) and our CICF (right) with different seeds.
23


Reviews:
[{"review_id": "s5Eah-Oh8U", "reviewer": "Reviewer_wmrP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a novel method (CICF) that is based on the idea of front-door adjustment for hidden confounders. They additionally demonstrate the relation between CICF and the popular meta-learning strategy MAML (Finn et al., 2017). The authors provide a theoretical interpretation of why MAML works.", "summary_of_the_review": "I find the proposed method and theoretical part of the part quite interesting, but I am not convinced by the experiments (described above my questions and concerns).", "strength_and_weaknesses": "**Strengths:**\n\nThe paper introduces a novel idea to use the front-door criterion instead of the back-door for covariate adjustment. They theoretically demonstrate how to incorporate front-door criteria ideas into the learning algorithm. They additionally demonstrate that front-door criteria could be used to theoretically explain the previously proposed meta-learning algorithm (MAML).\n\n**Weaknesses and questions:**\n\n- Why the approach proposed by eq. (4) won’t also lead to a trivial solution? Won’t the gradient become close to zero when the model converges? Won’t be the information on x tilda also ignored?\n- It is not clear to me why results for previously published models don’t agree with the DomainBed framework. The experiments would look more convincing if they were implemented and compared within DomainBed framework and good reproducibility practices.\n- Grad-CAM maps from supplementary figure 9 look less optimistic than the ones in figure 4. For example, it shows that the model still attention to the bone to classify the dog. Also, such visualizations would be more interesting to compare not to models learned with ERM strategy, but with any causal learning method, e.g. IRM.\n- Also for synthetic examples in order to show the effect of the confounding variable was removed, it would be more interesting to see results on colored MNIST.\n- A more explicit demonstration of the removed confounding effect would be very helpful.\n- The results in Table 1,3,4 are better on average, but not across all environments. If the model was truly correcting confounding, why would be there such inconsistencies?\n- Figure 9 is also not very convincing that the learned features are somehow significantly different. The plots look actually quite similar up to random seed or some t-SNE hyperparameters.\n- Confidence intervals in Tables 1,3,4 would be quite helpful.\n- How did the authors choose the best set of hyper-parameters for each dataset?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pcdmx0MTcso", "reviewer": "Reviewer_jBAD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel method of learning robust DNNs through the use of a network containing two si models, one that learns an intermediate variable Z and then combines it with the front door criterion to learn the interventional distribution p(Y|do(X = x)). This interventional distribution has invariance properties, and that is why it is a common learning strategy used when it comes to building robust models. Various strategies to learn certain intractable models are described, and state of the art performance is shown on certain datasets. Additionally, this provides a theoretical justification for an existing approach, which is appreciated. ", "summary_of_the_review": "This paper uses a two stage network to induce a front door causal graph and learn the interventional distribution p(Y | do(X+x)). since this interventional distribution is invariant, it is robust to distribution shift and domain generalization. This allows the authors to learn a prediction model that is robust and outperforms state of the art methods. ", "strength_and_weaknesses": "Strength: Uses a front-door graph instead of a back door graph, which has novelty. Describes how to fit intractable models and achieves state of the art results on  datasets.\n\nWeakness: A big challenge with causal inference is being able to correctly fit the nuisance models when we are interested in learning the target parameter. Here, I’d like to see some more assurance around the proposed training strategy being able to fit the nuisance models correctly and consequently correct learn the interventional distributions in a variety of situations. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G1M2e0alLy6", "reviewer": "Reviewer_pVYZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To eliminate domain effect in deep learning, existing approach relies on identifying explicitly the confounder and remove confounder effect via backdoor criterion.  The proposed method implements front door criterion via gradient. \n\nThe author proposed to use a neural network to represent $\\sum_{x}Pr(Y|Z=z, x)$, and shed light on the connection with Model Agnostic Meta Learning, which is, similar to MAML, use a “look-ahead” gradient operation to represent the conditional distribution. \n\nTo achieve this look ahead, the author proposed to sample a small portion of the whole sample which represent the whole distribution. \nTo achieve a sampling strategy which could mimic the all-observation distribution, the author use K-means algorithm to cluster the data into several clusters and take a small portion from each of the cluster. \n", "summary_of_the_review": "I think the paper is very innovative but still lacks some detailed explanation to the crucial part. Especially the way that the author use look-ahead gradient step to approximate the conditional distribution and using a neural network to extract a deterministic feature from image X to z, which is not how front door criterior works initially. ", "strength_and_weaknesses": "Strength:\nI like the way that the author try to use different subfield to address the domain generalization problem with meta learning and causal inference.\n\nWeakness:\nPlease see the questions: \n\n\n\nQuestion:\nThe original front door criteria is actually a concatenation of two time back-door criterion, which is concatenating two times do operation. The first do operation is p(Z=z|do(X=x)), which means if one cut the parent of node X, what would be the conditional distribution of Pr_{modified graph}(Z=z|X=x), usually, this z is observed and believed to be a direct cause of target variable Y and x is a direct cause of z.\n \n\nWhy p(Z=z|X=x)= 1? If this is true, then $z$ should be a deterministic map of X.\n\nFor the second backdoor part $\\sum_x Pr(Y|Z, x) p(x)$, which is essentially treating x as an indirect confounder, due to the deterministic map z=h(x), the summation over z is not needed, Pr(Y|Z, X) only depend on X. As the author has observed, directly concatenating $z=h(x)$ would result in $z$ being ignored. But the author introduced using a “look-ahead” (name I give) gradient to represent this probability, i did not see much explanation why? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5ITpyXu66C", "reviewer": "Reviewer_K8gB", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address the fact that deep networks may condition on spurious features that are correlated with the target class without being causal. They attempt to leverage the front door criterion in order to avoid the need to explicitly condition on confounders. Their approach works by treating the hidden representation as a mediator (this is incorrect) and then attempting to control for confound via a front door-like update (except they represent interventions as MAML-style gradient perturbed parameters). ", "summary_of_the_review": "This paper has very serious technical flaws and is not ready for publication/", "strength_and_weaknesses": "This papers is built on a series of serious misunderstandings of the relationship between a model and a data generating process in causal inference. The front door criterion says that if we have access to data of the form X -> Z -> Y, where Z mediates the relationship between X and Y, then we can control for any confounding between X and Y (including unobserved confounding) by the front door adjustment formula. But not - this is a statement about the true data generating process --- i.e. Z is really some variable in the real world that we observe --- it is not a statement about our model of the data generating process, as this paper assumes. Their key idea is to treat the hidden representation as a mediator, which they attempt to justify in appendix A.3, but this justification is just wrong: the hidden representation z = g(X) is *not* on the **causal** path from X to Y. It is just an intermediate step in the computation of model's estimate,  \\hat{Y}. We can see this by noting that if we were to intervene on z = g(X) (e.g. by setting it to zero), nothing would change in the true Y (because our model's estimate is not on the causal path of the true data generating process!), but \\hat{Y} would change. \n\nSimilarly, the section on the gradient-based instantiation of FEIE seriously misunderstanding what an interventional distribution is (again, it's a property of the true data generating process, not the model), so nothing that's written there makes sense from a causal perspective.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject