C-Disentanglement: Discovering Causally-Independent Generative Factors under  an Inductive Bias of Confounder
Authors: Xiaoyu Liu
Jiaxin Yuan
Bang An
Yuancheng Xu
Yifan Yang
Furong Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: causal disentanglement
causal generative process
generative factors
confounder
inductive bias
disentanglement
causal inference
TL;DR: A framework that discovers causally disentangled generative factors under inductive bias of confounder

Abstract: Representation learning assumes that real-world data is generated by a few semantically meaningful generative factors (i.e., sources of variation) and aims to discover them in the latent space. These factors are expected to be causally disentangled, meaning that distinct factors are encoded into separate latent variables, and changes in one factor will not affect the values of the others. Compared to statistical independence, causal disentanglement allows more controllable data generation, improved robustness, and better generalization. However, most existing work assumes unconfoundedness in the discovery process, that there are no common causes to the generative factors and thus obtain only statistical independence. In this paper, we recognize the importance of modeling confounders in discovering causal generative factors. Unfortunately, such factors are not identifiable without proper inductive bias. We fill the gap by introducing a framework entitled Confounded-Disentanglement (C-Disentanglement), the first framework that explicitly introduces the inductive bias of confounder via labels from domain expertise. In addition, we accordingly propose an approach to sufficiently identify the causally-disentangled factors under any inductive bias of the confounder.  We conduct extensive experiments on both synthetic and real-world datasets. Our method demonstrates competitive results compared to various SOTA baselines in obtaining causally disentangled features and downstream tasks under domain shifts.

C-Disentanglement: Discovering Causally-Independent Generative Factors under an Inductive Bias of Confounder
Jiaxin Xiaoyu Jixiii Yuan", Bang An' Yuancheng Xu 2, Yifan Yang', Furong Huang'
Departmeet of Computer Science, Department of Mathematics
University of Maryland, College Park {x1iu1231, jyuan98, bangan, yexu, yang7832, furongh) Qumd edu
Abstract
Representation learning assumes that real-world data is generated by few mantically meaningful generative factors (i.e., sources of variation) and aims to discover them in the latent space. These factors are expected to be causally disen- tangled, meaning that distinct factors are encoded into separate latent variables, and changes in one factor will not affect the values of the others. Compared statistical independence, causal disentanglement allows more controllable data generation, improved robustness, and better generalization. However, most isting works assume unconfoundedness (i.e., there are no common causes to the generative factors) in the discovery process, and thus obtain only statistical inde- pendence. In this paper, we recognize the importance of modeling confounders in discovering causal generative factors. Unfortunately, such factors are not identi- fiable without proper inductive bias. We fill the gap by introducing a framework named
(C-Disentanglement) the first framework that explicitly introduces the inductive bias of confounder via labels/knowleder from domain expertise. addition, we accordingly propose an approach suffi- ciently identify the mauusall--dsenttangle factors under any inductive bias of the confounder. We conduct extensive experiments on both synthetic and real-world datasets. Our method demonstrates competitive results compared to various SOTA baselines in obtaining causally disentangled features and downstream tasks under domain shifts. code available here
1 Introduction
Causally disentangled representation learning methods en- deavour to identify and manipulate the underlying explana- tory causes of variation (i.c., generative factors) within ob- servational data through obtaining causally disentangled representations [24, 8). Such a representation encodes these factors separately in different random variables in the latent space, where changes in one variable do not causally affect the others. Pursuing causal independence' makes possible to identify the ground truth generative factors that are not statistically independent, such as the color, shape and size in a fruit dataset as shown in Figure 1, which more realistic and allows more controlled data generation, improved robustness, and better generalization in out-of outtof-diitribution problems.
Color
Shupe
Size
p(color, shape, size) # picolor)p(cchppelpsiee)
Figure 1: Causally independent genera- tive factors (color, shape, size) may ap- pear statistically correlated in observa- tional data.
Wee will use causally disentangled and causally independent interchangeably.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Despite great success, existing methods suffer from the identifiability issue in discovering semantically meaningful generative factors. The main reason is that most of them equate disentangling in the latent space with enforcing statistical independence [9, 11, 6] in the latent space, or requiring no mutual information[8]. In other words, they explicitly or implicitly assume that the observational dataset is unconfounded in the generative process, which is not realistic in practice. A confounder C is a common cause of multiple variables. It could create a correlation among causally disentangled generative factors in the observational distribution. In the previous example of fruit dataset, the type of fruit is a confounder, creating a correlation between size and shape (apples are usually red and round objects), thus size and shape can never be captured by learning objective that requires statistically independent latent representations. Assuming unconfoundedness in such a dataset leads to discrepancies in characterizing the statistical relationship of the generative factors, and makes them non non-identifiable.
Considering the limitation of the unconfoundedness assumption, few of the recent studies take into account confounders [24, 23, 21] in the problem formulation. Specifically, [23, 21] proposed evaluation metries in the confounded causal generative process. [24] introduced an unsupervised learning scheme by regulating the learning process with the Seaependdnnc-Of-Suppoo Score (IOSS) under unobserved confounder, following the theory that if latent representation is causally disentangled, then it must have independent support. However, it has been shown almost impossible to obtain disentangled representations through purely unsupervised learning without proper inductive biases [10, 16]. While there are a few semi-supervian or weakly supervised methods [16, 17] developed seeking statistical independence, none of the mentenedt methods considers how to provide inductive bias for confounder in learning causally disentangled representations.
In this paper, we recognize the importance of providing inductive bias to confounder so that the ground truth generative factors can be identified and we fill the gap by introducing the inductive bias via knowledge from domain expertise. Specifically, we propose a framework called Confounded- Disentanglement (C-Disentanglemen)) C-Disentanglement is, to the best our knowledge, the first framework that discusses the identifiability issue of generative factors regarding the inductive bias of confounder, and thus opens up the possibility to discover the ground truth causally disentangled generative factors which are correlated in the observational dataset. Importantly. C- C-Disentangleman is a general framework that includes existing methods (in which unconfoundedness is assumed) as special cases, allowing customized inductive bias of confounders according diverse user needs.
Under the framework, We develop an algorithm to discover the causally disentangled generative factors in the latent space with inductive bias C, where C is label set. Instead of enforcing global statistical independence among variables on the observational dataset, we partition the dataset into subsets according to realizations of C, and regulate these variables within each subset.
Our proposed discovery methodology is general and can be applied to various disentangle mechanisms. But as a prototype for proof of concept, we implement our method, cdVE under the context of VAE [12]. Specifically. we formulate a mixture of Gaussian model in which each Gaussian component, conditioned on a specific value of C, represents the distribution of a latent variable that estimates the underlying generative factor. Intuitively, under the previous example of the fruit dataset, if we fix the type of fruit (e.g., the apple) and check the correlation between the color and the shape, we could find that the color change within the apples does not affect the shape distribution, thus can be captured by regulating the statistical independence conditioned on C (a fixed fruit type).
Our methodology is novel learning paradigm that can discover latent factors through weak supervi- sion labels (i.c., C).
We experiment with several tasks from image reconstruction and generation, to classification under the out-of-distribution scenario on both real-world and synthetic datasets. We demonstrate that our method outperforms various baselines in discovering generative factors in the latent space.
Summary of contributions: (1) We recognize the identifiability issue of discovering generative fac- tors in the latent space. We accordingly introduce framework, named Confounded Disentanglement (C-Disentanglmeent). It is the first framework that discusses how inductive bias of confounder could be explicitly provided via labels/knowledge from domain expertise. (2) We propose an algorithm, cdVAE, that discovers causally disentangled generative factors in the latent space. The algorithm sheds light on the easy injection of inductive bias into existing methods. We conduct extensive experiments and ablation studies across various datasets and tasks. Empirical results verify that
2

---

cdVAE outperforms existing methods in terms of inferring causally disentangled latent representation and also show cdVAE's superiority downstream tasks under OOD generalization.
2 Preliminaries
In this section, introduce basic concepts in causal inference and then show how to evaluate the causal relationship among variables latent space.
Causal graph through DAG. The causal relationship among variables can be reflected by a Directed Acyelic Graph (DAG). Each (potentially high-dimensional) variable is denoted as node. The directed edges indicate the causal relationships and always point from parents to children.
Intervention and do-operator. Intervention is one of the fundamental concepts in causal inference. When we intervene on a variable, we set the value of a variable and cut all incoming causal arrows since its value is thereby determined only by the intervention [20]. The intervention is mathemat- ically represented by the do-operator do(-). ). Let Z1 and Zz be two variables, P(Zz) [[Zz]do(22 = z1)) characterizes an interventional distribution and reflects how change of Z1 affects the distribution of Zz. The do-operation and the interventional distribution should be estimated on the interventional dataset. However, in practice, the true distribution of the data is unavailable but an observational subset. As result, we estimate the interventional distribution from the observational set following do-calculus introduced by [19]. A detailed introduction can be found in the Appendix A.
Confounders bring in spurious correlation. Although the detailed definitions vary from literature, a confounder usually refers to common cause (i.e., common parent in the causal graph) of multiple variables and it brings in certain level of correlation among these variables. Consequently, to estimate the causal effect of one variable on the other from the observational dataset, we have to eliminate the correlation introduced by the confounders. For example, when analyzing the causal relationship between the number of heat strokes and the rate of ice cream consumption, we may find that there is a correlation. However, the temperature is confounder in the situation. If we eliminate this spurious correlation by conditioning on the temperature, we may find out that under the same temperature, heat stroke is not correlated to the rate of ice cream consumption.
Evaluation of interventional distribution. We specifically consider the case where there variables Z and Zo, and we analyze how the existence of parental nodes affects the estimation of P(Zz Zi = 21 on the observational distribution.
Proposition 2.1. Let Z1 and Ze be two random variables, C" be the ground truth confounder set. If C is a superset of or is equivalent to C", i.e., C C C, with c being realization of C, we have
P(Zz)do(Zz))
Σ C7
if no C C is a descendent of Z.
The proof can be found in Appendix B.1.
Intuitively speaking, Proposition 2.1 states that to accurately estimate the causal relationship between variables, we have to eliminate the spurious correlation by conditioning on confounders. addition, conditioning on additional variables will affect the estimation if they are not decedents of Z.
Causal disentanglement or statistical independence. If we had access to a series of ground-truth features that generated a fruit dataset(e.g.. color, shape, texture), into a variable and concatenate them for disentangled representation. However, these features in the observational dataset are usually correlated, depending the type of fruits. If not considered from the causal perspective, traditional methods aiming for statistically independent representations, such B- B-V [9]. may not be able consider correlated features that are disentangled. In fact, neither causal disentangled entails statistical independence nor vice versa because the existence of confounders. We, therefore, formulate the task of inferring causally disentangled features from the observational under a certain level of confoundedness.
encode each of the features
3

---

3 Confounded causal disentanglement via inductive bias
3.1 Problem formulation
C
We formally frame the task of discovering set of generative factors from the observational dataset from causal perspective as shown Figure 2. Let X denote the observational data, the confounded causal generative pro- cess [23, 21] assumes that X are generated from K ground-truth causes of variations G [G1, G,,.., Gx] (i.e., G - X) that do not cause each other. These generative factors generally not available, and they are confounded by some unobserved confounding variables C'. The task of interest is discover those generative
G
Gx
4
Figure 2: The left figure shows the ground truth gen- erative process while the right figure demonstrates learning task.
factors in the latent space (denoted as Z = [21, Z2, Zpl) that best approximates G from X.
Causal disentanglement among latent generative factors. Generative factors, encoded in the latent representational space, are causally disentangled, meaning that intervening on the value of one factor does not affect the distribution of the others. formally defined as:
Definition 3.1 (Causal Disentanglement on Data X([19, 23, 24))). A representation is disentangled if, for i E [1,..,D}, D}.
P(Zz)do(Z- = z-1). X) = P(Z,)) Vz,.
where -i = (1,2, D3/i indicates the set all indices except for i.
Challenge of unobserved C The generative factors G are not identifiable without proper induc- tive bias of C* [16]. Previous works on discovering the generative factors either obtain disentan- glement by enforcing statistical independence variables do not capture information of each other [8]. Such a setting is equivalent to assuming unconfoundedness of the generative factors. It ignores the possibility that correlated latent variables can also be causally disentangled in the observational distribution, and hence is an assumption too restrictive. Fortunately, even though the ground truth generative factors unobserved, domain expertise may inform a "reasonable" or "likely" inductive bias of the confounder from an accessible label set, denoted as C. This is used account for all correlation among Z.
latent variables [9, 15, 6], require that latent
Note here that we do not assume the accessible label set C equals to C (but hope that it close to C²). The relationship between C* and label set C must fall into one of the following scenarios, despite the immeasurability of their exact relationships.
Case The label set contains information about the confounders, i.e., C
Case The label set contains partial information about the confounders, i.e.,C C*,
Case The label set contains all information about the confounders, =
One may argue that C may contain information irrelevant to the ground truth confounders. We show in Appendix that in the confounded generative process described in this paper, irrelevant information in C does not affect the evaluation of the interventional distribution, and therefore can be ignored. We only take into consideration how much information C* is captured here without loss of generality.
According Proposition 2. in case 3, we can estimate P(Z, (oooo set with inductive bias from C. However, in the rest of the cases, the equation does not hold. Therefore, we introduce an operator (-) to estimate the interventional distribution on the observational set under inductive C. The framework that applies do is named C-Disentanglement, shorten for Confounded Disentanglement.
2-1, 2- X) on the observational
3.2 C-Disentanglement as the learning objective
We formally introduce C-Disentanglement and do as follows:

---

Definition 3.2 C-Disentangllment and do²). Let X be the observational data, Z - [Z1 Z2.,.. Zp] be a concatenation of random variables, C be a label set selected from domain expertise to provide inductive bias for confounders of the observational data, we define do operation as:
P PZ
Σ
c)
Vi 1,2,3, D, where is realizations of C. Z obtains C-Disentanglement on X given if
P(Z, do (6-, X) = X))
C as an approximation of C". Whether the ground truth generative factors can be discovered depends on whether their statistical relationship on the observational distribution is correctly charac- terized, and it requires knowledge of C" However, due to the unobservable nature of the confounders, use C' as approximation of C", in Equation (4). Then the identifiability of the ground truth generative factors depends on the relationship between C and C We thus examine three possibilities as listed in section 3.1 together with case study as in Figure 3.
Ixpatisd Apple
aplle
Sauzaa
Inperiat
Co C-iepered heans
C- Cobal benem
Co co
C trai and wnighti braed
cannen dloaee groenteed
(a) C
(b) C C C
(c) C C*
(d) C D C*
Figure 3: Case study: fruit dataset. Suppose a fruit dataset is generated by color, shape and size. These factors are correlated depending on the type of fruit and the place of origin. One way to estimate the causal effect within latent variables, proposed as C-Disentanglement, is to condition on realizations of the confounders C, which partitions the data into subsets and inspects within each set. Figure (a)-(d) demonstrate partitions under different C. In fig. 3a, when C 0, color, shape and size cannot be identified, as the requirement of statistical independence contradicts the fact that they are correlated in the observational set. In fig. 3b, only partial information given. It still helps as can identify that size and shape are causally independent although entangled by color. In fig. 3c and fig. 3d, the ground truth generative factors can be discovered.
In Case 1, C is an empty set shown in fig. 3a. It is equivalent to assuming that the generative process is unconfounded and Equation (3) degrades to statistical independence on the observational set:
do 2 P X).
=
(5)
This contradicts the fact that these factors are actually correlated and such assumption makes the ground truth generative factors unidentifiable when there exists a underlying confounder.
In Case 3, as shown in fig. 3c and fig. 3d, C C*, do is equivalent to do Thus the ground truth generative factors can be identified.
In Case 2, partial information about the confounders is given as shown in fig. 3b. Heuristically speaking, this relaxes the constraint of statistical independence on the observational set and therefore enlarges the solution set, providing an increased possibility for the ground truth factors to be recovered. We empirically show in Section that even partial information of confounders improves accuracy, outperforming existing methods in various tasks, even under distribution shifts.
4 VAE: identify causally disentangled factors in the latent space.
In this section, we provide an algorithm, edVAF, for confounded disentangled VAE, to identify the latent causally disentangled generative factors under confounder C in the context of VAE.
5

---

2 ~ N(y. 2n) Zg ~ (V4y.
Z= + + +
DECODER
X
X
ENCODER
24 En)
2.)
L r0c + Les Dxu(((Z
Figure 4: Learning paradigm of edVAE. The input data is partitioned by realizations of the confounder C. We infer from each partition a Gaussian distribution and form a mixture of Gaussian model to characterize the distribution of on the observational distribution. We assume soft assignment of samples and further infer x (X) that resembles C.
4.1 Learning objective
Given X the dataset, hope to find deterministic function f. parameterized by 0, where f: 2 - X' such that (1) P(X)= = f(z; 0) P(Z)dz is maximized and (2) each Z; encodes causally disentangled generative factors, with Z [z. Zz, Zpl as concatenation of random variables in the latent space.
More concretely, [zz 0) is characterized by probability distribution P(Z|X; 0), and p(Z) is prior distribution from where Z can easily sampled. For each Zí E Z, we require
P(Zi'do (Z (Z_.), X) = X(Z E C.
(6)
Applying the variational Bayesian method [12], the learning object to optimize the evidence lower bound (ELBO) while satisfying the constraints causal disentanglement:
Ez Ez~qllogP(XI z: 0)] D (
8.0
s.t.
X) P(Z|() =0 vi 1,,...,,
(8)
For simplicity, we omit all model parameters 0, & in writing.
4.2
Learning strategy
We start from the estimation of the causal disentanglement constraint shown in Equation (6). Because the probability distribution hard to directly calculate, inspired by [23], we resort to its first-order moment approximation. Specifically, estimate the Lı distance between the expectation probability P(Z, (Z..) X) and (XZ as follows:
Le L= Σ
[ dE (Z-1),, X), B(Z(X))-
We show in the following theorem that eq. (9) is satisfied the learned latent variable subjects to Gaussian distribution with a diagonal covariance matrix. Proof can be found in appendix B.2.
Theorem 4.1. Suppose that the latent variable Z on dataset X given C = subjects to the Gaussian
Distribution ((p E (X)). Specifically,
P(Z = c, X) (2T) D/2 det( exp (- (170-190800)-111 -1)
where Z E R (X) is diagonal for all we have
D
1. d(E(Z, (Z-),, E(Z, X)) = 0.
(10)
From Theorem 4.1, we see that for each C c, enforcing the latent variable Z be statistically independent minimizes lc. Taking the whole C set into consideration, P(Z|X) subjects to a mixture

---

of Gaussian distribution where each centroid is inferred from observational data under specific realization of the confounder C:
X(Z = = [x)N(((()
(11)
GEC
The mixing coefficient ne P(C c(X) reads the probability of occurrence of C Nevertheless, such a hard assignment of coefficient varies with observable dataset and cannot accommodate scenarios in which the label set C does not exist. In this paper, we parameterize T. Gaussian distribution for a soft assignment of samples: Te ~ (x)(X,, (X), Parameters (((X) and (X are learned to minimize the discrepancy with P(C c(X).
In VAEs, the prior of the latent space P(Z) is assumed to follow a Gaussian distribution with mean zero and identity variance: Z ~~ (0, f). To avoid enforcing statistical independence of the overall latent spaces we Iearn, we only assume that the prior of to be a Gaussian distribution with variance one for each subset of C.
Concretely, suppose that the latent variable Z for X under C c follows a Gaussian distribution N( (X)), then the KL divergence in (7) regulates the distribution to (X)(())
By the Lagrangian multiplier method, the new loss function is
L
Flog P(X[Z)] Elog P(C|ma(XX)) + D PIZX
(12)
Drec
Geto
The algorithm pseudocode can be found in Appendix C.
5 Experiments
In this section, we experimentally compare cdVAE with various baselines on synthetic and real-world datasets, and study properties of cdVAE through the ablation studies. We demonstrate that edVAE allows for causally disentangled but statistically correlated features being discovered in the latent space, which better approximates the ground truth generative factors and outperforms SOTA baseline under distribution shifts.
Concretely, aim answer the following questions regarding the proposed model:
Q1: How does perform compared to the existing methods in the latent space?
Q2: How does it perform in downstream tasks such as classification under distribution shifts?
5.1 Basic setup
Datasets. we evaluate edVAE on three datasets: synthetic datasets 3dshape [3] and Candle [21], and real-world dataset CclebA [15]. 3dshape is dataset of 3D shapes generated from 6 ground-truth independent latent factors. These factors are floor color, wall color, object color, scale, shape, and orientation. There 480,000 images from all possible combinations of these factors, and these combinations are present exactly once. Candle is dataset generated using Blender, a free and open- source 3D CG suite that allows for manipulating the background and adding foreground elements that inherit the natural light of the background. It has floor hue, wall hue, object hue, scale, shape, and orientation as latent factors.
We use 3dshape and Candle as synthetic datasets. To mimic the real-world scenario where perfectly disentangled ground-truth generative factors do not exist or are hard to identify, manually create certain level correlation in these datasets by first making rules among certain attributes and then sampling accordingly.
Baselines and tasks. We compare the performance of cdVAE in the task of image generation and classification under distribution shift with the following architectures: (1) VAE-based methods (vanilla
VAE [12], 3-VAE [9], FactorVAE [11]) that equate disentanglement as statistical independence, (2) Existing causal regulation methods, CAUSAL-REP [24], (3) <VA [12] as it also applies the label information (4) GMVAR [7] as it also adopts a mixture of Gaussian model in variational autoencoder framework.
7

---

Table I: Compare with baselines in image generation task on celebA and candle. The reconstruc- tion error indicates the end-to-end performance of the image generation task. D-score measures from non-causal perspective and requires that the generation process is unconfounded. We expect that a good method that recovers causally disentangled factors should obtain poor (i.e., low) D-scores. IOSS, UC and CG are causal metrics that measure the level of disentanglement of representation.
CelebA
Candle
D1 (nea-causal)
1OSS (causal)
UC (causal)
CG 1 (causal)
IOSS | (causal)
Methods
Recon !
Recon
(non-causal)
VAE
0.33
0.11
0.78
.044
0.14
0.10
0.18
0.69
3-VAE
0.27
0.15
0.74
.017
0.18
0.11
0.24
0.54
Factor VAE
0.25
0.17
0.68
.014
0.15
0.13
0,26
0.51
CAUSAL-REP
0.29
0.16
0.34
.012
0.18
0.20
0.32
0.31
<VAE
0.32
0.14
0.64
.000
0.14
0.12
0,21
0.62
GMVAE
0.71
0.12
0.09
0.71
odVAR
0.18
0.12
0.21
.008
0.11
0.15
0.54
0.16
Note that in this paper, we do not compare VdE with CausalVAL [27]. despite the latter also
obtaining "disentanglement" in the latent space. The main reason is that the problem-setting and the learning objectives are fundamentally different. CausasIVAE aims to disentangle known ground truth generative factors in the latent space, it aims to bind these factors one by one to latent variables and the causal dependency is not considered. The task of interest in this paper is discover causally disentangled factors in the latent space and the ground truth is unavailable, thus these two methods are not comparable.
Evaluation metrics. The experimental results are evaluated on (1) end-to-end evaluation metrics: the accuracy of classification under distribution shifts and reconstruction loss image generation task. (2) how well the learned latent generative factors recover the ground truth one: Maximal Information Coefficient (MIC) and Total Information Coefficient (TIC) [13]. (3) Existing disentanglement scores from a causal perspective: IRS [23], UC/CG [21], IOSS [24], and statistical perspective: D-Score [8]. The higher these evaluation metrics, the better the model except for IOSS (the lower, the better). detailed introduction of these metrics can be found in Appendix C.
To be more concrete, IRS evaluates the general interventional robustness of learned representations while UC and CG are extended from IRS. UC (Unconfoundedness) evaluates how well distinct generative factors captured by distinct sets of latent dimensions with no overlap. CG (Counter- factual Generativeness) measures how robust a certain latent variable is against the others. IOSS measures whether learned random variables have independent supports necessary condition inferred from the definition of causal disentanglement. D score measures disentanglement in the non-causal definition.
5.2 Experimental results
(Q1) ed VAE significantly outperforms various baselines end-to-end measurement. We compare our cdVAE with baseline models in the image generation task on the CclebA, and Candle datasets. In CelebA dataset, we have C set to be whether there are eyeglasses in the image, i.c.,
|C| = 2. In the Candle dataset, we have |C| = 4, with 2 objective hues and two wall hues combined together. More details can be found in Appendix C.
As shown in Table 1, cdVAE are evaluated by several groups of metrics. Reconstruction error measures how well the images are reconstructed in an end-to-end fashion while the others are measurements of how disentangled the latent factors are: D score is from the non-causal perspective while UC, UG and IOSS are from the causal perspective. For the celebA dataset, we not measure the UC and CG score as it requires ground truth generative factors access to the true generative process. Our method outperforms all basclines on these metrics except for the score. Note the D score is only effective measurement when there are no confounders in the observational dataset, as it requires each latent variable not to contain information about the others. Under the confounded dataset such as CelebA and Candle, the ground truth generative factors are correlated, resulting in low D score with our model as expected.
(Q2.1) VdE are more robust under distribution shifts. We conduct the task of shape classification on the 3dshape dataset with distribution shift and use the classification accuracy
metric for source distribution, we sample a certain percentage of images which the object hue correlated with the object shape (i.c., red objects are cubes). The rest of the images are evenly generated by
generalization|22) Specifically, in the
8

---

disentangled factors, while in the target domain, all images are generated by disentangled factors. The proportion of highly correlated data is denoted by shift severity. For example, shift severity 0.4 means that 40% of training images are sampled under preset correlation between object hue and object shape.
We train cdVAE, B-VAE, and CAUSAL-REP using images from the source set, with decoders being replaced by classifiers. The trained classifier then tested on the targets set. We report the classification accuracy on the target set and the performance drop in Figure 5. We could observe that cdVAE more robust than other baselines under distribution shift as has the lowest performance drop and the highest target set accuracy.
An intuitive explanation of how C-Disentanglement improves the OOD generalization. Suppose we have fruit dataset and the goal is to classify fruits. In the training set, a large amount of apples red and round. If assuming unconfoundedness and requires statistical independence, is possible that a latent variable encodes whether this object is round-and-red. As a result, in the test set, if there a green apple, it will fail the and may not be able be identified an apple. However, under the framework of C-Disentanglement color and shape are disentangled in the latent space, and the shape and color contribute the classification separately, resulting in higher probability for the green apple also to be identified as an apple.
Table 2: Compare how ground truth genera- tive factors are recovered (MIC/TIC) and how disentangled they are the latent space clas- sification on 3dshape dataset with shift severity 0.5.
80.752:; 84.5
80.7 84.
80.3 81. 84.4
Accuracy(%))
Methods
MIC f
TIC
IRS
12.7 10.1
12.9 32.4 10.2
181
VAE
21.9
12.1
0.82
8-VAE
22.1
12.4
0.85
FactorVAE
24.3
15.6
0.89
shift severityy@.4
shift
sevarityaa.s
shift severity-0..
beta
CAUSAL-REP
cdVAE
CAUSAL-REP
26.8
16.1
0.88
eVAE
22.4
12.4
0.84
Figure 5: Compare edVAE with 9 Vac. CAUSAL-RED classification under distribution shift. T represents accuracy on the target data, represents performance drop when the classifier trained on the source data is directly tested on the target data.
GMVAE
23.2
12.8
0.81
edVAE
31.9
20.2
0.89
(Q.22 edVAE better discovers the ground truth generative factors. In the task of shape classification, we further examine how well the learned representations ap- proximate the ground truth generative labels and what extent they causally disentangled. Table 2 shows that cdVAE outperforms all baselines in approximating the generative factors and the disentanglement.
5.3 Ablations studies
We further conduct ablation studies to show that providing inductive bias indeed improves the discovery of generative factors in the latent space. Concretely, we compare cdVAE with conditional VAE (cVAE) [12] and GMVAE [7]. Compared with vanilla VAE, the method proposed uses label information provide inductive bias to confounders for partitioning the observational dataset. cVAE has the label information compared with vanilla VAE and GMVAE a VAE modelled by a mixture of Gaussian. As shown in Table cdVAA has universally better results, showing the necessity of introducing bias to factors discovered in the latent space. We also investigate how choice C affect the model performance and how to adapt C- C-Disentanglement the existing method aiming for statistical independent, as shown in appendix C.
Related work
Disentangled Representations The pursuit for disentangled representation can be dated to the surge of representation learning and is always closely associated with the generative process in modern machine learning, following the intuition that each dimension should encode different features.

---

[6] attempts to control the underlying factors by maximizing the mutual information between the images and the latent representations. [8] propose a quantitative metric with the information theory. They evaluate the disentanglement, completeness, and informativeness by fitting linear models and measuring the deviation from the ideal mapping. [9. 11, 5. 18] encourage statistical independence by penalizing the Kullback-Leibler divergence (KI.) term in the VAE objective. However, the non-causal definitions of disentanglement fail to consider the cases where correlated features in the observational dataset can be disentangled in the generative process. Such a challenge is well-approached through line research from the causal perspective.
Causal Generative Process. Causal methods are widely used for eliminating spurious features in various domains and improving understandable modelling behaviours|25, 26, 14]. is not until [23] that it was introduced for strict characterization of the generative process. [23] first provided a rigorous definition of causal generative process and the definition of disentangled causal representation as the non-existence of causal relationships between two variables, i.e., the intervention on one variable does not alter the distribution of the others. The authors further introduce interventional robustness
evaluation metric and show its advantage on multiple benchmarks. [21] follow the path of [23] and further propose two evaluation metries and the Candle dataset. The confounded assumption allows for correlation in the latent space without tempering with the disentanglement in the data generative. Despite effective evaluation tools, there is still a missing piece on how to infer set of causally disentangled features. Using the proposed evaluation metric as regulation, the model implicitly assumes unconfoundedness and it falls back to finding statistical independence in the latent space. The problem of unrealistic unconfoundedness assumption is identified by [24]. They assume that confounders exist but they are unobservable. They further propose evaluation metric considering the existence of confounders, that causally disentangled latent variables have independent support measured by the IOSS score. Similar to the evaluation metrics introduced in [23, 21J, IOSS is also a necessary condition of the causal disentanglement. More importantly, as in previous work focusing on obtaining statistical independence, such regulation suffers from the identifiability issue.
Weak Supervision for Inductive Bias. The identifiability issue in unsupervised disentangled representation learning is first identified in [16]. Specifically, they show from the theory that such learning task is impossible without inductive biases on both the models and the data. Naturally, a series of weak-supervised or semi-supervised methods [4, 1, 2] proposed with learning objective of statistical independence or alignment. In this paper, we take a step further for the confounding assumption, assuming that the confounders are observable with proper inductive bias so that the latent representation can be better identified. We, similarly, adopt partial labels of the dataset as the supervision signal. We treat the labels a source of possible confounders and allow the learning of correlated but causally disentangled latent generative factors to be learned.
7 Conclusion and discussions
In this paper, we recognize the importance of bringing in confounders with proper inductive bias into the discovery of causally disentangled latent factors. Specifically, we propose a framework, C-Disentanglementt that introduces the inductive bias from the domain knowledge and method, cdVAF, to identify the generative factors. Although cdVAE helps with discovering generative factors under confounders, the learning process relics on the domain knowledge, the choice of C, and that C has to be discrete/categorical and have a finite number of realizations. How to generalize to continuous C or select effective from the available knowledge set is beyond the scope of this paper and will be investigated in future studies. We hope this work could bring attention to the importance of inductive bias of confounders and inspire future works in this direction. In addition, better capturing of generative factors could bring more controlled generative process and positive social impact.
10

---

References
[1] K. Ahuja, J. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 35:15516-15528, 2022.
[2] J. Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. arXiv preprint arXiv:2203. 16437, 2022.
[3] C. Burgess and H. Kim. 3d shapes dataset. http://sshhahccomddeeemmm/////hs 2018.
[4] Chen and K. Batmanghelich. Weakly supervised disentanglement by pairwise similarities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3495-3502, 2020.
[5] R. Chen, X. Li, B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[6] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
[7] N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and
M. Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint 1r1.v:161.00264. 2016.
[8] C. Eastwood and C. K. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
[9] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
[10] D. Horan, E. Richardson, and Y. Weiss. When is unsupervised disentanglement possible? Advances in Neural Information Processing Systems, 34:5150-5161, 2021.
[11] H. Kim and A. Mnih. Disentangling by factorising, In International Conference on Machine Learning, pages 2649-2653 PMLR, 2018.
[12] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1412611144 2013.
[13] B. Kinney and G. S. Atwal. Equitability, mutual information, and the maximal information coefficient. Proceedings of the National Academy of Sciences, 11100)3354--339, 2014.
[14] X. Liu, H. Lu, J. Yuan, and X. Li. Cat: Causal audio transformer for audio classification. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[15] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[16] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14-4124. PMLR, 2019.
[17] F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weakly- supervised disentanglement without compromises. International Conference on Machine Learning, pages 6348 6359. PMLR, 2020.
[18] E. Mathieu, T. Rainforth, N. Siddharth, and Y. W. Teh. Disentangling disentanglement in variational autoencoders. In International Conference on Machine Learning, pages 4402 4412. PMLR, 2019,
[19] J. Pearl. Causal inference in statistics: An overview, Statisties surveys, 3:96-146, 2009.

---

[20] J. Pearl. The do-calculus revisited. arXiv preprint arXiv: 2012.
[21] A. G. Reddy, V. N. Balasubramanian, al. On causally disentangled representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8089-8097, 2022.
[22] J. Soch, T. B. of Statistical Proofs, T. J. Faulkenberry, K. Petrykowski, and C. Allefeld, Stat- StatProofBook 2020, Dec. 2020.
TrooBBookSaatroooBBok..gtttt
[23] R. Suter, D. Miladinovic, B. Schölkopf, and S. Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, pages 6056-6065. PMLR, 2019.
[24] Y. Wang and M. I. Jordan. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv: 1rX.v:2109.033795 2021.
[25] Y. Wang, D. Liang, L. Charlin, and M. Blei. Causal inference for recommender systems. In Proceedings of the 14th ACM Conference Recommender Systems, pages 426-431 226-431,22 2020.
[26] C. Xu, C. Liu, X. Sun, S. Yang, Y. Wang. C. Wang, and Y. Fu. Patchmix augmentation to identify causal features in few-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[27] M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9593 9602, 2021.
[28] Zhu, An, and Huang. Understanding the generalization benefit of model invariance from data perspective. Advances in Neural Information Processing Systems, 34:4328 4341, 2021.
12

---

A Introduction of do calculus.
Do-calculus consists of three rules that help with identifying causal effects.
Rule A.1 (Insertion/deletion of observations).
P(y]do(z), 2, w) = P(yldo(z),w)
(Y 1 Z|X, W)
(13)
Rule A.2 (Actionn/bsservation exchaas
Plyj)o(()),do<z,ee w) 2( if (Y 1 2 W)G
(14)
Rule A.3 (Insertion/deletioon of actions).
Ply)do(x)) do(=), w) = P(yldo(x),w)
(Y IL Z|X, W (x2(W)
(15)
where Gx is the graph with all incoming edges to X being removed, Gw is the graph with all outcoming edges to W being removed, and Z(W) is the set of Z-nodes that are not ancestors of any W-node.
Intuitively, Rule A.1 states when an observant can be omitted in estimating the interventional distribution, Rule A.2 illustrates under what condition, the interventional distribution can be estimated using the observational dataset, and Rule A.3 decides when we can ignore an intervention.
B Proofs
B.1 Proof of Proposition 2.1
Proposition B.1. Let Zi and Zo be two random variables, C' be the ground truth confounder C is a superset of or is equivalent to C* i.e., Oa C C, with e being realization of C, we have
P(Zz) (o 71)) = Σ C C
(16)
CEC
no C C is descendent of
Proof.
P(Zz)d((Zi)) = P(Ze (o( ()
P(Zz)do(Z)), (1)), C)
Rule A2
P(Z2| C)
P(C]do(21))
Rule A.3
P(C)
P(Zz)do(Z1)) = Σ
C= C=
oEC
B.2 Proof of Theorem 4.1
Theorem B.2. Suppose that the latent variable Z on dataset given c Gaussian
N()((X), (X)). Specifically,
P(Z X) (27) D/2 del(ET)-112 exp (-- (( (Z (2-))))
where Z E RDD, (X) is diagonal for all c, we huve
1. Σ
((Z,X)) = 0.
(17)
13

---

Proof. We suppose that
P = = (2x) D/2 dee( 1/2 exp (-212-1-) (=) -1))
(18)
where we omit X for simplicity and is the dimension of Z for any given c. By definition of la (Equation (9)) and proposition 2.1,
d (Z)
(19)
Σ ,
(20)
Σ d( -- d(E|ziZZ2,,E111
(21)
where we denote z = X = c] for simplicity. Notice that Z ~ Ni,, E RD, we therefore know that the conditional distribution of any subset vector Z given the complement vector Zj. is also multivariate Gaussian distribution [22]
ZiIZi (u 2a1s)
(22)
where
Me 11 + Ei, (2) ng), Enls ΣE,
-
(23)
given that Eg is nonsingular.
Hence we know that the first expectation in Equation (21) becomes
E| = uu +
(24)
assuming that DD, ; is nonsingular. Since E| = μij the loss Le can be written as
Le = -= d(pg + (EE
(25)
We assume further that Dd is a diagonal matrix. Therefore Dú,--i = 0 is zero row vector. Then
L. = Σ d(() =
(26)
C Experimental Details
C.1 Experimental Details
The experiments are conducted on 4 NVIDIA GeForce RTX 2080Ti. In each experiment, we repeat times with different seeds and report the averaged results. In all experiments, only partial information on the ground truth confounder is provided. Specifically, for example, the 3dshape dataset, we first make some predefined rules, such as 70% cubes are red". Then we generate 700 red cubes and 300 cubes in other colors. The generation process naturally divides the dataset into different subgroups, and we can thus explicitly control how inductive bias is provided, i.e., the grouping. In the celebA dataset, since we do have access the ground truth generative factors, so we assume any label sets only contain partial information.
14

---

Table Compare vdE with 3-Vae, CAUSAL-REP on classification under distribution shift. T represents accuracy on the target data, S represents the performance on the target domain when the classifier trained on the source data is directly tested on the target data.
shift 0.4
shift
0.5
severity 0.6
Methods
Ace-S
Acc-T
Acc-S
Ace-T
Acc-S
Acc-T
CAUSAL-REP
94.1++0.44
82.1+0.08
94.3++0.03
81.9++.07
94.3++0.22
81.8++0.11
3-VAE
93.440.07
80.710.12
93.620.05
80.7+0.03
93.440.04
80.310.09
cdVAE
94.6::0.02
84.520005
94.640.04
84.4+0.05
94.5.0003
84.400.04
C.2 Ablation study
We investigate how the choice of C affect the model performance and how to adapt C-Disentanglement to the existing method aiming for statistical independence, shown in appendix C.
Table Adapt cdVAE to existing methods. Table 3: Performance under different C on shape The IOSS and Reconstruction loss are measured classification on 3dshape dataset, shift sever- based on image generation task and the perfor- mance drop is measured on shape classification on the target domain under shift severity=0.5.
ity=0.5.
Choice of C
Acc- T
IRS
C=
79.2
0.82
Methods
IOSS |
Recon &
Ace-T
C = C
88.2
0.89
partial C
84.5
0.87
IOSS
0.14
0.12
81.8
cdVAE IOSS
0.12
0.08
84.5
C-Disentanglement improves the learning of ground truth generative factors under reasonable choice of label set. To understand how the choice of C affects the performance, we repeat the shape classification task with different choices of C under 50% shift severity. When with C, we assume the generative process is unconfounded, and edVAE degrades to the vanilla VAE model. With partial C, we partition the data according to only 2 values of the shifting variables instead of 4. With full C, we provide the full confounders. As shown in Table 3, even partial information of the confounders improves the model performance in OOD generalization and obtains more robust latent representations.
ddapting C-Disentanglement to existing works further improve their performance. We compare the performance between regulation through IOSS[24] and cdVAE + IOSS in image generation and classification tasks on 3dshape dataset. In the cdVAE + IOSS, we apply additional regularization terms based the Z. The results show that C-Disentanglement framework could further improve the performance with desired level of inductive bias given.
C.3 Pseudo-code
C.4
Additional Experimental Results
The classification accuracy on both the source and the target distribution with variance is given in the table below.
15

---

Algorithm Train VAE such that the latent representation is causally disentangled
Input: Number of labels Nc. training data X with labels ratio of each calegories/connfunndrss P(C = c) in the training set, dimension of latent space D
I: for E X do
2: for C do
3:
Define Z = ZZ|z, C = c). and obtain from encoder Z (x) for each
assuming (x) to diagonal matrix:
Oene (x) = je,, diag(E")] E R2d, 4e E ERR, diag(2>) E Rd
(27)
4:
Sample from Z (x) [x()):
Z n¹ +
(28)
5:
Parametrize TO ~ ((p==(1) a- (x)) E R with neural network.
6:
Regulate the covariance matrix to be identity matrix with KL divergence
1 det Ee + tr(9))
DKL = log
(29)
7:
end for
8:
Normalize =1= (,.)..., TNo such that ||[]|| = 1.
....,
9:
Compute classification loss between II with label
Lets (II,.c))
(30)
10: Let 7(x) = E (z), and obtain the reconstructed sample from decoder:
x'
θ Odeo Z(x)). Compute reconstruction loss for Z(x):
Lree = mse (x', x))
(31)
Compute total loss
Lrotat (1)) = Lron + Lidss + Σ
DiL
(32)
CEC
and update encoders and decoders. end for
16


Reviews:
[{"review_id": "LvZvQL6unQ", "reviewer": "Reviewer_Mytf", "summary": "The paper proposes a new framework called Confounded-Disentanglement (C-Disentanglement) to learn disentangled representations by using inductive bias of confounding. Instead of the common statistically independent assumption for latent distribution which is usually not true in real-world datasets, the proposed framework partitions the observed data distribution based on a label set C and regulates the latent distribution separately for these subsets. \n\nThe proposed framework is then implemented in a Variational AutoEncoder (VAE) setting, termed cdVAE. Practically, the proposed method models the latent distribution as a mixture of Gaussian model in which each individual Gaussian corresponds to a specific value of C. \n\nThe paper shows the effectiveness of the proposed method on the image generation task on CelebA and Candle datasets, and on a classification task under distribution shift on 3DShapes dataset.", "questions": "I have included all my questions that I would like the authors to answer in the weaknesses section above. The answers to all these issues are crucial to my final recommendation.", "limitations": "The authors have adequately addressed the limitations of the proposed method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper extends research in disentangled representation learning by leveraging inductive bias of confounding that makes it more applicable to real-world datasets that contain correlations in generative factors.\n- The proposed framework and method are technically sound.\n- The results seem promising. \n- The proposed method is evaluated on multiple tasks and multiple datasets which shows the robustness of the proposed method.", "weaknesses": "- The proposed method requires domain expertise knowledge to define the confounders. This can be easy in some cases but could be costly (in terms of time and cost) and error-prone in many cases. What would happen if an error is introduced in defining the label set? Also, how to distinguish from the cases where each subset of data (defined by a specific value of the label set) still contains correlations among generative factors e.g., in the example of Figure 3, if the imported apples contain large green apples and small yellow apples? Is it a requirement that the label set needs to be granular enough so that no correlations remain in the subset corresponding to each specific value of the label set? \n\n- What cases from section 3 do the experiments from the paper fall in? Please discuss this in detail.\n\n- The proposed method cdVAE effectively boils down to modeling the latent distribution as a mixture of Gaussians. How is it different from GMVAE [7] then?\n\n- Could the authors explain in more detail how the objective in CausalVAE [24] is fundamentally different from this paper?\n\n- The paper mentions that using a Mutual Information (MI) objective for latents is not sufficient/appropriate. Then, shouldn’t the evaluation metrics based on MI be avoided as well? Also, why not use DCI/D score in Table 2?\n\n- Can the authors explain in a bit of detail how all the evaluation metrics for disentanglement are being calculated and why UC and CG scores can't be calculated for CelebA? Can't the attribute annotations in CelebA be used as a proxy for generative factors?\n\n- The shape classification generalization experiment setting is a bit unclear. Can the authors please explain it again, maybe with an example?\n\n- All the results need error bars.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate Reviewer Mytf for the insightful feedback and suggestions. We are encouraged that Reviewer Mytf finds our work solid and promising. We address the concerns and questions raised by the reviewer below. Should Reviewer Mytf have any other questions or concerns, please do not hesitate to reach back. \n\n---\n\n> Q1 Whether the label set needs to be precise and exact when providing the inductive bias.\n\nWe thank the reviewer for this insightful and interesting question.\n\n- The proposed method is robust to errors, as using irrelevant information as confounder will not affect the learning. (Proposition 2.1 and Figure 3d).\n- When the confounder is not inclusive/granular enough, the generating factors can be partially recovered. In the reviewer's example, the shape can be disentangled from size and color. \n- We use partial $C$ in all our experiments. The results show that mild information about $C^*$ could help obtain robust and causally disentangled latent factors.\n- We show performance gain even with randomly selected $C$ CelebA in **Table 2 on the 1-page pdf** .\n\n---\n\n> Q2 What cases from section 3 do the experiments from the paper fall in? Please discuss this in detail.\n\n- Except for our method, the rest of the methods assume unconfoundedness, which falls into **case 1** where $C=\\emptyset$.\n- Our method in table 1 and table 2 contains partial information on the ground truth confounder **(case 2)**, as it is the most practical situation when applied to real-world applications.\n- We further conduct the ablation study to explore how different levels of inductive **(case1, 2, 3)** affect the final performance. The result is given in appendix D.2 table 3.  \n\n---\n\n> Q3 Difference between our work and GMVAE[7].\n\nThe main difference lies in whether confounder is modeled. \n\n- In our method, we partition the dataset according to realizations of the confounder, and within each subset, we model the latent distribution as Gaussian and it obtains C-Disentanglement.\n- In GMVAE, the number of Gaussian is a hyper-parameter and there is no specific partition for each Gaussian distribution. Thus the causal disentanglement can not be guaranteed.\n\n---\n\n> Q4 Difference between our work and CausalVAE[24].\n\n-  The goal of our work is to infer generative factors in the latent space, whereas the goal of CausalVAE is to use latent variables to encode any given labels.\n-  We require the latent variables to be causally disentangled, whereas CausalVAE tries to learn a causal graph of the given labels and requires the latent variables to follow the same graph structure.\n-  We explicitly model the confounder whereas CausalVAE assumed unconfoundedness.\n\n\n---\n\n> Q5 The paper mentions that using a Mutual Information (MI) objective for latents is not sufficient/appropriate. Then, shouldn’t the evaluation metrics based on MI be avoided as well? Also, why not use DCI/D score in Table 2?\n\n\nWe totally agree that the MI evaluation metrics should be avoided in measuring how well the latent variables being causally disentangled. \n\n- However, in table 2, we want to show how much information of the ground truth generative factors is captured by our learned latent representation as a whole. Therefore, we compute MIT and TIC. High mutual information among them indicates the learned set of latent variables better captures the ground truth generative factors. \n- We include $D$ in Table 1 to demonstrate that it cannot accurately reflect level of causal disentanglement of a representation. \n- We do not include $D$ as it is used to compare among latent variables, but not between the learned representation and the ground truth. \n\n\n---\n\n> Q6 why UC and CG scores can't be calculated for CelebA with annotation attributes as a proxy of ground truth generative factors?\n\nWe thank the reviewer for this question and we will also add the detailed explanation and calculation of all evaluation metrics to the appendix. Here we focus more on UC and CG score due to the word limit. \n\n- The UC metric evaluates how well distinct generative factors are captured by distinct sets of latent dimensions with no overlap. $UC = 1- \\mathbb{E}[\\frac{1}{S}\\sum_{I,J}\\frac{Z_I \\cap Z_J}{Z_I \\cup Z_J}]$ where $S$ is the number of pairs of ground truth generative factors $(G_I, G_J)$, $Z_I, Z_J$ is latent variables. With annotation attributes as a proxy of generative factors, we could calculate UC score. **The results are given in the 1-page pdf as Table 1**. \n\n- The CG metric evaluates how well the learned generative factor set is able to generate counterfactual instances in a flexible and controlled manner. $CG = \\mathbb{E}[|ACE_{Z_I} - ACE_{Z_{-I}}|]$ where ACE is the averaged causal effect. With celebA dataset we are not able to generate any new images but only sample from it. Consequently, we are not able to obtain enough data points for counterfactual images, and thus could not calculate CG scores.\n\n---\n\n> Q7 The shape classification generalization experiment setting is a bit unclear. Can the authors explain it again, maybe with an example?\n\nIn shape classification experiment, we test how our cdVAE works under the distribution shift. In the source set, we set a confounder to bring in correlation and sample a certain amount of data accordingly. For example, the ground truth generative factors are size, shape and color, and the confounder is the type of objects, which requires a certain percentage cubes are red, cylinders are yellow and etc. This way, spurious correlation exists in the source set between the object shape and color. In the target set, images are generated with random combination of the generative factors (the will also be green cubes, or blue cubes which may not exist in the source set, and the distribution also changes). We train the model on the source and test it on both sets. \n\n--- \n> Q8 Error bar\n\nThe error bar for Figure 5 is given in the appendix and We have added the error bars for table 1 and table 2 to the revised paper. \n"}, {"review_id": "QzGxN9yi6L", "reviewer": "Reviewer_TdLu", "summary": "This paper delves into the domain of representation learning, focusing on the discovery of meaningful generative factors within a latent space. The authors propose the C-Disentanglement framework, which incorporates the inductive bias of confounders through labels or knowledge obtained from domain expertise. This framework enables the identification of generative factors even in the presence of confounding variables. The paper offers a comprehensive theoretical analysis and a practical approach to effectively identify these factors. Empirical results obtained from diverse settings validate the efficacy of the proposed method.", "questions": "\tHow can we ensure that the number of C is adequate?\n\tIn Line 215, Equation (10), the right-hand side of the equation involves several terms that are not enclosed within absolute value symbols. Does these terms are non-negative?\n", "limitations": "The authors adequately addressed the limitations of the work.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "\tThis paper is very well written and easy to follow.\n\tThe research addresses an important and fundamental question: how to identify Causally-Independent generative factors in the presence of common causes.\n\tThe idea proposed in this work is simple but certainly make sense. And the sufficient experiments validate the effectiveness of the proposed method. \n", "weaknesses": "\tThe assumptions made in this paper are quite strict. One of the main concerns is that the paper assumes the probability distributions of the generative factors to be Gaussian distributions. This assumption may limit the applicability and generalizability of the proposed method to real-world scenarios where the underlying distributions may not strictly follow Gaussian patterns. Another strong assumption in this paper is the linear mixing of the generative factors. While this assumption might simplify the modeling process, it may not accurately capture the complex relationships and interactions between the generative factors in real-world scenarios. It is important for the authors to discuss the potential impact of this assumption on the performance and generalizability of the proposed method.\n\tThe proposed inductive bias in this paper is hard to obtain. The utilization of domain expertise knowledge as an inductive bias is an expensive and non-scalable approach.\n\tThe evaluations of the proposed method primarily focus on quantitative aspects, and incorporating qualitative evaluations would provide a more intuitive understanding of the method's performance.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe greatly appreciate Reviewer TdLu for the valuable feedback and suggestions. We are encouraged that Reviewer TdLu finds the problem setting is fundamental and important, and our paper in good clarity and quality. We address the concerns and discuss the insightful suggestions raised by the reviewer below.\n\n---\n\n> [Weakness 1] The assumptions made in this paper are quite strict. One of the main concerns is that the paper assumes the probability distributions of the generative factors to be Gaussian distributions. This assumption may limit the applicability and generalizability of the proposed method to real-world scenarios where the underlying distributions may not strictly follow Gaussian patterns. Another strong assumption in this paper is the linear mixing of the generative factors. While this assumption might simplify the modeling process, it may not accurately capture the complex relationships and interactions between the generative factors in real-world scenarios. It is important for the authors to discuss the potential impact of this assumption on the performance and generalizability of the proposed method.\n\nWe thank the reviewer for this insightful question and we will add a discussion to our paper for these assumptions. \n\nWe would like to clarify that the framework we proposed in this paper, Confounded Disentanglement (C-Disentanglement) is not confined in the VAE but a general framework. It does not have assumptions on distributions of the observational dataset.\n\n\n**Gaussian prior**\n\n\nOur cdVAE, under this premise, is an algorithm and a practical solution that utilizes this framework. In cdVAE, we also do not assume the distribution of the observed data. In VAE-based methods, the latent is Gaussian, but there always exist a map from Gaussian to the observed data distribution that can be learned.\n\n\n**Linear mixing of the gaussian distributions**\n \n\nWe want to clarify that the linearity does not come from our assumption but statistical derivation. The conditional probability $P(Z|X)$ can be factored into $$P(Z|X) = \\sum_{c \\in C} P(Z|X, \\mathbf{C} = c)P(\\mathbf{C} = c|X).$$ \n\nGiven our assumption that the latent variables are Gaussian for each labels, this reads \n$$ P(Z|X) = \\sum_{c \\in C} P(\\mathbf{C} = c|X)\\mathcal{N}(\\mu^c(X), \\Sigma^c(X)) $$ where $P(\\mathbf{C} = c|X)$ is the mixing coefficient. To enable more flexibility, we model this coefficient as a Gaussian variable with its own mean and variance instead of using a hard assignment. \n\n---\n\n> [Weakness 2] The proposed inductive bias in this paper is hard to obtain. The utilization of domain expertise knowledge as an inductive bias is an expensive and non-scalable approach. \n\n- We thank Reviewer for the raising the question of accessibility of the inductive bias. Empirically, while the ground truth $C^*$ is hard to obtain, it is not that hard to obtain some additional labels of a dataset. we observed improvements even with $C$ contains partial information about the ground truth confounder (as shown in appendix D.2).\n\nWe have added more results on celebA with randomly-chosen labels as shown **in Table 2 in the 1-page pdf** for rebuttal. The results universally outperform the baseline models in both accuracy and casual disentanglement metrics.\n\nAlthough a challenging task, one way to automatically generate proper $C$ is to train a model for proper $C$ in an end-to-end fashion, or infer $C$ with the help of large pretrained models. We will save it to future works.\n\n \n---\n\n> [Weakness 3] The evaluations of the proposed method primarily focus on quantitative aspects, and incorporating qualitative evaluations would provide a more intuitive understanding of the method's performance.\n\nWe appreciate the reviewer for suggesting qualitative evaluations for more intuitive understanding of the methods. We currently have an ablation study on how different choices of $C$ (partial information, full information, no information about the ground truth $C$) affect the learning in appendix D.2 Table. We will add more qualitative evaluations in revised version.\n\n---\n\n\n>[Q1]How can we ensure that the number of C is adequate? \n\nThanks for this insightful question. \n\n- In our paper, we propose cdVAE under the framework of C-Disentanglement to learn causally independent representational variables in the latent space. Instead of enforcing global statistical independence among variables on the observational dataset, we partition the dataset into subsets according to realizations of $C$, and regulate these variables within each subset.\n\n- The number of $C$ determines how dataset are partitioned, thus an adequate $C$ is expected to leave enough samples in each subset for model inference and we now rely on domain expertise for the selection. Since one of the main messages of our paper is to encourage learning of causally disentanglement through modeling confounders, we save the automated selection of $C$ to future works.\n\n---\n\n\n>[Q2] In Line 215, Equation (10), the right-hand side of the equation involves several terms that are not enclosed within absolute value symbols. Does these terms are non-negative?\n\nWe apologize for the typo in equation 10. \n\nIt should be $l_c=\\sum_{i=1}^{D}d\\left[\\mathbb{E}(Z_i | do^c({Z_{-i}}), X),  \\mathbb{E}(Z_i | X)\\right] = 0$, where $d$ is the euclidean distance and it is non-negative."}, {"review_id": "LQdMaRuXpJ", "reviewer": "Reviewer_j8n4", "summary": "This paper is motivated by the assumption that a few semantically significant generative factors generate real-world data. Then they study causal independence in generative methods by introducing a new framework, C-Disentanglement, that discusses the identifiability issue of generative factors regarding the inductive bias of confounder to handle the disadvantage brought by the statistical independence assumption of previous methods. At last, the authors implement their method, cdVAE, and evaluate it on three datasets. The results show that cdVAE performs better in the latent space compared to the existing methods and performs better in downstream tasks.", "questions": "1. I think implementing your ideas in VAE is a limitation for you. I hope you can implement your ideas in more generative methods.\n2. The Related Work in the main paper is very similar to the Related Work in the appendix, so I doubt the significance of the Related Work in the appendix. However, I do not think this is a weakness of your paper, so I put this into Question.", "limitations": "The authors point out the limitation of their method at the end of the paper; that is, the choice of C is difficult but essential, which is the same as our understanding of their method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The motivation is obvious, which emphasizes the importance of causal independence in generative methods and illustrates the advantages of causal independence for statistical independence, which is very interesting and important. To the best of our knowledge, the authors are the first to study how to explore causal independence confounders in generative methods. \nThis article has complete content. It conducts a thorough investigation of the background. Then it proposes a new framework and elaborates in detail. Moreover, it has several experiments on three datasets to demonstrate its advantages.", "weaknesses": "This article includes but is not limited to the following shortcomings:\n1. There was much ambiguity about collaboration, which led me to rely a lot on my guesswork in understanding the article, like:\n\na) D in line 141 and D in equation 7 do not seem like the same thing but use the same symbol.\n\nb) D_KL in Figure 4 and D in Equation 7 seem to represent KL divergence but in different symbols.\n\nc) What is the meaning of \\Pi? I do not find its definition. Is that represent the soft assignment of samples? If it is, what is the connection between \\Pi and \\pi?\n\n2. In Table 1, you evaluate your method on CelebA and Candle datasets. However, the measurement metrics between these datasets are different. You claim that the CelebA dataset does not have ground truth factors. However, the experiments on only two datasets cannot fully illustrate your methods' advantages, especially since CelebA does not have ground truth factors.\n3. There are many kinds of generative methods, like the diffusion model. There is no comparison between your method and other generative methods.\n4. The baseline vanilla VAE (2013), \\beta-VAE (2016), FactorVAE (2018), CAUSAL-REP (2021), which do not compare, cVAE (2013), GMVAE (2016), are out-of-date.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank Reviewer j8n4 for the effort of reviewing our paper and the recognition of the novelty and significance of the proposed method. We hope that our response and additional results could clarify doubts toward our work.\n\n---\n\n> [Q1] Notation issue \n\nWe would like to thank reviewer j8n4 for pointing out some notation issues in the narrative. We have carefully revised our manuscript accordingly to ensure the consistency of these notations.\n\nFor questions listed here. $D$ in line 141 denotes the dimension of the latent representation and the $D$ in equation 7 should be changed to $D_{KL}$ to represent the KL divergence. For $\\pi_c$ in equation 11, it is the  mixing coefficient as stated in line 220. $\\pi_c = P(C = c|X)$ reads the probability of occurrence of $C = c$. And in line 223, $\\pi_c \\sim \\mathcal{N}(\\mu^{\\pi_c}(X), \\Sigma^{\\pi_c}(X))$ where the $\\mathcal{N}(\\mu^{\\pi_c}(X), \\Sigma^{\\pi_c}(X))$ should be $\\mathcal{N}(\\mu^{c}(X), \\Sigma^{c}(X))$. \n\n---\n\n>[Q2] Lack of ground truth for evaluation of causality.\n\n- In addition to CelebA real-world dataset, we also evaluate on two synthetic datasets, Candle and 3dshape, that have ground truth generative factors. The results (in Table 2 & Figure 5)  show that our method has better performance in obtaining causal disentanglement, can recover ground truth generative factors better, and is more robust to distribution shift.\n\n- CelebA, Candle & 3dShape datasets are commonly used benchmarks for evaluating causal representation learning methods. Our work is consistent with this line of works. For example, [23,24] tested their method on CelebA and their own small-scale synthetic dataset. [21] evaluated their method on Candle. [R1] also tested on one real-world dataset and one synthetic dataset. \n\n[R1] Lachapelle, Sébastien, et al. \"Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA.\" Conference on Causal Learning and Reasoning. PMLR, 2022.\n\n---\n\n> [Q3] Comparison with other generative models like Diffusion.\n\n- Our contribution is not to introduce a generative model, but to introduce a general representation learning framework, C-Disentanglement, that can be applied to all models that have a latent space.\n\n- Our representation learning framework aims for capturing causally disentangled representations in the latent space. The contribution is significant especially when the ground truth generative factors are *statistically correlated* but *causally disentangled*, which cannot be captured by exiting methods (since the existing methods learn *statistically independent* latent features). \n\n**Additional baselines: latent diffusion[R2] and InfoGAN[R3]**\n\nAs suggested by the reviewer, we have further added two additional baselines.\n\n- [Additional generative model, latent diffusion] We have added the latest latent diffusion model [R2] as a baseline.\n- [Additional generative model, IB-GAN] We also have added another baslie: IB-GAN [R3]. It is the latest GAN-based method on disentangling latent representation although the definition of disentanglement in this paper differs from ours. \n- **The result is given in Table 3 of the 1-page pdf.**\n\n[R2] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[R3] Jeon, Insu, et al. \"Ib-gan: Disentangled representation learning with information bottleneck generative adversarial networks.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 9. 2021\n\n---\n\n> [Q4] Baseline model out-of-date?\n\nSince our contribution lies in learning causally disentangled representation and most of the baselines use VAE as the backbone, we also choose VAE for a fair comparison. We believe the baseline models are not out-of-data, since they are SOTA causal disentangled representation learning methods.\n- For example, We compare our method with CAUSAL-REP [23], which is the sota method on obtaining causal disentanglement.\n- We compare with cVAE and GMVAE as ablation studies, because cVAE also has additional label information, GMVAE is also a mixture of Gaussian model. \n- By showing that ensuring C-disentangled representation under confounder modeling improves VAE, we anticipate similar improvement in other models.\n\n---\n> [Q5] Implementing the ideas only in VAE is a limitation?  Implement your ideas in more generative methods?\n\nWe thank Reviewer j8n4 for the recognition of the significance and soundness of our proposed idea. We would like to clarify that, the framework we proposed in this paper, Confounded Disentanglement (C-Disentanglement) is not confined in the VAE but a general framework that can be used in common generative models and even outside the scope of generative models.\n\nOur cdVAE, under this premise, is an algorithm and a practical solution that utilizes this framework. We choose VAE because  (1) it is one of the mainstream formulation of the generation problem, and (2) with the Gaussian assumption, perfect C-Disentanglement can be theoretically obtained in an easy-to-implement and effective way as stated in Theorem 4.1 (line 213). \n\n\nApplying the C-Disentanglemt in generative models is a non-trivial task and does not have a common algorithmic solution. For example, diffusion models do not have a latent space. The development of corresponding algorithms under different generative models may go beyond the scope of this paper and we'd like to save it to future works. \n\n---\n\n> [Q6] [Related work in the main paper and appendix]\n\nWe have shortened the related work section in the main paper due to the page limit, and put a complete version to the appendix. \n\n---\nWe thank the reviewer again for the time and effort. We have addressed all the concerns raised by the reviewer. We hope Reviewer j8n4 could consider revising the scores accordingly. Should Reviewer j8n4 have any other questions or concerns, please do not hesitate to reach back. \n\n\n\n"}, {"review_id": "R2Iqvbwc6e", "reviewer": "Reviewer_S7Fk", "summary": "The paper proposed Confounded-Disentanglement a method that makes use of inductive bias of a confounder by leveraging labels/knowledge from domain expertise. The aim is to recover the true/causal generative factors for the observed data which can recover a more meaningful latent representation that can be further useful for downstream tasks especially in the case of distribution shifts.", "questions": "1. A recent stream of disentanglment that also leverages additional/auxiliary information, similar to ICA but even closer to cdVAE are the following [1, 2, 3]. In those cases, the additional information is treated as intervention, or environment. Could you please position your work wrt that? What makes a confounder different than their auxiliary variable? \n2. What are the advantages of interpreting the additional variable as confounder and not intervention?\n3. There are other metrics for measuring the quality of disentangled representations that haven't been used R2, MCC for example\n4. Have you tried the controlled generative process under your framework? What could you conclude from the results? \n\n[1] Lachapelle, Sébastien, et al. \"Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA.\" Conference on Causal Learning and Reasoning. PMLR, 2022.\n\n[2] Khemakhem, Ilyes, et al. \"Variational autoencoders and nonlinear ica: A unifying framework.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.\n\n[3] Lu, Chaochao, et al. \"Invariant causal representation learning for out-of-distribution generalization.\" International Conference on Learning Representations. 2021.\n\n\n---- Update after Author response ----\nI thank the authors for the detailed and timely responses, they have mostly addressed my concerns, so I increase my score to BA.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Originality: the authors propose a new way of leveraging observed confounders to learn disentangled representations.\n- Quality: the manuscript includes both theoretical and empirical evidence for the claims.\n- Clarity: I appreciate the schematics and toy running example to illustrate the intuition of the method.\n- Significance: Better latent representations useful for overcoming domain generalisation challenges is timely and important contribution.", "weaknesses": "The problem of disentanglement, learning meaningful representations in general is an important , unresolved problem and I do agree that when additional information is available, we should make use of it, such as the scenario proposed by the authors of cdVAE. However, I do find the paper lacking in some recent relevant work and the overall presentation would benefit from including/positioning this method with regards to it. Please see more details in the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank Reviewer S7Fk for the effort of reviewing our paper. We are encouraged to see that the reviewer appreciates the quality and the significance of our work. \n\n\nMost of the concerns and questions we believe are due to the confusions from various definitions of disentanglement in existing works. Below, our response clarifies the fundamental differences between our work and the works listed by the reviewer. Specifically, we emphasize our contribution of showing how to use confounders to tackle the mystery of learning semantically meaningful latent representations. \n\nBelow we address the concerns of reviewer S7fk. \n\n---\n\n> [Weakness and Q1] How do the methods [1,2,3], listed by reviewer, differ from our work?\n\nWe thank the reviewer for the valuable suggestions of adding this line of works to the related work section and we will revise accordingly. However, we would like to point out that the works listed here [1, 3] are based on a nonlinear ICA method in [2]. They address different problems compared to our work. \n\n1. The goal of [2] is to obtain conditional factorizable $p(z|u)$ while ours is to obtain causally disentangled $z$. **These two goals are irrelevant and are not mutually exclusive**.\n\n    - Let $z$ be the latent variable, $u$ be the auxiliary variable. The goal of non-linear ICA [2] is to learn $\\tau$, $\\lambda$ such that $$p_{\\tau, \\lambda} = \\Pi_i\\frac{Q_i(z_i)}{constant}exp[\\sum_{j=i}^k \\tau_{i,j}(z_i)\\lambda_{i,j}(u)]\\quad (1)$$\n    - If we further require $z$ to be a two dimensional vector, and $u$ only has one realization. Omitting the normalization constant in the first equation, Eq (1) becomes $$p(z_1, z_2 | u) = Q_1(z_1)\\text{exp}(\\tau_1(z_1)\\lambda_1(u))Q_2(z_2)\\text{exp}(\\tau_2(z_2)\\lambda_2(u))\\quad (2)$$\n    - Our goal is to learn causally disentangled $z$ which satisfies\n\n    $$p(z_i | do(z_{-i})) = \\sum_{u}p(z_i | z_{-i}, u) = p(z_i) \\quad \\forall i\\quad (3)$$\n    \n    - under the same assumption of $u$ and $z$, causally disentangled $z$ in our paper satisfies $$p(z_1 | z_2, u) = p(z_1) \\text{ and } p(z_2 | z_1, u) = p(z_2)\\quad (4)$$ \n\n\n    **It can be easily observed that, the causal disentanglement property in Eq (4) above is irrelevant to the goal of ICA-based methods (2). In fact, it has also been pointed out in [1] that methods in [2] cannot achieve causal disentanglement.**\n\n2. Using [2] as a backbone, [1] and [3] introduce causal representation learning methods. However, our method is different from theirs due to the following reasons:\n    - Although [1] also aims for causal disentanglement, it assumes the dataset is **unconfounded** and specializes in dealing with sparsity regulation of sequential data.\n    - [3] does not aim for causal disentanglement representations; instead it finds an invariant causal graph across different domains. \n\n\n\n---\n\n> [Q2] What are the advantages of interpreting the additional variable $u$ as confounder and not intervention?\n\n\n- As illustrated above, confounder $u$ in Eq (3,4) is conceptually different from the auxiliary variable $u$ in Eq (1,2) (in [1,2,3] pointed out by the reviewer). Auxiliary variables are weak supervision labels conditioned on which we get factorial $p(z|u)$, whereas confounder is a concept in causal inference which is the common parents of generative factors, to allow causally independent factors be statistical dependent in the latent space. \n\n\n- However, $u$ can be interpreted as an inductive bias for both the confounder and the intervention (mathematically, $u$ and $z$ satisfy both (1) and (3)). We defer it to future work to investigate the benefit of combing these two. \n\n- Below, we explain the benefit of modeling the confounder.\n    - Confounder allows causally independent but correlated generative factors to be detected. Assuming unconfoundedness, on the contrary, requires global statistical independence among variables.\n    - For instance, a fruit dataset is generated from three generative factors (color,shape,size). The confounder is the fruit type. Although color, shape and size are causally independent, due to the type of the fruit, they are correlated in the observational dataset (banana is yellow, apple is red and green). But if you fix the fruit type, we could observe that color and shape are not correlated.\n\n\n---\n\n> [Q3] There are other metrics for measuring the quality of disentangled representations that haven't been used R2, MCC for example.\n\nWe choose evaluation metrics to measure \n- (1) whether we can recover the ground truth generative factors in the synthetic datasets where ground truth factors are available, as the metrics 'MIT' and 'TIC' used in Table 1. \n- or (2) whether the recovered factors are causally disentangled (how changing one factor affect the others) in real-world datasets where the ground truth factors are not available, as the metrics 'IRS', 'CG'/'UC' and 'IOSS' used in Table 1 and 2.\n\nMCC and R2 do not fall into these categories and do not measure the quality of causal disentanglement. \n\n\n---\n\n> [Q4] Apply our method to controlled generative process?\n\nWe have used the CG scores [21] (the higher, the better) to measure the controlled generation of the counterfactual images. The CG metric evaluates how well the learned generative factor set is able to generate counterfactual instances in a flexible and controlled manner. Our method achieved higher CG scores as shown in Table 1.\n\n\n---\n\nWe thank the reviewer again for the time and effort. We have addressed all the concerns raised by the reviewer; most of them are due to different definitions of disentanglement in existing literature. We hope Reviewer S7Fk could consider revising the scores accordingly. Please do not hesitate to reach back if the reviewer has any questions.\n\n\n"}]

Author rebuttal:


### Summary of Rebuttal Discussion


We thank all reviewers for the valuable feedback and insightful questions. We are particularly encouraged that they consider our proposed framework and method fundamental, important and novel.

---

### Review Highlights

Reviewer Mytf finds our work "**technically sound**" and "**promising**".

Reviewer TdLu believes that "The research **addresses an important and fundamental question**" and "the sufficient experiments validate the effectiveness of the proposed method"

Reviewer j8n4 agrees that our work is **well motivated and novel**, in that "The authors are the first to study how to explore causal independence confounders in generative methods"

Reviewer S7Fk appreciates the originality, quality, clarity and the significance of our work, and mentions that our paper is a "**timely and important contribution**".

---

### Summary of Core Contributions

We would like to highlight our core contribution as follows:

1. We **recognize the identifiablility issue** of discovering generative factors in the latent space. That correlated but causally independent generative factors cannot be recovered by existing methods assuming unconfoundedness in modeling.
2. We accordingly introduce **a novel and general framework**, **C-Disentanglement**. It is the first framework that discusses how inductive bias of confounder could be explicitly provided via labels/knowledge from domain expertise.
3. We propose **an algorithm, cdVAE**, that discovers causally disentangled generative factors in the latent space. The algorithm sheds light on the easy injection of inductive bias into existing methods.

We hope our work could 

- brings out new perspectives on discovering semantically meaningful latent representations,
- inspires theoretical and empirical ideas in modeling confounder and causal representation learning.


---

### Summary of Changes 

We address individual questions of reviewers in separate responses.

Here we briefly outline the updates to the revised version of our paper based on the reviews. 


**[Problem Formulation]**  We added a discussion of assumptions and the potential influence. (Reviewer TdLu)

**[Experiment]** We added several experimental results as follows

- baselines of Latent Diffusion and IB-GAN (Reviewer j8n4),
- UC scores on CelebA using annotation attributes as a proxy of the ground truth generative factors, and error bars in appendix (Reviewer Mytf) 
- How randomly selected C affects the model performance on real-world celebA dataset. (Reviewer TdLu, Reviewer Mytf)
 
**[Related Work]** We added comparison between our work and another line of work with various definitions of disentanglement (Reviewer S7Fk) and remove similar parts in the appendix (Reviewer j8n4)

**[General]** We revised the notation for consistency and corrected typos. (Reviewer j8n4)



Meta-review: The paper introduces a novel framework called "Confounded-Disentanglement" (C-Disentanglement) for acquiring disentangled representations by considering possible confounding in the latent space. To solve this ill-posed problem, this papers introduces proper inductive bias to identify the causally-disentangled factors. The paper provides empirical evidence of the effectiveness of this approach in tasks such as image generation using CelebA and Candle datasets, as well as in addressing a classification task involving distribution shift using the 3DShapes dataset.

This paper tackes the disentanglement problem in a more practical setting and the proposed method is technically sound. The paper's contribution is expected to be highly beneficial and influential in the field of causal disentanglement and more controllable generation. The experiments in the original paper was not thorough enough, but the authors have added more analysis in the rebuttal. These additioinal experiments should be included in the final version. Overall, I would recommend acceptance of this paper given its solid contribution.
Decision: Accept (poster)