Title: C-Disentanglement: Discovering Causally-Independent Generative Factors under  an Inductive Bias of Confounder
Venue: NeurIPS 2023
Accepted: yes
Keywords: causal disentanglement

causal generative process

generative factors

confounder

inductive bias

disentanglement

causal inference
Abstract: Representation learning assumes that real-world data is generated by a few semantically meaningful generative factors (i.e., sources of variation) and aims to discover them in the latent space. These factors are expected to be causally disentangled, meaning that distinct factors are encoded into separate latent variables, and changes in one factor will not affect the values of the others. Compared to statistical independence, causal disentanglement allows more controllable data generation, improved robustness, and better generalization. However, most existing work assumes unconfoundedness in the discovery process, that there are no common causes to the generative factors and thus obtain only statistical independence. In this paper, we recognize the importance of modeling confounders in discovering causal generative factors. Unfortunately, such factors are not identifiable without proper inductive bias. We fill the gap by introducing a framework entitled Confounded-Disentanglement (C-Disentanglement), the first framework that explicitly introduces the inductive bias of confounder via labels from domain expertise. In addition, we accordingly propose an approach to sufficiently identify the causally-disentangled factors under any inductive bias of the confounder.  We conduct extensive experiments on both synthetic and real-world datasets. Our method demonstrates competitive results compared to various SOTA baselines in obtaining causally disentangled features and downstream tasks under domain shifts.

C-Disentanglement: Discovering Causally-Independent Generative Factors under an Inductive Bias of Confounder

Abstract

Representation learning assumes that real-world data is generated by a few semantically meaningful generative factors (i.e., sources of variation) and aims to discover them in the latent space. These factors are expected to be causally disentangled, meaning that distinct factors are encoded into separate latent variables, and changes in one factor will not affect the values of the others. Compared to statistical independence, causal disentanglement allows more controllable data generation, improved robustness, and better generalization. However, most existing works assume unconfoundedness (i.e., there are no common causes to the generative factors) in the discovery process, and thus obtain only statistical independence. In this paper, we recognize the importance of modeling confounders in discovering causal generative factors. Unfortunately, such factors are not identifiable without proper inductive bias. We fill the gap by introducing a framework named C onfoundedDisentanglement (C-Disentanglement), the first framework that explicitly introduces the inductive bias of confounder via labels/knowledge from domain expertise. In addition, we accordingly propose an approach to sufficiently identify the causally-disentangled factors under any inductive bias of the confounder. We conduct extensive experiments on both synthetic and real-world datasets. Our method demonstrates competitive results compared to various SOTA baselines in obtaining causally disentangled features and downstream tasks under domain shifts. code available here

1 Introduction

Causally disentangled representation learning methods endeavour to identify and manipulate the underlying explanatory causes of variation (i.e., generative factors) within observational data through obtaining causally disentangled representations [24, 8]. Such a representation encodes these factors separately in different random variables in the latent space, where changes in one variable do not causally affect the others. Pursuing causal independence 1 makes it possible to identify the ground truth generative factors that are not statistically independent, such as the color, shape and size in a fruit dataset as shown in Figure 1, which is more realistic and allows more controlled data

generation, improved robustness, and better generalization in out-of-distribution problems.

Despite great success, existing methods suffer from the identifiability issue in discovering semantically meaningful generative factors. The main reason is that most of them equate disentangling in the latent space with enforcing statistical independence [9, 11, 6] in the latent space, or requiring no mutual information[8]. In other words, they explicitly or implicitly assume that the observational dataset is unconfounded in the generative process, which is not realistic in practice. A confounder C is a common cause of multiple variables. It could create a correlation among causally disentangled generative factors in the observational distribution. In the previous example of a fruit dataset, the type of fruit is a confounder, creating a correlation between size and shape (apples are usually red and round objects), thus size and shape can never be captured by a learning objective that requires statistically independent latent representations. Assuming unconfoundedness in such a dataset leads to discrepancies in characterizing the statistical relationship of the generative factors, and makes them non-identifiable .

Considering the limitation of the unconfoundedness assumption, a few of the recent studies take into account confounders [24, 23, 21] in the problem formulation. Specifically, [23, 21] proposed evaluation metrics in the confounded causal generative process. [24] introduced an unsupervised learning scheme by regulating the learning process with the Independence-Of-Support Score (IOSS) under an unobserved confounder, following the theory that if a latent representation is causally disentangled, then it must have independent support. However, it has been shown almost impossible to obtain disentangled representations through purely unsupervised learning without proper inductive biases [10, 16]. While there are a few semi-supervised or weakly supervised methods [16, 17] developed seeking statistical independence, none of the above-mentioned methods considers how to provide inductive bias for confounder in learning causally disentangled representations.

In this paper, we recognize the importance of providing inductive bias to confounder so that the ground truth generative factors can be identified and we fill the gap by introducing the inductive bias via knowledge from domain expertise. Specifically, we propose a framework called C onfoundedDisentanglement ( C-Disentanglement ). C-Disentanglement is, to the best of our knowledge, the first framework that discusses the identifiability issue of generative factors regarding the inductive bias of confounder, and thus opens up the possibility to discover the ground truth causally disentangled generative factors which are correlated in the observational dataset. Importantly, C-Disentanglement is a general framework that includes existing methods (in which unconfoundedness is assumed) as special cases, allowing customized inductive bias of confounders according to diverse user needs.

Under the framework, We develop an algorithm to discover the causally disentangled generative factors in the latent space with inductive bias C , where C is a label set. Instead of enforcing global statistical independence among variables on the observational dataset, we partition the dataset into subsets according to realizations of C , and regulate these variables within each subset.

Our proposed discovery methodology is general and can be applied to various disentangle mechanisms. But as a prototype for a proof of concept, we implement our method, cdVAE, under the context of VAE [12]. Specifically, we formulate a mixture of Gaussian model in which each Gaussian component, conditioned on a specific value of C , represents the distribution of a latent variable that estimates the underlying generative factor. Intuitively, under the previous example of the fruit dataset, if we fix the type of fruit (e.g., the apple) and check the correlation between the color and the shape, we could find that the color change within the apples does not affect the shape distribution, thus can be captured by regulating the statistical independence conditioned on C (a fixed fruit type).

Our methodology is a novel learning paradigm that can discover latent factors through weak supervision labels (i.e., C ).

We experiment with several tasks from image reconstruction and generation, to classification under the out-of-distribution scenario on both real-world and synthetic datasets. We demonstrate that our method outperforms various baselines in discovering generative factors in the latent space.

Summary of contributions: (1) We recognize the identifiability issue of discovering generative factors in the latent space. We accordingly introduce a framework, named Confounded Disentanglement (C-Disentanglement). It is the first framework that discusses how inductive bias of confounder could be explicitly provided via labels/knowledge from domain expertise. (2) We propose an algorithm, cdVAE, that discovers causally disentangled generative factors in the latent space. The algorithm sheds light on the easy injection of inductive bias into existing methods. (3) We conduct extensive experiments and ablation studies across various datasets and tasks. Empirical results verify that

cdVAE outperforms existing methods in terms of inferring causally disentangled latent representation and also show cdVAE's superiority in downstream tasks under OOD generalization.

2 Preliminaries

In this section, we introduce basic concepts in causal inference and then show how to evaluate the causal relationship among variables in latent space.

Causal graph through DAG. The causal relationship among variables can be reflected by a Directed Acyclic Graph (DAG). Each (potentially high-dimensional) variable is denoted as a node. The directed edges indicate the causal relationships and always point from parents to children.

Intervention and do-operator. Intervention is one of the fundamental concepts in causal inference. When we intervene on a variable, we set the value of a variable and cut all incoming causal arrows since its value is thereby determined only by the intervention [20]. The intervention is mathematically represented by the do-operator do ( · ) . Let Z 1 and Z 2 be two variables, P ( Z 2 | do ( Z 1 = z 1 )) characterizes an interventional distribution and reflects how a change of Z 1 affects the distribution of Z 2 . The do-operation and the interventional distribution should be estimated on the interventional dataset. However, in practice, the true distribution of the data is unavailable but an observational subset. As a result, we estimate the interventional distribution from the observational set following do-calculus introduced by [19]. A detailed introduction can be found in the Appendix A.

Confounders bring in spurious correlation. Although the detailed definitions vary from literature, a confounder usually refers to a common cause (i.e., a common parent in the causal graph) of multiple variables and it brings in a certain level of correlation among these variables. Consequently, to estimate the causal effect of one variable on the other from the observational dataset, we have to eliminate the correlation introduced by the confounders. For example, when analyzing the causal relationship between the number of heat strokes and the rate of ice cream consumption, we may find that there is a correlation. However, the temperature is a confounder in the situation. If we eliminate this spurious correlation by conditioning on the temperature, we may find out that under the same temperature, heat stroke is not correlated to the rate of ice cream consumption.

Evaluation of interventional distribution. We specifically consider the case where there are variables Z 1 and Z 2 , and we analyze how the existence of parental nodes affects the estimation of P ( Z 2 | do ( Z 1 = z 1 )) on the observational distribution.

Proposition 2.1. Let Z 1 and Z 2 be two random variables, C ∗ be the ground truth confounder set. If C is a superset of or is equivalent to C ∗ , i.e., C ∗ ⊆ C , with c being a realization of C , we have

if no C ∈ C is a descendent of Z .

Intuitively speaking, Proposition 2.1 states that to accurately estimate the causal relationship between variables, we have to eliminate the spurious correlation by conditioning on confounders. In addition, conditioning on additional variables will not affect the estimation if they are not decedents of Z .

Causal disentanglement or statistical independence. If we had access to a series of ground-truth features that generated a fruit dataset(e.g., color, shape, texture), we can encode each of the features into a variable and concatenate them for a disentangled representation. However, these features in the observational dataset are usually correlated, depending on the type of fruits. If not considered from the causal perspective, traditional methods aiming for statistically independent representations, such as β -VAE [9], may not be able to consider correlated features that are disentangled. In fact, neither causal disentangled entails statistical independence nor vice versa because of the existence of confounders. We, therefore, formulate the task of inferring causally disentangled features from the observational under a certain level of confoundedness.

3 Confounded causal disentanglement via inductive bias

3.1 Problem formulation

We formally frame the task of discovering a set of generative factors from the observational dataset from a causal perspective as shown in Figure 2. Let X denote the observational data, the confounded causal generative process [23, 21] assumes that X are generated from K ground-truth causes of variations G = [ G 1 , G 2 , ..., G K ] (i.e., G → X ) that do not cause each other. These generative factors are generally not available, and they are confounded ∗

by some unobserved confounding variables C . The task of interest is to discover those generative factors in the latent space (denoted as Z = [ Z 1 , Z 2 , ..., Z D ] ) that best approximates G from X .

Causal disentanglement among latent generative factors. Generative factors, encoded in the latent representational space, are causally disentangled, meaning that intervening on the value of one factor does not affect the distribution of the others. It is formally defined as:

Definition 3.1 (Causal Disentanglement on Data X ([19, 23, 24])) . A representation is disentangled if, for i ∈ { 1 , ..., D } ,

where -i = { 1 , 2 , ..., D } /i indicates the set of all indices except for i .

Challenge of unobserved C ∗ . The generative factors G are not identifiable without proper inductive bias of C ∗ [16]. Previous works on discovering the generative factors either obtain disentanglement by enforcing statistical independence on latent variables [9, 15, 6], or require that latent variables do not capture information of each other [8]. Such a setting is equivalent to assuming unconfoundedness of the generative factors. It ignores the possibility that correlated latent variables can also be causally disentangled in the observational distribution, and hence is an assumption too restrictive. Fortunately, even though the ground truth generative factors are unobserved, domain expertise may inform a 'reasonable' or 'likely' inductive bias of the confounder from an accessible label set, denoted as C . This C is used to account for all correlation among Z .

Note here that we do not assume the accessible label set C equals to C ∗ (but hope that it is close to C ∗ ). The relationship between C ∗ and label set C must fall into one of the following scenarios, despite the immeasurability of their exact relationships.

Case 1 The label set contains no information about the confounders, i.e., C = ∅

Case 2 The label set contains partial information about the confounders, i.e., C ⊂ C ∗ ,

Case 3 The label set contains all information about the confounders, i.e., C = C ∗

One may argue that C may contain information irrelevant to the ground truth confounders. We show in Appendix B that in the confounded generative process described in this paper, irrelevant information in C does not affect the evaluation of the interventional distribution, and therefore can be ignored. We only take into consideration how much information in C ∗ is captured here without loss of generality.

According to Proposition 2.1, in case 3, we can estimate P ( Z i | do ( Z -i = z -i , X ) on the observational set with inductive bias from C . However, in the rest of the cases, the equation does not hold. Therefore, we introduce an operator do c ( · ) to estimate the interventional distribution on the observational set under inductive C . The framework that applies do c ( · ) is named C-Disentanglement, shorten for Confounded Disentanglement .

3.2 C-Disentanglement as the learning objective

We formally introduce C-Disentanglement and do c as follows:

∀ i ∈ 1 , 2 , 3 , ..., D , where c is realizations of C . Z obtains C-Disentanglement on X given C if

C as an approximation of C ∗ . Whether the ground truth generative factors can be discovered depends on whether their statistical relationship on the observational distribution is correctly characterized, and it requires knowledge of C ∗ . However, due to the unobservable nature of the confounders, we use C as an approximation of C ∗ , as in Equation (4). Then the identifiability of the ground truth generative factors depends on the relationship between C and C ∗ . We thus examine three possibilities as listed in section 3.1 together with a case study as in Figure 3.

(a)

C

=

∅

(b)

C

⊂

C

∗

(c)

C

=

C

∗

(d)

C

⊃

C

∗

brand A

local apple

brand B

C= local apples

imported apple

C=imported apples

imported apple

C = apple

C=banana

imported banana

local banana

C = imported banana

C=Iocal banana

C: fruit type

C: fruit type and place of origin

C: fruit type and place of origin and brand

cannot discover ground truth

C

C

C

Z

Z

Z

Z

Z3

Color

Shape

Size

Color

Shape

Size

X

X

X

In Case 1 , C is an empty set as shown in fig. 3a. It is equivalent to assuming that the generative process is unconfounded and Equation (3) degrades to statistical independence on the observational set:

This contradicts the fact that these factors are actually correlated and such an assumption makes the ground truth generative factors unidentifiable when there exists a underlying confounder.

In Case 3 , as shown in fig. 3c and fig. 3d, C ⊇ C ∗ , do is equivalent to do c . Thus the ground truth generative factors can be identified.

In Case 2 , partial information about the confounders is given as shown in fig. 3b. Heuristically speaking, this relaxes the constraint of statistical independence on the observational set and therefore enlarges the solution set, providing an increased possibility for the ground truth factors to be recovered. We empirically show in Section 5 that even partial information of confounders improves accuracy, outperforming existing methods in various tasks, even under distribution shifts.

4 cdVAE: identify causally disentangled factors in the latent space.

In this section, we provide an algorithm, cdV AE, for confounded disentangled VAE, to identify the latent causally disentangled generative factors under confounder C in the context of V AE.

(m)~Z

Z=πZ+...+πDZD

DECODER

X'

X

ENCODER

ZD ~ N(μD,ED)

(Z)d(x|Z)d) +  +∞ =

4.1 Learning objective

Given X as the dataset, we hope to find a deterministic function f , parameterized by θ , where f : Z → X such that (1) P ( X ) = ∫ f ( Z ; θ ) P ( Z ) dz is maximized and (2) each Z i encodes causally disentangled generative factors, with Z = [ Z 1 , Z 2 , ..., Z D ] as a concatenation of random variables in the latent space.

More concretely, f ( z ; θ ) is characterized by a probability distribution P ( Z | X ; θ ) , and p ( Z ) is a prior distribution from where Z can be easily sampled. For each Z i ∈ Z , we require

Applying the variational Bayesian method [12], the learning object is to optimize the evidence lower bound (ELBO) while satisfying the constraints on causal disentanglement:

For simplicity, we omit all model parameters θ, ϕ in writing.

4.2 Learning strategy

We start from the estimation of the causal disentanglement constraint as shown in Equation (6). Because the probability distribution is hard to directly calculate, inspired by [23], we resort to its first-order moment as an approximation. Specifically, we estimate the L 1 distance between the expectation of probability P ( Z i | do c ( Z -i ) , X ) and P ( Z i | X ) as follows:

We show in the following theorem that eq. (9) is satisfied the learned latent variable Z subjects to a Gaussian distribution with a diagonal covariance matrix. Proof can be found in appendix B.2.

Theorem 4.1. Suppose that the latent variable Z on dataset X given C = c subjects to the Gaussian Distribution N ( µ c ( X ) , Σ c ( X )) . Specifically,

where Z ∈ R D . If Σ c ( X ) is diagonal for all c , we have

From Theorem 4.1, we see that for each C = c , enforcing the latent variable Z to be statistically independent minimizes l c . Taking the whole C set into consideration, P ( Z | X ) subjects to a mixture

of Gaussian distribution where each centroid is inferred from observational data under a specific realization of the confounder C :

The mixing coefficient π c = P ( C = c | X ) reads the probability of occurrence of C = c . Nevertheless, such a hard assignment of coefficient varies with observable dataset and cannot accommodate scenarios in which the label set C does not exist. In this paper, we parameterize π c as a Gaussian distribution for a soft assignment of samples: π c ∼ N ( µ c ( X ) , Σ c ( X )) . Parameters µ c ( X ) and Σ c ( X ) are learned to minimize the discrepancy with P ( C = c | X ) .

In VAEs, the prior of the latent space P ( Z ) is assumed to follow a Gaussian distribution with mean zero and identity variance: Z ∼ N ( 0 , I ) . To avoid enforcing statistical independence of the overall latent spaces we learn, we only assume that the prior of Z to be a Gaussian distribution with variance one for each subset of C .

Concretely, suppose that the latent variable Z for X under C = c follows a Gaussian distribution N ( µ c ( X ) , Σ c ( X )) , then the KL divergence in (7) regulates the distribution to N ( µ c ( X ) , I ) .

By the Lagrangian multiplier method, the new loss function is

The algorithm pseudocode can be found in Appendix C.

5 Experiments

In this section, we experimentally compare cdV AE with various baselines on synthetic and real-world datasets, and study properties of cdV AE through the ablation studies. We demonstrate that cdVAE allows for causally disentangled but statistically correlated features being discovered in the latent space, which better approximates the ground truth generative factors and outperforms SOTA baseline under distribution shifts.

Concretely, we aim to answer the following questions regarding the proposed model:

▷ Q1 : How does it perform compared to the existing methods in the latent space?

▷ Q2 : How does it perform in downstream tasks such as classification under distribution shifts?

5.1 Basic setup

Datasets. we evaluate cdVAE on three datasets: synthetic datasets 3dshape [3] and Candle [21], and real-world dataset CelebA [15]. 3dshape is a dataset of 3D shapes generated from 6 ground-truth independent latent factors. These factors are floor color, wall color, object color, scale, shape, and orientation. There are 480,000 images from all possible combinations of these factors, and these combinations are present exactly once. Candle is a dataset generated using Blender, a free and opensource 3D CG suite that allows for manipulating the background and adding foreground elements that inherit the natural light of the background. It has floor hue, wall hue, object hue, scale, shape, and orientation as latent factors.

We use 3dshape and Candle as synthetic datasets. To mimic the real-world scenario where perfectly disentangled ground-truth generative factors do not exist or are hard to identify, we manually create a certain level of correlation in these datasets by first making rules among certain attributes and then sampling accordingly.

Baselines and tasks . We compare the performance of cdVAE in the task of image generation and classification under distribution shift with the following architectures: (1) V AE-based methods (vanilla VAE [12], β -VAE [9], FactorVAE [11]) that equate disentanglement as statistical independence, (2) Existing causal regulation methods, CAUSAL-REP [24], (3) cVAE [12] as it also applies the label information (4) GMVAE [7] as it also adopts a mixture of Gaussian model in a variational autoencoder framework.

Note that in this paper, we do not compare cdVAE with CausalVAE [27], despite the latter also obtaining 'disentanglement' in the latent space. The main reason is that the problem-setting and the learning objectives are fundamentally different. CausaslV AE aims to disentangle known ground truth generative factors in the latent space, it aims to bind these factors one by one to latent variables and the causal dependency is not considered. The task of interest in this paper is to discover causally disentangled factors in the latent space and the ground truth is unavailable, thus these two methods are not comparable.

Evaluation metrics . The experimental results are evaluated on (1) end-to-end evaluation metrics: the accuracy of classification under distribution shifts and reconstruction loss in image generation task. (2) how well the learned latent generative factors recover the ground truth one: Maximal Information Coefficient (MIC) and Total Information Coefficient (TIC) [13]. (3) Existing disentanglement scores from a causal perspective: IRS [23], UC/CG [21], IOSS [24], and statistical perspective: D-Score [8]. The higher these evaluation metrics, the better the model except for IOSS (the lower, the better). A detailed introduction of these metrics can be found in Appendix C.

To be more concrete, IRS evaluates the general interventional robustness of learned representations while UC and CG are extended from IRS. UC (Unconfoundedness) evaluates how well distinct generative factors are captured by distinct sets of latent dimensions with no overlap. CG (Counterfactual Generativeness) measures how robust a certain latent variable is against the others. IOSS measures whether learned random variables have independent supports as a necessary condition inferred from the definition of causal disentanglement. D score measures disentanglement in the non-causal definition.

5.2 Experimental results

(Q1) cdVAE significantly outperforms various baselines in end-to-end measurement.

We compare our cdVAE with baseline models in the image generation task on the CelebA, and Candle datasets. In CelebA dataset, we have C set to be whether there are eyeglasses in the image, i.e., | C | = 2 . In the Candle dataset, we have | C | = 4 , with 2 objective hues and two wall hues combined together. More details can be found in Appendix C.

As shown in Table 1, cdVAE are evaluated by several groups of metrics. Reconstruction error measures how well the images are reconstructed in an end-to-end fashion while the others are measurements of how disentangled the latent factors are: D score is from the non-causal perspective while UC, UG and IOSS are from the causal perspective. For the celebA dataset, we do not measure the UC and CG score as it requires ground truth generative factors or access to the true generative process. Our method outperforms all baselines on these metrics except for the D score. Note the D score is only an effective measurement when there are no confounders in the observational dataset, as it requires each latent variable not to contain information about the others. Under the confounded dataset such as CelebA and Candle, the ground truth generative factors are correlated, resulting in a low D score with our model as expected.

(Q2.1) cdVAE are more robust under distribution shifts.

We conduct the task of shape classification on the 3dshape dataset with distribution shift and use the classification accuracy as a metric for out-of-distribution generalization[28]. Specifically, in the source distribution, we sample a certain percentage of images in which the object hue is correlated with the object shape (i.e., red objects are cubes). The rest of the images are evenly generated by

disentangled factors, while in the target domain, all images are generated by disentangled factors. The proportion of highly correlated data is denoted by shift severity . For example, shift severity = 0.4 means that 40% of training images are sampled under preset correlation between object hue and object shape.

We train cdVAE, β -VAE, and CAUSAL-REP using images from the source set, with decoders being replaced by classifiers. The trained classifier is then tested on the targets set. We report the classification accuracy on the target set and the performance drop in Figure 5. We could observe that cdVAE is more robust than other baselines under distribution shift as it has the lowest performance drop and the highest target set accuracy.

An intuitive explanation of how C-Disentanglement improves the OOD generalization. Suppose we have a fruit dataset and the goal is to classify fruits. In the training set, a large amount of apples are red and round. If assuming unconfoundedness and requires statistical independence, it is possible that a latent variable encodes whether this object is round-and-red. As a result, in the test set, if there is a green apple, it will fail the test and may not be able to be identified as an apple. However, under the framework of C-Disentanglement, color and shape are disentangled in the latent space, and the shape and color contribute to the classification separately, resulting in a higher probability for the green apple also to be identified as an apple.

80.782.1 84.5

80.7 81.984.4

80.3 81.884.4

Accuracy(%)

12.7 12 10.1

12.9 12.4 10.2

13.112.510.1

T

S-T

T

S-T

T

S-T

shift severity=0.4

shift severity=0.5

shift severity=0.6

betaVAE

CAUSAL-REP

cdVAE

(Q2.2) cdVAE better discovers the ground truth generative factors.

In the task of shape classification, we further examine how well the learned representations approximate the ground truth generative labels and to what extent they are causally disentangled. Table 2 shows that cdVAE outperforms all baselines in approximating the generative factors and the disentanglement.

5.3 Ablations studies

We further conduct ablation studies to show that providing inductive bias indeed improves the discovery of generative factors in the latent space. Concretely, we compare cdV AE with conditional VAE (cVAE) [12] and GMVAE [7]. Compared with vanilla VAE, the method proposed uses label information to provide inductive bias to confounders for partitioning the observational dataset. cV AE has the label information compared with vanilla VAE and GMVAE is a VAE modelled by a mixture of Gaussian. As shown in Table 1, cdVAE has universally better results, showing the necessity of introducing bias to factors discovered in the latent space. We also investigate how choice of C affect the model performance and how to adapt C-Disentanglement to the existing method aiming for statistical independent, as shown in appendix C.

6 Related work

Disentangled Representations The pursuit for disentangled representation can be dated to the surge of representation learning and is always closely associated with the generative process in modern machine learning, following the intuition that each dimension should encode different features.

[6] attempts to control the underlying factors by maximizing the mutual information between the images and the latent representations. [8] propose a quantitative metric with the information theory. They evaluate the disentanglement, completeness, and informativeness by fitting linear models and measuring the deviation from the ideal mapping. [9, 11, 5, 18] encourage statistical independence by penalizing the Kullback-Leibler divergence (KL) term in the V AE objective. However, the non-causal definitions of disentanglement fail to consider the cases where correlated features in the observational dataset can be disentangled in the generative process. Such a challenge is well-approached through a line of research from the causal perspective.

Causal Generative Process. Causal methods are widely used for eliminating spurious features in various domains and improving understandable modelling behaviours[25, 26, 14]. It is not until [23] that it was introduced for a strict characterization of the generative process. [23] first provided a rigorous definition of a causal generative process and the definition of disentangled causal representation as the non-existence of causal relationships between two variables, i.e., the intervention on one variable does not alter the distribution of the others. The authors further introduce interventional robustness as an evaluation metric and show its advantage on multiple benchmarks. [21] follow the path of [23] and further propose two evaluation metrics and the Candle dataset. The confounded assumption allows for correlation in the latent space without tempering with the disentanglement in the data generative. Despite effective evaluation tools, there is still a missing piece on how to infer a set of causally disentangled features. Using the proposed evaluation metric as regulation, the model implicitly assumes unconfoundedness and it falls back to finding statistical independence in the latent space. The problem of unrealistic unconfoundedness assumption is identified by [24]. They assume that confounders exist but they are unobservable. They further propose an evaluation metric considering the existence of confounders, that causally disentangled latent variables have independent support measured by the IOSS score. Similar to the evaluation metrics introduced in [23, 21], IOSS is also a necessary condition of the causal disentanglement. More importantly, as in previous work focusing on obtaining statistical independence, such a regulation suffers from the identifiability issue.

Weak Supervision for Inductive Bias. The identifiability issue in unsupervised disentangled representation learning is first identified in [16]. Specifically, they show from the theory that such a learning task is impossible without inductive biases on both the models and the data. Naturally, a series of weak-supervised or semi-supervised methods [4, 1, 2] are proposed with a learning objective of statistical independence or alignment. In this paper, we take a step further for the confounding assumption, assuming that the confounders are observable with proper inductive bias so that the latent representation can be better identified. We, similarly, adopt partial labels of the dataset as the supervision signal. We treat the labels as a source of possible confounders and allow the learning of correlated but causally disentangled latent generative factors to be learned.

7 Conclusion and discussions

In this paper, we recognize the importance of bringing in confounders with proper inductive bias into the discovery of causally disentangled latent factors. Specifically, we propose a framework, C-Disentanglement, that introduces the inductive bias from the domain knowledge and a method, cdVAE, to identify the generative factors. Although cdV AE helps with discovering generative factors under confounders, the learning process relies on the domain knowledge, the choice of C , and that C has to be discrete/categorical and have a finite number of realizations. How to generalize to continuous C or select effective C from the available knowledge set is beyond the scope of this paper and will be investigated in future studies. We hope this work could bring attention to the importance of inductive bias of confounders and inspire future works in this direction. In addition, better capturing of generative factors could bring a more controlled generative process and positive social impact.

References

K. Ahuja, J. S. Hartford, and Y. Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems , 35:15516-15528, 2022.

J. Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. arXiv preprint arXiv:2203.16437 , 2022.

C. Burgess and H. Kim. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018.

J. Chen and K. Batmanghelich. Weakly supervised disentanglement by pairwise similarities. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3495-3502, 2020.

R. T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems , 31, 2018.

X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems , 29, 2016.

N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648 , 2016.

C. Eastwood and C. K. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations , 2018.

I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.

D. Horan, E. Richardson, and Y. Weiss. When is unsupervised disentanglement possible? Advances in Neural Information Processing Systems , 34:5150-5161, 2021.

H. Kim and A. Mnih. Disentangling by factorising. In International Conference on Machine Learning , pages 2649-2658. PMLR, 2018.

D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

J. B. Kinney and G. S. Atwal. Equitability, mutual information, and the maximal information coefficient. Proceedings of the National Academy of Sciences , 111(9):3354-3359, 2014.

X. Liu, H. Lu, J. Yuan, and X. Li. Cat: Causal audio transformer for audio classification. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning , pages 4114-4124. PMLR, 2019.

F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen. Weaklysupervised disentanglement without compromises. In International Conference on Machine Learning , pages 6348-6359. PMLR, 2020.

E. Mathieu, T. Rainforth, N. Siddharth, and Y. W. Teh. Disentangling disentanglement in variational autoencoders. In International Conference on Machine Learning , pages 4402-4412. PMLR, 2019.

J. Pearl. Causal inference in statistics: An overview. Statistics surveys , 3:96-146, 2009.

J. Pearl. The do-calculus revisited. arXiv preprint arXiv:1210.4852 , 2012.

A. G. Reddy, V. N. Balasubramanian, et al. On causally disentangled representations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8089-8097, 2022.

J. Soch, T. B. of Statistical Proofs, T. J. Faulkenberry, K. Petrykowski, and C. Allefeld. StatProofBook/StatProofBook.github.io: StatProofBook 2020, Dec. 2020.

R. Suter, D. Miladinovic, B. Schölkopf, and S. Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning , pages 6056-6065. PMLR, 2019.

Y. Wang and M. I. Jordan. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv:2109.03795 , 2021.

Y. Wang, D. Liang, L. Charlin, and D. M. Blei. Causal inference for recommender systems. In Proceedings of the 14th ACM Conference on Recommender Systems , pages 426-431, 2020.

C. Xu, C. Liu, X. Sun, S. Yang, Y. Wang, C. Wang, and Y. Fu. Patchmix augmentation to identify causal features in few-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9593-9602, 2021.

S. Zhu, B. An, and F. Huang. Understanding the generalization benefit of model invariance from a data perspective. Advances in Neural Information Processing Systems , 34:4328-4341, 2021.

A Introduction of do calculus.

Do-calculus consists of three rules that help with identifying causal effects.

Rule A.1 (Insertion/deletion of observations) .

Rule A.2 (Action/observation exchange) .

Rule A.3 (Insertion/deletion of actions) .

where G X is the graph with all incoming edges to X being removed, G W is the graph with all outcoming edges to W being removed, and Z ( W ) is the set of Z -nodes that are not ancestors of any W -node.

Intuitively, Rule A.1 states when an observant can be omitted in estimating the interventional distribution, Rule A.2 illustrates under what condition, the interventional distribution can be estimated using the observational dataset, and Rule A.3 decides when we can ignore an intervention.

B Proofs

B.1 Proof of Proposition 2.1

Proposition B.1. Let Z 1 and Z 2 be two random variables, C ∗ be the ground truth confounder set. If C is a superset of or is equivalent to C ∗ , i.e., C ∗ ⊆ C , with c being a realization of C , we have

if no C ∈ C is a descendent of Z .

Proof.

B.2 Proof of Theorem 4.1

Theorem B.2. Suppose that the latent variable Z on dataset X given C = c is Gaussian N ( µ c ( X ) , Σ c ( X )) . Specifically,

where Z ∈ R D . If Σ c ( X ) is diagonal for all c , we have

Proof. We suppose that

where we omit X for simplicity and D is the dimension of Z for any given c . By definition of l c (Equation (9)) and proposition 2.1,

where we denote Z c = [ Z | X , C = c ] for simplicity. Notice that Z c ∼ N ( µ c , Σ c ) ∈ R D , we therefore know that the conditional distribution of any subset vector Z c k , given the complement vector Z c j , is also a multivariate Gaussian distribution [22]

where

given that Σ c j,j is nonsingular.

Hence we know that the first expectation in Equation (21) becomes

assuming that Σ c -i, -i is nonsingular. Since E [ Z c i ] = µ c i , the loss l c can be written as

We assume further that Σ c is a diagonal matrix. Therefore Σ c -i, -i = 0 is a zero row vector. Then

C Experimental Details

C.1 Experimental Details

The experiments are conducted on 4 NVIDIA GeForce RTX 2080Ti. In each experiment, we repeat 5 times with different seeds and report the averaged results. In all experiments, only partial information on the ground truth confounder is provided. Specifically, for example, the 3dshape dataset, we first make some predefined rules, such as ' 70% cubes are red'. Then we generate 700 red cubes and 300 cubes in other colors. The generation process naturally divides the dataset into different subgroups, and we can thus explicitly control how inductive bias is provided, i.e., the grouping. In the celebA dataset, since we do not have access to the ground truth generative factors, so we assume any label sets only contain partial information.

C.2 Ablation study

Weinvestigate how the choice of C affect the model performance and how to adapt C-Disentanglement to the existing method aiming for statistical independence, as shown in appendix C.

Choice of

Acc - T

C

=

∅

C

=

C

partial

Table 4: Adapt cdVAE to existing methods. The IOSS and Reconstruction loss are measured based on image generation task and the performance drop is measured on shape classification on the target domain under shift severity=0.5.

∗

C

∗

79.2

88.2

84.5

IRS

↑

0.82

0.89

0.87

C-Disentanglement improves the learning of ground truth generative factors under a reasonable choice of label set. To understand how the choice of C affects the performance, we repeat the shape classification task with different choices of C under 50% shift severity. When with C , we assume the generative process is unconfounded, and cdVAE degrades to the vanilla VAE model. With partial C , we partition the data according to only 2 values of the shifting variables instead of 4. With full C , we provide the full confounders. As shown in Table 3, even partial information of the confounders improves the model performance in OOD generalization and obtains more robust latent representations.

Adapting C-Disentanglement to existing works further improve their performance. We compare the performance between regulation through IOSS[24] and cdVAE + IOSS in image generation and classification tasks on 3dshape dataset. In the cdV AE + IOSS, we apply additional regularization terms based on the Z . The results show that C-Disentanglement framework could further improve the performance with desired level of inductive bias given.

C.3 Pseudo-code

C.4 Additional Experimental Results

The classification accuracy on both the source and the target distribution with variance is given in the table below.

C

↑

Algorithm 1 Train a VAE such that the latent representation is causally disentangled

Input: Number of labels N C , training data X with labels c , ratio of each categories/confounders P ( C = c ) in the training set, dimension of latent space D

1: for x ∈ X do

2: for c ∈ C do

3: Define Z c = [ Z | x, C = c ] , and obtain from encoder Z c ∼ N ( µ c ( x ) , Σ c ( x )) for each c , assuming Σ c ( x ) to be diagonal matrix:

4: Sample from Z c ∼ N ( µ c ( x ) , Σ c ( x )) :

5: Parametrize π c ∼ N ( µ π c ( x ) , σ π c ( x )) ∈ R with neural network.

6: Regulate the covariance matrix to be identity matrix with KL divergence

7: end for

8: Normalize Π C = ( π c 1 , ..., π c NC ) such that ∥ Π C ∥ 2 = 1 .

9: Compute classification loss between Π C with label c :

10: Let Z ( x ) = ∑ c ∈ C π c Z c ( x ) , and obtain the reconstructed sample from decoder: x ′ = Θ dec ( Z ( x )) . Compute reconstruction loss for Z ( x ) :

and update encoders and decoders.

12: end for

11: Compute total loss

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed Confounded-Disentanglement a method that makes use of inductive bias of a confounder by leveraging labels/knowledge from domain expertise. The aim is to recover the true/causal generative factors for the observed data which can recover a more meaningful latent representation that can be further useful for downstream tasks especially in the case of distribution shifts.\n\nSTRENGTHS:\n- Originality: the authors propose a new way of leveraging observed confounders to learn disentangled representations.\n- Quality: the manuscript includes both theoretical and empirical evidence for the claims.\n- Clarity: I appreciate the schematics and toy running example to illustrate the intuition of the method.\n- Significance: Better latent representations useful for overcoming domain generalisation challenges is timely and important contribution.\n\nWEAKNESSES:\nThe problem of disentanglement, learning meaningful representations in general is an important , unresolved problem and I do agree that when additional information is available, we should make use of it, such as the scenario proposed by the authors of cdVAE. However, I do find the paper lacking in some recent relevant work and the overall presentation would benefit from including/positioning this method with regards to it. Please see more details in the question section.\n\nQUESTIONS:\n1. A recent stream of disentanglment that also leverages additional/auxiliary information, similar to ICA but even closer to cdVAE are the following [1, 2, 3]. In those cases, the additional information is treated as intervention, or environment. Could you please position your work wrt that? What makes a confounder different than their auxiliary variable? \n2. What are the advantages of interpreting the additional variable as confounder and not intervention?\n3. There are other metrics for measuring the quality of disentangled representations that haven't been used R2, MCC for example\n4. Have you tried the controlled generative process under your framework? What could you conclude from the results? \n\n[1] Lachapelle, Sébastien, et al. \"Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA.\" Conference on Causal Learning and Reasoning. PMLR, 2022.\n\n[2] Khemakhem, Ilyes, et al. \"Variational autoencoders and nonlinear ica: A unifying framework.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.\n\n[3] Lu, Chaochao, et al. \"Invariant causal representation learning for out-of-distribution generalization.\" International Conference on Learning Representations. 2021.\n\n\n---- Update after Author response ----\nI thank the authors for the detailed and timely responses, they have mostly addressed my concerns, so I increase my score to BA.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is motivated by the assumption that a few semantically significant generative factors generate real-world data. Then they study causal independence in generative methods by introducing a new framework, C-Disentanglement, that discusses the identifiability issue of generative factors regarding the inductive bias of confounder to handle the disadvantage brought by the statistical independence assumption of previous methods. At last, the authors implement their method, cdVAE, and evaluate it on three datasets. The results show that cdVAE performs better in the latent space compared to the existing methods and performs better in downstream tasks.\n\nSTRENGTHS:\nThe motivation is obvious, which emphasizes the importance of causal independence in generative methods and illustrates the advantages of causal independence for statistical independence, which is very interesting and important. To the best of our knowledge, the authors are the first to study how to explore causal independence confounders in generative methods. \nThis article has complete content. It conducts a thorough investigation of the background. Then it proposes a new framework and elaborates in detail. Moreover, it has several experiments on three datasets to demonstrate its advantages.\n\nWEAKNESSES:\nThis article includes but is not limited to the following shortcomings:\n1. There was much ambiguity about collaboration, which led me to rely a lot on my guesswork in understanding the article, like:\n\na) D in line 141 and D in equation 7 do not seem like the same thing but use the same symbol.\n\nb) D_KL in Figure 4 and D in Equation 7 seem to represent KL divergence but in different symbols.\n\nc) What is the meaning of \\Pi? I do not find its definition. Is that represent the soft assignment of samples? If it is, what is the connection between \\Pi and \\pi?\n\n2. In Table 1, you evaluate your method on CelebA and Candle datasets. However, the measurement metrics between these datasets are different. You claim that the CelebA dataset does not have ground truth factors. However, the experiments on only two datasets cannot fully illustrate your methods' advantages, especially since CelebA does not have ground truth factors.\n3. There are many kinds of generative methods, like the diffusion model. There is no comparison between your method and other generative methods.\n4. The baseline vanilla VAE (2013), \\beta-VAE (2016), FactorVAE (2018), CAUSAL-REP (2021), which do not compare, cVAE (2013), GMVAE (2016), are out-of-date.\n\nQUESTIONS:\n1. I think implementing your ideas in VAE is a limitation for you. I hope you can implement your ideas in more generative methods.\n2. The Related Work in the main paper is very similar to the Related Work in the appendix, so I doubt the significance of the Related Work in the appendix. However, I do not think this is a weakness of your paper, so I put this into Question.\n\nLIMITATIONS:\nThe authors point out the limitation of their method at the end of the paper; that is, the choice of C is difficult but essential, which is the same as our understanding of their method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the domain of representation learning, focusing on the discovery of meaningful generative factors within a latent space. The authors propose the C-Disentanglement framework, which incorporates the inductive bias of confounders through labels or knowledge obtained from domain expertise. This framework enables the identification of generative factors even in the presence of confounding variables. The paper offers a comprehensive theoretical analysis and a practical approach to effectively identify these factors. Empirical results obtained from diverse settings validate the efficacy of the proposed method.\n\nSTRENGTHS:\n\tThis paper is very well written and easy to follow.\n\tThe research addresses an important and fundamental question: how to identify Causally-Independent generative factors in the presence of common causes.\n\tThe idea proposed in this work is simple but certainly make sense. And the sufficient experiments validate the effectiveness of the proposed method.\n\nWEAKNESSES:\n\tThe assumptions made in this paper are quite strict. One of the main concerns is that the paper assumes the probability distributions of the generative factors to be Gaussian distributions. This assumption may limit the applicability and generalizability of the proposed method to real-world scenarios where the underlying distributions may not strictly follow Gaussian patterns. Another strong assumption in this paper is the linear mixing of the generative factors. While this assumption might simplify the modeling process, it may not accurately capture the complex relationships and interactions between the generative factors in real-world scenarios. It is important for the authors to discuss the potential impact of this assumption on the performance and generalizability of the proposed method.\n\tThe proposed inductive bias in this paper is hard to obtain. The utilization of domain expertise knowledge as an inductive bias is an expensive and non-scalable approach.\n\tThe evaluations of the proposed method primarily focus on quantitative aspects, and incorporating qualitative evaluations would provide a more intuitive understanding of the method's performance.\n\nQUESTIONS:\n\tHow can we ensure that the number of C is adequate?\n\tIn Line 215, Equation (10), the right-hand side of the equation involves several terms that are not enclosed within absolute value symbols. Does these terms are non-negative?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework called Confounded-Disentanglement (C-Disentanglement) to learn disentangled representations by using inductive bias of confounding. Instead of the common statistically independent assumption for latent distribution which is usually not true in real-world datasets, the proposed framework partitions the observed data distribution based on a label set C and regulates the latent distribution separately for these subsets. \n\nThe proposed framework is then implemented in a Variational AutoEncoder (VAE) setting, termed cdVAE. Practically, the proposed method models the latent distribution as a mixture of Gaussian model in which each individual Gaussian corresponds to a specific value of C. \n\nThe paper shows the effectiveness of the proposed method on the image generation task on CelebA and Candle datasets, and on a classification task under distribution shift on 3DShapes dataset.\n\nSTRENGTHS:\n- The paper extends research in disentangled representation learning by leveraging inductive bias of confounding that makes it more applicable to real-world datasets that contain correlations in generative factors.\n- The proposed framework and method are technically sound.\n- The results seem promising. \n- The proposed method is evaluated on multiple tasks and multiple datasets which shows the robustness of the proposed method.\n\nWEAKNESSES:\n- The proposed method requires domain expertise knowledge to define the confounders. This can be easy in some cases but could be costly (in terms of time and cost) and error-prone in many cases. What would happen if an error is introduced in defining the label set? Also, how to distinguish from the cases where each subset of data (defined by a specific value of the label set) still contains correlations among generative factors e.g., in the example of Figure 3, if the imported apples contain large green apples and small yellow apples? Is it a requirement that the label set needs to be granular enough so that no correlations remain in the subset corresponding to each specific value of the label set? \n\n- What cases from section 3 do the experiments from the paper fall in? Please discuss this in detail.\n\n- The proposed method cdVAE effectively boils down to modeling the latent distribution as a mixture of Gaussians. How is it different from GMVAE [7] then?\n\n- Could the authors explain in more detail how the objective in CausalVAE [24] is fundamentally different from this paper?\n\n- The paper mentions that using a Mutual Information (MI) objective for latents is not sufficient/appropriate. Then, shouldn’t the evaluation metrics based on MI be avoided as well? Also, why not use DCI/D score in Table 2?\n\n- Can the authors explain in a bit of detail how all the evaluation metrics for disentanglement are being calculated and why UC and CG scores can't be calculated for CelebA? Can't the attribute annotations in CelebA be used as a proxy for generative factors?\n\n- The shape classification generalization experiment setting is a bit unclear. Can the authors please explain it again, maybe with an example?\n\n- All the results need error bars.\n\nQUESTIONS:\nI have included all my questions that I would like the authors to answer in the weaknesses section above. The answers to all these issues are crucial to my final recommendation.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}