SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
l _ {c} = \sum_ {i} ^ {D} d \left(\mu_ {i} ^ {c} + \Sigma_ {i, - i} ^ {c} \left(\Sigma_ {- i, - i} ^ {c}\right) ^ {- 1} \left(Z _ {- i} ^ {c} - \mu_ {- i} ^ {c}\right), \mu_ {i} ^ {c}\right). \tag {25}
$$
We assume further that  $\Sigma^c$  is a diagonal matrix. Therefore  $\Sigma_{-i, - i}^{c} = 0$  is a zero row vector. Then
$$
l _ {c} = \sum_ {i} ^ {D} d \left(\mu_ {i} ^ {c}, \mu_ {i} ^ {c}\right) = 0 \tag {26}
$$
C.1 Experimental Details
The experiments are conducted on 4 NVIDIA GeForce RTX 2080Ti. In each experiment, we repeat 5 times with different seeds and report the averaged results. In all experiments, only partial information on the ground truth confounder is provided. Specifically, for example, the 3d shape dataset, we first make some predefined rules, such as “ $70\%$  cubes are red”. Then we generate 700 red cubes and 300 cubes in other colors. The generation process naturally divides the dataset into different subgroups, and we can thus explicitly control how inductive bias is provided, i.e., the grouping. In the celebA dataset, since we do not have access to the ground truth generative factors, so we assume any label sets only contain partial information.
Table 5: Compare cdVAE with  $\beta$ -Vae, CAUSAL-REP on classification under distribution shift. T represents accuracy on the target data, S represents the performance on the target domain when the classifier trained on the source data is directly tested on the target data.
Methods	shift = 0.4	shift = 0.5	severity = 0.6
Acc-S	Acc-T	Acc-S	Acc-T	Acc-S	Acc-T
CAUSAL-REP	94.1±0.04	82.1±0.08	94.3±0.03	81.9±0.07	94.3±0.02	81.8±0.11
β-VAE	93.4±0.07	80.7±0.12	93.6±0.05	80.7±0.03	93.4±0.04	80.3±0.09
cdVAE	94.6±0.02	84.5±0.05	94.6±0.04	84.4±0.05	94.5±0.03	84.4±0.04
C.2 Ablation study
We investigate how the choice of  $\mathbf{C}$  affects the model performance and how to adapt C-Disentanglement to the existing method aiming for statistical independence, as shown in appendix C.
Table 3: Performance under different C on shape classification on 3dshape dataset, shift sever- Table 4: Adapt cdVAE to existing methods. ity=0.5. tion drop is measured on shape classification on the target domain under shift severity  $= {0.5}$  .
Choice of C	Acc - T ↑	IRS ↑
C = ∅	79.2	0.82
C = C*	88.2	0.89
partial C*	84.5	0.87
Methods	IOSS ↓	Recon ↓	Acc-T ↑
IOSS	0.14	0.12	81.8
cdVAE + IOSS	0.12	0.08	84.5
C-Disentanglement improves the learning of ground truth generative factors under a reasonable choice of label set. To understand how the choice of C affects the performance, we repeat the shape classification task with different choices of C under  $50\%$  shift severity. When with C, we assume the generative process is unconfounded, and cdVAE degrades to the vanilla VAE model. With partial C, we partition the data according to only 2 values of the shifting variables instead of 4. With full C, we provide the full confounders. As shown in Table 3, even partial information of the confounders improves the model performance in OOD generalization and obtains more robust latent representations.
Adapting C-Disentanglement to existing works further improve their performance. We compare the performance between regulation through IOSS[24] and cdVAE + IOSS in image generation and classification tasks on 3dshape dataset. In the cdVAE + IOSS, we apply additional regularization terms based on the Z. The results show that C-Disentanglement framework could further improve the performance with desired level of inductive bias given.
C.3 Pseudo-code
C.4 Additional Experimental Results
The classification accuracy on both the source and the target distribution with variance is given in the table below.
Algorithm 1 Train a VAE such that the latent representation is causally disentangled
Input: Number of labels  $N_{C}$ , training data  $\mathbf{X}$  with labels  $c$ , ratio of each categories/confounders  $P(\mathbf{C} = c)$  in the training set, dimension of latent space  $D$
1: for  $x \in \mathbf{X}$  do
2: for  $c \in \mathbf{C}$  do
3: Define  $\mathbf{Z}^c = [\mathbf{Z}|x,C = c]$ , and obtain from encoder  $\mathbf{Z}^c\sim \mathcal{N}(\mu^c (x),\Sigma^c (x))$  for each  $c$ , assuming  $\Sigma^c (x)$  to be diagonal matrix:
$$
\Theta_ {e n c} ^ {c} (x) = \left[ \mu^ {c}, \operatorname {d i a g} \left(\Sigma^ {c}\right) \right] \in \mathbb {R} ^ {2 d}, \quad \mu^ {c} \in \mathbb {R} ^ {d}, \quad \operatorname {d i a g} \left(\Sigma^ {c}\right) \in \mathbb {R} ^ {d} \tag {27}
$$
4: Sample from  $\mathbf{Z}^c\sim \mathcal{N}(\mu^c (x),\Sigma^c (x))$
$$
\mathbf {Z} ^ {c} = \mu^ {c} + \left(\Sigma^ {c}\right) ^ {\frac {1}{2}} \epsilon^ {c}, \epsilon^ {c} \sim \mathcal {N} (\mathbf {0}, I) \tag {28}
$$
5: Parametrize  $\pi^c\sim \mathcal{N}(\mu_{\pi^c}(x),\sigma_{\pi^c}(x))\in \mathbb{R}$  with neural network.
6: Regulate the covariance matrix to be identity matrix with KL divergence
$$
D _ {K L} ^ {c} = \frac {1}{2} \left[ \log \frac {1}{\det \Sigma^ {c}} - D + \operatorname {t r} \left(\Sigma^ {c}\right) \right] \tag {29}
$$
8: Normalize  $\Pi_C = (\pi^{c_1},\dots,\pi^{c_{NC}})$  such that  $\| \Pi_C\| ^2 = 1$
9: Compute classification loss between  $\Pi_C$  with label  $c$ :
$$
\mathcal {L} _ {c l s} = H \left(\Pi_ {C}, c\right) \tag {30}
$$
10: Let  $\mathbf{Z}(x) = \sum_{c\in \mathbf{C}}\pi^{c}\mathbf{Z}^{c}(x)$ , and obtain the reconstructed sample from decoder:  $x^{\prime} = \Theta_{dec}(\mathbf{Z}(x))$ . Compute reconstruction loss for  $\mathbf{Z}(x)$ :
$$
\mathcal {L} _ {\text {r e c}} = \operatorname {m s e} \left(x ^ {\prime}, x\right) \tag {31}
$$
11: Compute total loss
$$
\mathcal {L} _ {\text {t o t a l}} (x) = \mathcal {L} _ {\text {r e c}} + L _ {\text {c l s}} + \sum_ {c \in \mathbf {C}} D _ {K L} ^ {c} \tag {32}
$$
and update encoders and decoders.
12: end for