SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we provide discussions of how to better understand PNS value, the understanding of Theorem 4.3, the connection on domain generalization, and the limitations of this work.
A.1 Examples to Understand PNS Value
Figure 3: Example for causal sufficiency and necessity in image classification problem. The images on the left are for training and the rights are for OOD tests.
We provided two examples to help understand the PNS (Probability of Necessity and Sufficiency) value. We will add the following explanations to our revision.
Example.1. We use the feature 'has cat legs' represented by the variable  $C$  (taking binary values 1 or 0) to predict the label of being a cat or a fox. 'has cat legs' is the sufficient but unnecessary cause because the image contains cat legs must have cat but cat image might not contain cat leg.
We assume  $P(Y_{do(C = 1)} = 1) = 1$  and  $P(Y_{do(C = 0)} = 0) = 0.5$ ,  $P(Y = 1) = 0.75$ ,  $P(C = 1, Y = 1) = 0.5$ ,  $P(C = 0, Y = 0) = 0.25$ ,  $P(C = 0, Y = 1) = 0.25$ .
Now, applying the concept of the probability of sufficiency and necessity, we obtain:
Probability of necessity:  $P(Y_{do(C = 0)} = 0 | Y = 1, C = 1) = \frac{P(Y = 1) - P(Y_{do(C = 0)} = 1)}{P(Y = 1, C = 1)} = \frac{0.75 - 0.5}{P(Y = 1, C = 1)} = 0.5$
Probability of sufficiency:  $P(Y_{do(\mathbf{C} = 1)} = 1 | Y = 0, C = 0) = \frac{P(Y_{do(C = 1)} = 1) - P(Y = 1)}{P(Y = 0, C = 0)} = \frac{1 - 0.75}{P(Y = 1, C = 1)} = 1$
In this example, we can state that variable  $C$  has a probability of being a sufficient cause. Note that the calculation of PN and PS are from eq.9.29 and eq.9.30 in Chapter 9 at (Pearl, 2009).
Example.2. If we use feature 'pointy ear'  $C$  (taking values 1, 0. Value 1 means pointy ear), to predict  $Y$  (cat 1, other animal 0). Since a cat must have pointy ears but if an animal has pointy ears, it should not be a cat, we assume  $P(Y_{do(C = 1)} = 1) = 0.5$  and  $P(Y_{do(C = 0)} = 0) = 1$ ,  $P(Y = 1) = 0.25$ ,  $P(C = 1, Y = 1) = 0.25$ ,  $P(C = 0, Y = 0) = 0.5$ ,  $P(C = 0, Y = 1) = 0.25$ .
Probability of necessity:  $P(Y_{do(C = 0)} = 0|Y = 1, X = 1) = 1$
Probability of sufficiency:  $P(Y_{do(C = 1)} = 1 | Y = 0, X = 0) = 0.5$
In this example, we can state that variable  $C$  has a probability of being a necessary cause.
Example.3. If we use feature 'eye size' C (taking values 1, 0.5 or 0 with the probability of  $\frac{1}{3}$ , respectively, the lower value means smaller eye size on face), to predict  $Y$  (cat 1 or fox 0). Assuming  $P(Y_{do(C = 1)} = 1) = 1$ ,  $P(Y_{do(C = 0.5)} = 1) = 0.5$  and  $P(Y_{do(C = 0)} = 1) = 0$  (cat have larger eye size than fox). In this example, we have  $P(C = 0, Y = 0) = P(C = 1, Y = 1) = \frac{1}{3}$  and  $P(C = 0.5, Y = 0) = P(C = 0.5, Y = 1) = \frac{1}{6}$ .
In Definition 2.1,  $\mathrm{PNS}(c,\bar{c})$  can take multiple choice. We provide Case.1:  $c = 1$  and  $\bar{c} = 0.5$  and Case.2:  $c = 1$  and  $\bar{c} = 0$ .
Case.1 In this case,  $\mathrm{PN} = P(Y_{do(C = 0.5)} = 0|C = 1,Y = 1) = 0$
$$
\mathrm {P S} = P \left(Y _ {d o (C = 1)} = 1 \mid Y = 0, C = 0. 5\right) = 3
$$
$$
\mathrm {P N S} (1, 0. 5) = \mathrm {P N P} (C = 1, Y = 1) + \mathrm {P S P} (Y = 0, C = 0. 5) = 0. 5
$$
Case.2 In this case,  $\mathrm{PN} = P(Y_{do(C = 0)} = 0|C = 1,Y = 1) = 1.5$
$$
\mathrm {P S} = P \left(Y _ {d o (C = 1)} = 1 \mid Y = 0, C = 0\right) = 1. 5
$$
$$
\mathrm {P N S} (1, 0) = P (C = 1, Y = 1) + \mathrm {P S P} (Y = 0, C = 0) = 1.
$$
We note that Case.1 indicate the feature  $C$  has more sufficiency than necessity. Thus, if  $C$  is not binary, we should consider the worst-case PNS value.
We introduced a PNS risk to investigate the PNS value between the representation and Y. When the algorithm optimizes the objective function to minimize the PNS risk, it indicates that the representation we found contains more sufficient and necessary causal information.
Considering there are multiple choices of  $c$  and  $\bar{c}$  (like Example.2), we extend our algorithm by min-max optimization (Eq. 8 in the main text), which aims to find the worst  $\bar{c}$  for  $c$  with highest PNS risk.
A.2 Causal Assumptions and Invariant Learning
Our assumption of the causal graph in Figure 1 is based on previous works on out-of-distribution (OOD) generalization, such as the causal graph in (Liu et al., 2021a; Ahuja et al., 2020b, 2021), they conclude the assumptions in OOD generalization task as (1) fully informative invariant features (FIIF): Feature contains all the information about the label that is contained in input. (2) Non-spurious: The environment information is not correlated with the causal information. (2) partially informative invariant features (PIIF). Our assumption  $\mathbf{X} \perp Y \mid \mathbf{C}$  holds in the case of FIIF. As presented in Ahuja et al. (2021) page 3, IRM is based on the assumption of PIIF, where  $\mathbf{X} \perp Y \mid \mathbf{C}$  does not hold and IRM fails in the case of FIIF. These three causal assumptions are shown in Figure 1 (b).
Given that in the domain generalization problem, the target domain is unavailable,  $\beta_{k}$  and  $\eta$  cannot be optimized directly. Thus, instead of directly optimizing the term  $\beta_{k}$ , we address the OOD generalization problem based on theoretical results in Theorem 4.3, which shows that the algorithm can identify the invariant features. We will then take the invariant learning under Assumption 4.2(1) as an example to show the relationship between invariant learning and satisfaction of Exogeneity. Theorem 4.3 establishes the equivalence between learning to optimize the Eq. (8) and intuitively demonstrates that by optimizing the objective function, the method learns representation  $\mathbf{C}$  that satisfies conditional independence. Specifically, the logic is as follows:
i) In Section 4.3, we describe that our paper is based on the assumption of conditional independence  $\mathbf{X} \perp Y|\mathbf{C}$  (PIIF assumption in Ahuja et al. (2021)), where  $C$  is the invariant variable.
ii) Theorem 4.3 describes that the proposed algorithm makes representation satisfy such conditional independence. The proof of Theorem 4.3 indicates the learned representation is minimal sufficient statistics of  $Y$  (Definition E.2). Therefore, the learned representation identifies the essential information relative to  $Y$  in cause  $\mathbf{C}$  theoretically.
Combining the above two points, the conclusion of Theorem 4.3 implies that the invariant variable can be learned by objective function Eq. (8).
A.3 Framework on Domain Adaptation
In the main text, we provide the learning framework by PNS on OOD generalization task with the practical algorithm. Compared with OOD generalization, Domain adaptation is a task with the data from test distributions provided in the learning process. The framework can be extended to the Domain Adaptation task, extra samples from the test domain help the method to get better generalization performance. Same with the learning framework for OOD generalization, we start from Proposition 3.1. The Eq. (5) is adoptable in the domain adaptation tasks. In that case, the
observation data  $\mathbf{x}$  in the test domain is provided during the source process. Therefore, the terms in Eq. (5) which do not require the label information  $Y$  can be directly evaluated on the test domain. The same with OOD Generalization, for the terms which need to be calculated from label information, we link the test risk  $R_{t}$  and source risk  $R_{s}$  by the divergent measurement  $\beta_{k}(\mathcal{T}\| \mathcal{S})$ . The divergence between  $\mathcal{T}$  and  $\mathcal{S}$  in infinite norm measures support of the source domain that is included in the test domain. We define  $t\backslash s$  as the distribution of the part of test domain distribution  $\mathcal{T}$  which is not included in  $\mathcal{S}$ , we give the upper bound of the worst risk below.
The value of  $\eta_s$  is always lower than  $P_{(\mathbf{x},y),\sim \mathcal{T}}((x,y)\notin \mathrm{supp}(t))$ , which indicates the data distribution for domain  $t$  not covered by domain  $s$ .
Theorem A.1. Given a source domain  $S$  and a test domain  $T$ , when  $T(\mathbf{x})$  is provided then the risk on the test domain is bounded by the risk on source domain as follows.
$$
R _ {t} (\mathbf {w}, \phi , \xi) \leq \lim  _ {k \rightarrow + \infty} M _ {t} ^ {\mathbf {w}} (\phi , \xi) + \beta_ {k} (\mathcal {T} \| \mathcal {S}) (2 [ S F _ {s} (\mathbf {w}, \phi) ] ^ {1 - \frac {1}{k}}) + \eta_ {t \backslash s} (\mathbf {X}, Y). \tag {9}
$$
Theorem A.1 shows the face of our proposed framework under a domain adaptation perspective. In domain adaptation task, we can evaluate the are not available in real-world scenarios, we only have the data samples from the test domain. The proof of Theorem A.1 is similar to Theorem 3.2, the difference is that in domain adaptation, the term  $M_t^{\mathbf{w}}(\phi, \xi)$  can be directly estimated on the provided test data. Because this term does not require the label information.
A.4 Limitations
Firstly, we propose a risk satisfying the identification condition Exogeneity under the assumption of a common causal graph in OOD generalization. However, there exist corner cases of causal assumption like anti-causal and confounder, etc. Hence, we plan to extend our framework with more causal assumptions. In addition, the theorem in our paper is based on the linear classification function. Future work would look into more complex functions in non-linear settings. We leave this exciting direction for future work as it is out of the scope of this paper.
In this section, we introduce the experimental setups and implementation details include the generate rules of synthetic dataset, real-world benchmarks, evaluation strategies, model architecture and hyperparameters setting.
In order to evaluate whether the proposed method captures the most essential information sufficient and necessary causes or not, we design synthetic data which contains four part of variables:
- Sufficient and necessary cause (SN): the sufficient and necessary cause is generated from a Bernoulli distribution  $\mathrm{sn}_i \sim B(0.5)$ , and the label  $y_i$  is generated based on the sufficient and necessary cause where  $y_i = \mathrm{sn}_i \oplus B(0.15)$ . Since  $Y$  is generated from  $\mathrm{SN} \in \{0, 1\}$ , The sufficient and necessary cause has high probability of  $P(Y = 0|do(\mathrm{SN} = 0)) + P(Y = 1|do(\mathrm{SN} = 1))$ .
- Sufficient and unnecessary cause (SF): From the definition and identifiability results of PNS, the sufficient and unnecessary cause  $\mathrm{SF} \in \{0,1\}$  in synthetic data has the same probability with SN, i.e.  $P(Y = 1|do(\mathrm{SF} = 1)) = P(Y = 1|do(\mathrm{SN} = 1))$ , but has lower the probability of  $P(Y = 0|do(\mathrm{SF} = 0))$  than  $P(Y = 0|do(\mathrm{SN} = 0))$ . To generate the value of  $\mathrm{sf}_i$ , we design a transformation function  $f_{\mathrm{sf}}: \{0,1\} \to \{0,1\}$  to generate  $\mathrm{sf}_i$  from sufficient and necessary cause value  $\mathrm{sn}_i$ .
$$
\operatorname {s f} _ {i} = f _ {\mathrm {s f}} \left(\operatorname {s n} _ {i}\right) := \left\{ \begin{array}{l l} B (0. 1) & \text {w h e r e} \operatorname {s n} _ {i} = 0 \\ \operatorname {s n} _ {i} & \text {w h e r e} \operatorname {s n} _ {i} = 1 \end{array} . \right. \tag {10}
$$
From the functional intervention defined by (Puli et al., 2020; Pearl, 2009; Wang & Jordan, 2021), we demonstrate the identifiability property to connect the SF and  $Y$  in following lemma:
Lemma B.1. Let  $SF \coloneqq f_{sf}(SN)$ , the intervention distribution  $P(Y|do(SF = sf_i))$  is identified by the condition distribution  $P(Y|SF = sf_i)$ .
proof:
$$
\begin{array}{l} P (Y | d o (\mathrm {S F} = \mathrm {s f} _ {i})) = \int_ {\substack {\mathrm {S N} \\ c}} p (y | d o (\mathrm {S N})) p (\mathrm {S N} | f _ {\mathrm {s f}} (\mathrm {S N}) = \mathrm {s f} _ {i}) d \mathrm {S N} \tag{11} \\ = \int_ {\mathrm {S N}} p (y | \mathrm {S N}) p (\mathrm {S N} | f _ {\mathrm {s f}} (\mathrm {S N}) = \mathrm {s f} _ {i}) d \mathrm {S N}. \\ \end{array}
$$
The lemma shows that even  $Y$  is only generated from SN in our synthetic simulator, the sufficient cause SF is exogenous relative to  $Y$ .
- Insufficient and necessary cause (NC): From the definition and identifiability results of PNS, the insufficient and necessary cause has the same probability of  $P(Y = 0|do(\mathrm{NC} = 0))$  with  $P(Y = 0|do(\mathrm{SN} = 0))$  but lower the probability of  $P(Y = 1|do(\mathrm{NC} = 1))$  than  $P(Y = 1|do(\mathrm{SN} = 1))$ . To generate the value of NC, we design a transformation function  $f_{\mathrm{nc}}: \{0,1\} \to \{0,1\}$  to generate  $\mathsf{nc}_i$  from sufficient and necessary cause  $\mathsf{sn}_i$ . The generating process of  $\mathsf{nc}_i$  is defined below, and  $\mathsf{nc}_i$  is the cause of  $y$ . Similar with identifiability result of SF in Lemma B.1, NC is exogenous relative to  $Y$ .
$$
\mathrm {n c} _ {i} = f _ {\mathrm {n c}} \left(\mathrm {s n} _ {i}\right) := \mathrm {s n} _ {i} * B (0. 9) \tag {12}
$$
- Spurious: We also generate the variable with spurious correlated with sufficient and necessary cause. We set the degree of spurious correlation as  $s$ , the generator is defined as  $\mathfrak{sp}_i = s*\mathfrak{sn}_i*\mathbf{1}_d + (1 - s)\mathcal{N}(0,1)$ , where  $\mathcal{N}(0,1)$  denotes the Gaussian distribution, we select  $d = 5$  in synthetic generative process. When  $s$  gets higher, the spurious correlation becomes stronger in the data sample. We select  $s = 0.1$  and  $s = 0.7$  in synthetic experiments.
To make the data more complex, we apply a non-linear function to generate  $\mathbf{x}$  from  $\{\mathrm{sn}_i, SF_i, \mathrm{nc}_i, \mathrm{sp}_i\}$ . We firstly generate a temporary vector, which is defined as  $\mathbf{t} = [\mathrm{sn}_i * \mathbf{1}_d, SF_i * \mathbf{1}_d, \mathrm{nc}_i * \mathbf{1}_d, \mathrm{sp}_i * \mathbf{1}_d] + \mathcal{N}(0, 0.3)$ . Then we define functions  $\kappa_1(\mathbf{t}) = \mathbf{t} - 0.5$  if  $\mathbf{t} > 0$ , otherwise  $\kappa_1(\mathbf{t}) = 0$  and  $\kappa_2(\mathbf{t}) = \mathbf{t} + 0.5$  if  $\mathbf{t} < 0$ , otherwise  $\kappa_2(\mathbf{t}) = 0$ .  $\mathbf{x}$  is generated by  $\mathbf{x} = \sigma(\kappa_1(\mathbf{t}) \cdot \kappa_1(\mathbf{t}))$ , where  $\sigma: \mathbb{R}^{4d} \to \mathbb{R}^{4d}$  is sigmoid function. We generate 20000 data samples for training and 500 data samples for evaluation.
B.2 Experiment setup on synthetic data
Methods. In the synthetic experiment, we compare the proposed CaSN with the reduced method CaSN (-m). In the CaSN(-m), we remove the monotonicity evaluator  $M_{s}^{\mathbf{w}}(\phi ,\xi)$  and the  $\delta$ 's constraint  $\| \mathbf{c} - \overline{\mathbf{c}}\| _2\geq \delta$  from the objective Eq. (8).
Evaluation. We evaluate the distance correlation (Jones et al., 1995) between the learned representation  $\hat{\mathbf{c}}_i$  and four features  $\{\mathrm{sn}_i, \mathrm{sf}_i, \mathrm{nc}_i, \mathrm{sp}_i\}$ . For example, the larger distance correlation value between  $\hat{\mathbf{c}}_i$  and  $\mathrm{sn}_i$  means  $\hat{\mathbf{c}}_i$  contains more information of  $\mathrm{sn}_i$ .
Model Architecture. For the synthetic dataset, we use the three-layer MLP networks with an activate function as the feature inference model. Denoting  $V_{k}$  as weighting parameters, the neural network is designed as  $V_{1}\mathrm{ELU}(V_{2}\mathrm{ELU}(V_{3}[\mathbf{x}])$ , where  $ELU()$  is the activation function (Clevert et al., 2015). The dimension of hidden vectors calculated from  $V_{1}, V_{2}, V_{3}$  are specified as 64, 32, 128 separately. Since the output of the feature learning network is consist of the mean and variance vector, the dimension of representation is 64. The prediction neural networks w which predict  $y$  by representation c.
B.3 Experiment setup on DomainBed
Dataset. In addition to the experiments on synthetic data, we also evaluate the proposed method on real-world OOD generalization datasets. We use the code from the popular repository DomainBed (Gulrajani & Lopez-Paz, 2020), and use the dataset which is downloaded or generated by the DomainBed code. The dataset includes the following three:
- PACS (Li et al., 2017): PACS dataset contains overall 9,991 data samples with dimension (3,224, 224). There are 7 classes in the dataset. The datasets have four domains art, cartoons, photos, sketches.
- VLCS (Fang et al., 2013): VLCS dataset contains overall 10,729 data samples with dimension (3, 224, 224). There are 5 classes in the dataset. The datasets have four domains Caltech101, LabelMe, SUN09, VOC2007.
The DomainBed randomly split the whole dataset as training, validation and test dataset.
Baselines. For OOD Generalization task, we consider following baselines realized by DomainBed.
- Empirical Risk Minimization (ERM) (Vapnik, 1999) (ERM, Vapnik [1998]) is a common baseline that minimizes the risk on all the domain data.
- Invariant Risk Minimization (IRM) (Arjovsky et al., 2019) minimize the risk for different domains separately. It extract the representation on all the domains and project them on a shared representation space. Then, all the domain share one prediction model. The algorithm is formed as bi-level optimization objective.
- Group Distributionally Robust Optimization (DRO) (Sagawa et al., 2019) weigh the different domains by their evaluated error. It reweights the optimization of domains in objective. The domain with a larger error gets a larger weight.
- Inter-domain Mixup (Mixup) (Xu et al., 2020; Yan et al., 2020; Wang et al., 2020), ERM is performed for linear interpolation of samples of random pairs from domains and their labels.
- Meta-Learning for Domain Generalization (MLDG) (Li et al., 2018a) uses meta learning strategy (Finn et al., 2017) to learn the cross domain generalization.
- MMD method for domain generalization (Li et al., 2018b) method uses the MMD (Gretton et al., 2012) of the feature distribution to measure the distance of domain representation. They learn the invariant representation across domains by minimize MMD.
- Domain-Adversarial Neural Networks DANN (Ganin et al., 2016) uses adversarial networks to learn the invariant features across domains.
- Class-conditional DANN CDANN (Li et al., 2018c) is an extension of DANN, which is learned to match the conditional distribution by giving the label information.
Evaluation. All the evaluation results are reported from the DomainBed paper (Gulrajani & Lopez-Paz, 2020). We use the evaluation strategy from DomainBed. DomainBed provides the evaluation strategy Training-domain validation set. The training dataset is randomly split as training and validation datasets, the hyperparameters are selected on the validation dataset, which maximizes the performance of the validation dataset. The overall accuracy results are evaluated on the test dataset rather than the validation dataset.
All the experiments are conducted based on a server with a 16-core CPU,  $128\mathrm{g}$  memory and RTX 5000 GPU.
Implementation Details. We implement CaSN based on the DomainBed repository. The feature extractor and classifier use different neural networks across different datasets.
We set distribution of representation  $\mathbf{C}$  as  $\hat{P}_s^\phi (\mathbf{C}|\mathbf{X} = \mathbf{x}) = \mathcal{N}(\mu (\mathbf{x};\phi),\sigma (\mathbf{x};\phi))$  and its prior as  $\pi_{\mathbf{C}} = \mathcal{N}(\mu_0,\sigma_0)$ .  $\mu_0$  and  $\sigma_0$  are pre-defined and not updated in the learning process. Similarly, for  $\bar{\mathbf{C}}$ , we define  $\hat{P}_s^\xi (\bar{\mathbf{C}} |\mathbf{X} = \mathbf{x}) = \mathcal{N}(\bar{\mu} (\mathbf{x};\xi),\bar{\sigma} (\mathbf{x};\xi))$  and  $\pi_{\bar{\mathbf{C}}} = \mathcal{N}(\bar{\mu}_{\mathbf{0}},\bar{\sigma}_{\mathbf{0}})$ . Colored Mnist: The model architecture on Colored Mnist dataset is given by DomainBed repository. We use MNIST ConvNet for feature extraction. The details of the layers are referred from Table 7 in Gulrajani & Lopez-Paz (2020). We simply use a linear function as the label prediction network and transformation function  $T$ . In the Colored Mnist dataset, the variance of  $\mathbf{c}$  is fixed as 0.001.
PACS and VLCS: The model architecture on PACS and VLCS datasets are both given by DomainBed repository. They use ResNet-50 for feature extraction. In DomainBed, they replace the last layer of a ResNet50 pre-trained on ImageNet and fine-tune and freeze all the bach normalization layer before fine-tune. We simply use a linear function as the label prediction network  $\mathbf{w}$  and transformation function  $T$ . In the PACS and VLCS datasets, the variances of  $\mathbf{c}$  are also fixed as 0.001
The Hyperparameters are shown in Table 3. The general hyperparameters (e.g. ResNet, Mnist, Not Minist) are directly given from Table 8 in Gulrajani & Lopez-Paz (2020). All the experiments run for 2 times.
Table 2: Results on Colored Mnist
Condition	+90%	+80%	-90%	Avg	Min
ERM	71.7 ± 0.1	72.9 ± 0.2	10.0 ± 0.1	51.5	10.0
IRM	72.5 ± 0.1	73.3 ± 0.5	10.2 ± 0.3	52.0	10.2
GroupDRO	73.1 ± 0.3	73.2 ± 0.2	10.0 ± 0.2	52.1	10.0
Mixup	72.7 ± 0.4	73.4 ± 0.1	10.1 ± 0.1	52.1	10.1
MLDG	71.5 ± 0.2	73.1 ± 0.2	9.8 ± 0.1	51.5	9.8
MMD	71.4 ± 0.3	73.1 ± 0.2	9.9 ± 0.3	51.5	9.9
DANN	71.4 ± 0.9	73.1 ± 0.1	10.0 ± 0.0	51.5	10.0
CDANN	72.0 ± 0.2	73.0 ± 0.2	10.2 ± 0.1	51.7	10.2
CaSN (Ours)	72.6 ± 0.1	73.7 ± 0.1	10.3 ± 0.3	52.2	10.3
Table 3: Hyperparameters setting
Parameter	Default value	Random distribution
Basic	batch size	32	2Uniform(3,5.5)
minimization process learning rate	0.0001	10Uniform(-5,-3.5)
maximization process learning rate	0.00001	10Uniform(-6,-4)
Not Mnist	weight decay	0	10Uniform(-6,-2)
generator weight decav	0	10Uniform(-6,-2)
Mnist	weight decay	0	0
generator weight decav	0	0
CaSN	δ	0.7	{0.3,0.7,1.0,1.5,3.0}
λ	0.01	{0.001,0.01,0.1}
Weight of constraint ||c-¯c||2	0.1	{0.001,0.01,0.1,0.5,0.7}
CaSN(irm)	δ	0.7	{0.3,0.7,1.0,1.5,3.0}
λ	0.01	{0.001,0.01,0.1,0.5,0.7}
Weight of constraint ||c-¯c||2	0.1	{0.001,0.01,0.1,0.5,0.7}
Weight of IRM penalty Lirm	0.001	{0.01,0.001,1e-5}
IRM penalty iters	1000	{500,1000}
CaSN(mmd)	δ	0.7	{0.3,0.7,1.0,1.5,3.0}
λ	0.01	{0.001,0.01,0.1,0.5,0.7}
Weight of constraint ||c-¯c||2	0.1	{0.001,0.01,0.1, 0.5,0.7}
Weight of MMD penalty Lmmd	1	{1}
Adversarial	Max optimization per iters	500	{500,1000}
B.4 Experiment setup on SpuCo
We provide the experiment on the large scale spurious correlation dataset SpuCo (Joshi et al., 2023). SpuCo is a Python package developed to address spurious correlations in the context of visual prediction scenarios. It includes two datasets: the simulated dataset SpuCoMNIST and the large-scale real-world dataset SpuCoANIMALS. This Python package provides code for benchmark methods on these datasets, along with test results.
Dataset. We test CaSN on the large-scale real-world dataset SpuCoANIMALS, which is derived from ImageNet (Russakovsky et al., 2015). This dataset captures spurious correlations in real-world settings with greater fidelity compared to existing datasets. There are 4 classes in this dataset: landbirds, waterbirds, small dog breeds and big dog breeds, which are spuriously correlated with different backgrounds. Based on the different backgrounds, this dataset contains 8 groups with more than 10,000 samples.
Baselines. We realize CaSN based on the code of SpuCo. We have CaSN implementation, one is based on ERM, the other is based on the strategy of Group Balance. We compare CaSN with ERM, GroupDRO and Group Balance.
Table 4: Results on SpuCo
ERM	CaSN	GroupDRO	GB	CaSN-GB
Average	80.26±4.1	84.34±0.6	50.7±4.2	49.9±1.4	50.54±7.6
Min	8.3±1.3	9.7±2.7	36.7±0.6	41.1±2.7	42.0±0.8
Hyperparameters. The hyperparameters employed for robust retraining are detailed as follows. During the robust training phase, we set the hyperparameters as their default value:
When conducting training from scratch, we utilize SGD as the optimizer with a learning rate of 0.0001, a weight decay of 0.0001, a batch size of 128, and a momentum of 0.9. The model undergoes training for 300 epochs, with early stopping applied based on its performance on the validation set. In addition to the basic hyperparameters, we set  $\delta = 3$  and  $\lambda \in \{0.005, 0.01\}$ .
C.1 OOD generalization results on Colored MNIST