SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Due to the page limitation in the main text, we show OOD generalization results on Colored Mnist data in Table 2.
C.2 Domain generalization on SpuCo
The results on SpuCo (Joshi et al., 2023) based on the code  $\mathrm{SpuCo}^4$ . We test CaSN on SpuCoAnimal which comprises 8 different spurious correlation groups.
We set hyperparameters as  $\delta = 3$  and  $\lambda \in 0.005, 0.01$ , while the remaining parameters followed SpuCo's default values. We conducted two types of experiments: the first involved training the model without providing any additional information, and the second involved resampling based on the spurious correlation group information, where the base models corresponded to ERM and Group Balance (GB) as presented in SpuCo. We also include GroupDRO as one of the baselines, the baseline performance is reported SpuCo. Data augmentation-based methods, ERM* and GB*, were not included in these experiments.
We reported the results in Table 4, with each result obtained from 4 fixed random seeds. The baseline results are directly taken from SpuCo. Our method showed higher average accuracy on average accuracy compared to the baseline, and even the worst spurious correlation group exhibited higher accuracy than the baseline.
D Proof of Theorems
In this section, we provide the proof of the risk bounds which include Proposition 3.1, Theorem 3.2 and Theorem 3.3 for OOD generalization task.
D.1 Proposition 3.1
To explicitly form the Monotonocity evaluator, we decompose the original objective by three terms defined by sufficiency objective  $SF_{t}(\mathbf{w},\phi)$ , necessity objective  $NC_{t}(\mathbf{w},\xi)$  and Monotonicity evaluator objective  $M_{t}^{\mathbf{w}}(\phi ,\xi)$ . The Monotonicity  $M_{t}^{\mathbf{w}}(\phi ,\xi)$  term can be decomposed as
$$
M _ {t} ^ {\mathbf {w}} (\phi , \xi) = S F _ {t} (\mathbf {w}, \phi) (1 - N C _ {t} (\mathbf {w}, \xi)) + (1 - S F _ {t} (\mathbf {w}, \phi)) N C _ {t} (\mathbf {w}, \xi). \tag {13}
$$
The following equation understands the above decomposition.
$$
\begin{array}{l} P (\operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \operatorname {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}})) \\ = P \left(\operatorname {s i g n} \left(\mathbf {w} ^ {\top} \mathbf {c}\right) = y\right) P \left(\operatorname {s i g n} \left(\mathbf {w} ^ {\top} \bar {\mathbf {c}}\right) = y\right) + P \left(\operatorname {s i g n} \left(\mathbf {w} ^ {\top} \mathbf {c}\right) \neq y\right) P \left(\operatorname {s i g n} \left(\mathbf {w} ^ {\top} \bar {\mathbf {c}}\right) \neq y\right). \tag {14} \\ \end{array}
$$
$^{4}$ https://github.com/BigML-CS-UCLA/SpuCo
We can further derive Eq.13 as follows.
$$
\begin{array}{l} M _ {t} ^ {\mathbf {w}} (\phi , \xi) = S F _ {t} (\mathbf {w}, \phi) (1 - N C _ {t} (\mathbf {w}, \xi)) + (1 - S F _ {t} (\mathbf {w}, \phi)) N C _ {t} (\mathbf {w}, \xi) \\ = \underbrace {S F _ {t} (\mathbf {w} , \phi) + N C _ {t} (\mathbf {w} , \xi)} _ {R _ {t} (\mathbf {w}, \phi , T)} - 2 S F _ {t} (\mathbf {w}, \phi) N C _ {t} (\mathbf {w}, \xi) \tag {15} \\ = R _ {t} (\mathbf {w}, \phi , \xi) - 2 S F _ {t} (\mathbf {w}, \phi) N C _ {t} (\mathbf {w}, \xi). \\ \end{array}
$$
Then we can rewrite the original objective Eq. (4) by
$$
R _ {t} (\mathbf {w}, \phi , \xi) = M _ {t} ^ {\mathbf {w}} (\phi , \xi) + 2 S F _ {t} (\mathbf {w}, \phi) N C _ {t} (\mathbf {w}, \xi). \tag {16}
$$
Then we get the results of Proposition 3.1. The reasons why we need this proposition are because the final objective need to explicitly evaluate the Monotonicity.
D.2 Theorem 3.2
The motivation of Theorem 3.2 is to bridge the gap between the risk on the source domain and the risk on the test domain. To prove the result in Theorem 3.2, we refer to the technicals in Germain et al. (2016), We first define  $o = \mathbb{E}_{(\mathbf{x},y)\sim \mathcal{T}}\mathrm{I}[(\mathbf{x},y)\notin \sup (S)]$  , then we can get the value of  $\delta$  -Monotonicity measurement on the samples from test domain that are not included in source domain is:
$$
\begin{array}{l} \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {T}} \mathrm {I} [ (\mathbf {x}, y) \notin \operatorname {s u p p} (\mathcal {S}) ] \mathbb {E} _ {\mathbf {c} \sim P _ {t} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathbb {E} _ {\bar {\mathbf {c}} \sim P _ {t} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \mathrm {I} [ \operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \operatorname {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ] \\ = o \mathbb {E} _ {t \setminus s} \mathbb {E} _ {\mathbf {c} \sim P _ {t} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathbb {E} _ {\bar {\mathbf {c}} \sim P _ {t} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} I [ \mathrm {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \mathrm {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ] = o M _ {t} ^ {\mathbf {w}} (\phi , \xi). \\ \end{array}
$$
Similarly, the overall risk on the samples from the test domain that is not included in the source domain is  $\eta_{t\backslash s}(\mathbf{X},Y)$
Then we change the distribution measure, we also take  $M_t^{\mathbf{w}}(\phi ,\xi)$  of Eq. (5) as an example.
$$
\begin{array}{l} M _ {t} ^ {\mathbf {w}} (\phi , \xi) \\ = \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {T}} \mathbb {E} _ {\mathbf {c} \sim P _ {t} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathbb {E} _ {\bar {\mathbf {c}} \sim P _ {t} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \mathrm {I} [ \mathrm {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \mathrm {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ] \\ = \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {S}} \frac {\mathcal {T}}{\mathcal {S}} \mathbb {E} _ {\mathbf {c} \sim P _ {t} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathbb {E} _ {\bar {\mathbf {c}} \sim P _ {t} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \mathrm {I} [ \mathrm {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \mathrm {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ] + o M _ {t \backslash s} ^ {\mathbf {w}} (\phi , \xi) \\ \leq \beta_ {k} (\mathcal {T} \| \mathcal {S}) [ \mathbb {E} _ {\mathbf {c} \sim P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathbb {E} _ {\bar {\mathbf {c}} \sim P _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \mathrm {I} [ \mathrm {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \mathrm {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ] ^ {\frac {k}{k - 1}} ] ^ {1 - \frac {1}{k}} + o M _ {t \setminus s} ^ {\mathbf {w}} (\phi , \xi) \\ = \beta_ {k} (\mathcal {T} \| \mathcal {S}) \left[ \mathbb {E} _ {\mathbf {c} \sim P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathbb {E} _ {\bar {\mathbf {c}} \sim P _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \mathrm {I} [ \operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \operatorname {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ] \right] ^ {1 - \frac {1}{k}} + o M _ {t \backslash s} ^ {\mathbf {w}} (\phi , \xi). \tag {17} \\ \end{array}
$$
The third line is due to Hölder inequality. For the last line, we remove the exponential term  $\frac{k}{k-1}$  in  $\mathrm{I}[\mathrm{sign}(\mathbf{w}^\top \mathbf{c}) = \mathrm{sign}(\mathbf{w}^\top \overline{\mathbf{c}})]^{\frac{k}{k-1}}$  since the function takes the results from  $\{0,1\}$ . Similarly on term  $SF_s(\mathbf{w},\phi)$ , we can get the final bound of overall  $R_t(\mathbf{w},\phi,\xi)$  as Eq. (7) shows.
D.3 Theorem 3.3
In this theorem, we study how the risk on distribution is bounded by empirical risk. The proof of the theorem refers to popular inequality Jensen's inequality, Markov inequality and Hoeffding's inequality. We first focus on the term  $SF_{s}(\mathbf{w},\phi)$ . The sketch of the proof is that firstly we will use the variational inference process to change the measure of distribution. Then, we use Markov's inequality to calculate the bound of risk.
Define  $\Delta (SF_s(\mathbf{w},\phi)) = \widehat{SF}_s(\mathbf{w},\phi) - SF_s(\mathbf{w},\phi)$ . We apply the variational inference trick and use Jensen's inequality to get the following inequality:
$$
\begin{array}{l} \Delta \left(S F _ {s} (\mathbf {w}, \phi)\right) \\ = \widehat {S F} _ {s} (\mathbf {w}, \phi) - S F _ {s} (\mathbf {w}, \phi) \\ \leq \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} (\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}) - 2 (n - 1) \mathbb {E} _ {\mathcal {S}} \mathbb {E} _ {P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \ln \frac {P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})}{\pi_ {\mathbf {C}}} \\ + \ln \mathbb {E} _ {\mathbf {c} \sim \pi_ {\mathbf {C}}} \exp (\mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {I} [ \operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) \neq y ]) \\ - \ln \mathbb {E} _ {\mathbf {c} \sim \pi_ {\mathbf {C}}} \exp \left(\mathbb {E} _ {\mathcal {S}} \mathrm {I} [ \operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) \neq y ]\right), \tag {18} \\ = \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} \big (\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}} \big) - \mathbb {E} _ {\mathcal {S}} \mathbb {E} _ {P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \ln \frac {P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})}{\pi_ {\mathbf {C}}} \\ + \ln \mathbb {E} _ {\mathbf {c} \sim \pi_ {\mathbf {C}}} \exp (\Delta (S)), \\ = \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} \left(\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}\right) - \mathbb {E} _ {\mathcal {S}} \mathrm {K L} \left(P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}\right) \\ + \ln \mathbb {E} _ {\mathbf {c} \sim \pi_ {\mathbf {C}}} \exp (\Delta (S)), \\ \end{array}
$$
where  $\Delta (S) = |\mathbb{E}_{\mathcal{S}^n}\mathrm{I}[\mathrm{sign}(\mathbf{w}^\top \mathbf{c})\neq y] - \mathbb{E}_S\mathrm{I}[\mathrm{sign}(\mathbf{w}^\top \mathbf{c})\neq y]|$ . Recall that Hoeffding's inequality, we get the following inequality.
$$
P [ \Delta (S) \geq \eta ] \leq \exp (- 2 n) \eta^ {2} \tag {19}
$$
Then, denoting the density function of  $\Delta (S)$  as  $f(\Delta (S))$
$$
\begin{array}{l} P [ \Delta (S) \geq \eta ] = e ^ {- 2 n \eta^ {2}} \\ \Rightarrow \int_ {\eta} ^ {\infty} f (\Delta (S)) d \Delta (S) = e ^ {- 2 n \eta^ {2}} \\ \Rightarrow f (\eta) = 4 n \eta e ^ {- 2 n \eta^ {2}}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {S} \left[ \exp (2 (n - 1) \Delta^ {2} (S)) \right] \\ = \int_ {0} ^ {1} f (\Delta (S)) \exp (2 (n - 1) \Delta^ {2} (S)) d \Delta (S) \\ \leq \int_ {0} ^ {1} 4 n \Delta (S) \exp (- 2 n \Delta^ {2} (S)) \exp (2 (n - 1) \Delta^ {2} (S)) d \Delta (S) \tag {20} \\ = - n \exp (- 2 \Delta^ {2} (S)) | _ {0} ^ {1} \\ = (1 - e ^ {- 2}) n <   n \\ \end{array}
$$
Combining Eq. (20) with Eq. (18). Suppose  $g(\Delta(S)) = \ln \mathbb{E}_{\pi_{\mathbf{C}}}[\exp(2(n - 1)\Delta^2(S))]$ , since we have,
$$
\mathbb {E} _ {\mathcal {S}} \mathbb {E} _ {\pi_ {\mathbf {C}}} \left[ \exp \left(2 (n - 1) \Delta^ {2} (S)\right) \right] \leq n
$$
by Markov's inequality, we further get,
$$
P _ {(\mathcal {S})} [ g (\Delta (S)) \geq \eta ] \leq \frac {n}{e ^ {\eta}}, \tag {21}
$$
Suppose that  $\eta = \ln (n / \epsilon)$ , and with the probability of at least  $1 - \epsilon$ , we have that for all  $\pi_{\mathbf{C}}$
$$
\Rightarrow 2 (n - 1) \ln \mathbb {E} _ {\pi_ {\mathbf {C}}} [ \exp (\Delta^ {2} (S)) ] \leq \eta = \ln \frac {n}{\epsilon} \tag {22}
$$
Since  $\Delta S\in [0,1]$ , then we have,
$$
\exp \left(\Delta^ {2} (S)\right) \leq \left(\exp \left(\Delta (S)\right)\right) ^ {2} \leq e \cdot \exp \left(\Delta^ {2} (S)\right) \tag {23}
$$
The last inequality due to  $\exp (\Delta^2 (S)) - (\exp (\Delta (S)))^2$  is a monotonically decreasing when  $\Delta S\in [0,1]$ . Then, according to Jensen's inequality, we conclude that
$$
\begin{array}{l} \ln (\mathbb {E} _ {\mathbf {c} \sim \pi (\mathbf {C})} \exp \Delta (S)) ^ {2} \leq \ln \mathbb {E} _ {\mathbf {c} \sim \pi (\mathbf {C})} (\exp \Delta (S)) ^ {2} \\ \leq \ln (e \cdot \mathbb {E} _ {\mathbf {c} \sim \pi (\mathbf {C})} \exp \Delta^ {2} (S)) \leq \frac {1}{2 (n - 1)} \ln \frac {n}{\epsilon} + 1 \tag {24} \\ \Rightarrow \ln (\mathbb {E} _ {\mathbf {c} \sim \pi (\mathbf {C})} \exp \Delta (S)) \leq \frac {1}{4 (n - 1)} \ln \frac {n}{\epsilon} + \frac {1}{2} \\ \end{array}
$$
$$
\begin{array}{l} \left. \widehat {S F} _ {s} (\mathbf {w}, \phi) - S F _ {s} (\mathbf {w}, \phi) \right| \\ \leq \left| \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} \left(\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}\right) - \mathbb {E} _ {\mathcal {S}} \mathrm {K L} \left(P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}\right) + \frac {1}{4 (n - 1)} \ln (n / \epsilon) + \frac {1}{2} \right| \tag {25} \\ \leq | \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} (\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}) + \mathbb {E} _ {\mathcal {S}} \mathrm {K L} (P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}) + \frac {1}{4 (n - 1)} \ln (n / \epsilon) + \frac {1}{2} |. \\ \end{array}
$$
According to the assumption, we have
$$
\left| \widehat {S F} _ {s} (\mathbf {w}, \phi) - S F _ {s} (\mathbf {w}, \phi) \right| \leq \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} \left(\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}\right) + \frac {1}{4 (n - 1)} \ln (n / \epsilon) + C. \tag {26}
$$
We get the results demonstrated in Theorem 3.3 (1).
For the term  $M_s^{\mathbf{w}}(\phi, \xi)$ , we define  $\Delta(M_s) = M_s^{\mathbf{w}}(\phi, \xi) - \widehat{M_s^{\mathbf{w}}}(\phi, \xi)$ , where  $\widehat{M_s^{\mathbf{w}}}(\phi, \xi) := \mathbb{E}_{\mathcal{S}^n} \mathbb{E}_{\mathbf{c} \sim \hat{P}_s^\phi(\mathbf{C}|\mathbf{X} = \mathbf{x})} \mathbb{E}_{\bar{\mathbf{c}} \sim \hat{P}_s^\xi(\bar{\mathbf{C}}|\mathbf{X} = \mathbf{x})} \mathrm{I}[\mathrm{sign}(\mathbf{w}^\top \mathbf{c}) = \mathrm{sign}(\mathbf{w}^\top \bar{\mathbf{c}})]$ . Different from Theorem 3.3 (1), the monotonicity measurement has an extra expectation on  $\bar{\mathbf{c}}$ . We apply Jensen's inequality again and then use the variational inference trick to get the derivation results.
$$
4 (n - 1) ^ {2} \Delta \left(M _ {s}\right) ^ {2} = 4 (n - 1) ^ {2} \left(M _ {s} ^ {\mathbf {w}} (\phi , \xi) - \widehat {M} _ {s} ^ {\mathbf {w}} (\phi , \xi)\right) ^ {2} \tag {27}
$$
Then, we consider the term  $M_s^{\mathbf{w}}(\phi ,\xi)$  and  $\widehat{M}_s^{\mathbf{w}}(\phi ,\xi)$  separately.
$$
\begin{array}{l} M _ {s} ^ {\mathbf {w}} (\phi , \xi) = \mathbb {E} _ {\mathcal {S}} \mathbb {E} _ {\mathbf {c} \sim P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathbb {E} _ {\bar {\mathbf {c}} \sim P _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \mathrm {I} [ \operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \operatorname {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ] \\ = \mathbb {E} _ {\mathcal {S}} [ \mathbb {E} _ {\mathbf {c}} \mathbb {E} _ {\bar {\mathbf {c}}} \ln \frac {P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})}{\pi_ {\mathbf {C}}} + \mathbb {E} _ {\mathbf {c}} \mathbb {E} _ {\bar {\mathbf {c}}} \ln \frac {P _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})}{\pi_ {\bar {\mathbf {C}}}} \\ + \mathbb {E} _ {\mathbf {c}} \mathbb {E} _ {\bar {\mathbf {c}}} \ln \frac {\pi_ {\mathbf {C}}}{P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \frac {\pi_ {\bar {\mathbf {C}}}}{P _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \exp (\mathrm {I} [ \operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \operatorname {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ]) ] \tag {28} \\ \leq \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim \mathcal {S}} \left[ \mathrm {K L} \left(P _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}\right) + \mathrm {K L} \left(P _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x}) \| \pi_ {\bar {\mathbf {C}}}\right) \right] \\ + \ln \mathbb {E} _ {\mathbf {c} \sim \pi_ {\mathbf {C}}} \mathbb {E} _ {\bar {\mathbf {c}} \sim \pi_ {\bar {\mathbf {C}}}} \exp (\mathbb {E} _ {\mathcal {S}} \mathrm {I} [ \mathrm {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \mathrm {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ]). \\ \end{array}
$$
Similarly, for the empirical risk  $\widehat{M}_s^{\mathbf{w}}(\phi ,\xi)$
$$
\begin{array}{l} \widehat {M} _ {s} ^ {\mathbf {w}} (\phi , \xi) = \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {S} ^ {n}} \mathbb {E} _ {\mathbf {c} \sim \hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathbb {E} _ {\bar {\mathbf {c}} \sim \hat {P} _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \mathrm {I} [ \operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \operatorname {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ] \\ = \mathbb {E} _ {\mathcal {S} ^ {n}} [ \mathbb {E} _ {\mathbf {c}} \mathbb {E} _ {\bar {\mathbf {c}}} \ln \frac {\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})}{\pi_ {\mathbf {C}}} + \mathbb {E} _ {\mathbf {c}} \mathbb {E} _ {\bar {\mathbf {c}}} \ln \frac {\hat {P} _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})}{\pi_ {\bar {\mathbf {C}}}} \\ + \mathbb {E} _ {\mathbf {c}} \mathbb {E} _ {\bar {\mathbf {c}}} \ln \frac {\pi_ {\mathbf {C}}}{\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \frac {\pi_ {\bar {\mathbf {C}}}}{\hat {P} _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \exp (\mathrm {I} [ \operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \operatorname {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ]) ] \tag {29} \\ \leq \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim \mathcal {S} ^ {n}} [ \mathrm {K L} (\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}) + \mathrm {K L} (\hat {P} _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x}) \| \pi_ {\bar {\mathbf {C}}}) ] \\ + \ln \mathbb {E} _ {\mathbf {c} \sim \pi_ {\mathbf {C}}} \mathbb {E} _ {\bar {\mathbf {c}} \sim \pi_ {\bar {\mathbf {C}}}} \exp (\mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {I} [ \mathrm {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \mathrm {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ]). \\ \end{array}
$$
Combining Eq. (29) with Eq. (28) and plugin to Eq. (27), we have
$$
\begin{array}{l} 4 (n - 1) ^ {2} \Delta \left(M _ {s}\right) ^ {2} = 4 (n - 1) ^ {2} \left(M _ {s} ^ {\mathbf {w}} (\phi , \xi) - \widehat {M} _ {s} ^ {\mathbf {w}} (\phi , \xi)\right) ^ {2} \\ \leq (2 (n - 1) (\mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} (\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}) + \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} (\hat {P} _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x}) \| \pi_ {\bar {\mathbf {C}}}) \\ + \ln \mathbb {E} _ {\mathbf {c} \sim \pi_ {\mathbf {C}}} \mathbb {E} _ {\mathbf {c} \sim \pi_ {\bar {\mathbf {C}}}} \exp {(2 (n - 1) \Delta (M _ {s} ^ {\prime}))) ^ {2}}, \\ \end{array}
$$
where  $\Delta(M_s') = |\mathbb{E}_{\mathcal{S}^n}\mathrm{I}[\mathrm{sign}(\mathbf{w}^\top \mathbf{c}) = \mathrm{sign}(\mathbf{w}^\top \bar{\mathbf{c}})] - \mathbb{E}_S\mathrm{I}[\mathrm{sign}(\mathbf{w}^\top \mathbf{c}) = \mathrm{sign}(\mathbf{w}^\top \bar{\mathbf{c}})]|$ . The rest of the proof is similar to Theorem 3.3 (1), thus we get the theoretical results of Theorem 3.3 (2).
E Satisfaction of Exogeneity
In this section, we provide the proof of Theorem 4.3 and demonstrate why minimizing the objective is equivalent to finding the  $\mathbf{C}$  satisfying conditional independence.
The proof process consists of three steps: (i) proof the optimization of
$$
\min  _ {\phi , \mathbf {w}} \widehat {S F} _ {s} (\mathbf {w}, \phi) + \lambda \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} \left(\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}\right) \tag {30}
$$
is equivalent with the optimization of Information Bottleneck Shamir et al. (2010a) objective  $L_{\mathrm{ib}} = \max I(\mathbf{C},Y) - \lambda I(\mathbf{X},\mathbf{C})$ , where  $I(A,B)$  denotes the mutual information between  $A$  and  $B$ .  $\mathbf{X}, Y$  are from  $S^n$ . (ii) The optimal solution of Information Bottleneck satisfies  $\mathbf{X} \perp Y|\mathbf{C}$ . To simplify the writing, we represent  $P_s(\cdot)$  by  $P(\cdot)$ .
For step (i): The objective Eq. (30) is coincide with the objective of variational autoencoder (Doersch, 2016) which is proved to be equivalent to the IB objective. The theoretical results are provided in Alemi et al. (2016), and we report the proof process from the notations in our paper. The derivation starts with bound the  $\mathrm{I}(\mathbf{C},Y)$  and  $\mathrm{I}(\mathbf{C},\mathbf{X})$  separately, for the term  $I(\mathbf{C},Y)$
$$
I (\mathbf {C}, Y) = \int P (Y, \mathbf {C}) \log \frac {P (Y , \mathbf {C})}{P (Y) P (\mathbf {C})} d Y d \mathbf {C} = \int P (Y, \mathbf {C}) \log \frac {P (Y \mid \mathbf {C})}{P (Y)} d Y d \mathbf {C}. \tag {31}
$$
Decompose the  $P(Y\mid \mathbf{C})$  as below
$$
P (Y \mid \mathbf {C}) = \int d X P (\mathbf {X}, Y \mid \mathbf {C}) = \int d \mathbf {X} P (Y \mid \mathbf {X}) P (\mathbf {X} \mid \mathbf {C}) = \int d \mathbf {X} \frac {P (Y \mid \mathbf {X}) \hat {P} ^ {\phi} (\mathbf {C} \mid \mathbf {X}) P (\mathbf {X})}{P (\mathbf {C})}.
$$
In the process, the  $P(Y \mid \mathbf{C})$  is approximated by parameterized  $\hat{P}(Y \mid \mathbf{C})$ . Since the KL divergence is always larger than 0,
$$
\mathrm {K L} [ P (Y \mid \mathbf {C}), \hat {P} (Y \mid \mathbf {C}) ] \geq 0 \Longrightarrow \int P (Y \mid \mathbf {C}) \log P (Y \mid \mathbf {C}) d Y \geq \int P (Y \mid \mathbf {C}) \log \hat {P} (Y \mid \mathbf {C}) d Y.
$$
$$
\begin{array}{l} I (\mathbf {C}, Y) \geq \int P (Y, \mathbf {C}) \log \frac {\hat {P} (Y \mid \mathbf {C})}{P (Y)} \\ = \int P (Y, \mathbf {C}) \log \hat {P} (Y \mid \mathbf {C}) - \int d Y P (Y) \log P (Y) d Y d \mathbf {C} \\ = \int P (Y, \mathbf {C}) \log \hat {P} (Y \mid \mathbf {C}) + H (Y) d Y d \mathbf {C} \\ \end{array}
$$
Then  $I(\mathbf{C},Y)$  is lower bounded by follows:
$$
I (\mathbf {C}, Y) \geq \int P (\mathbf {X}) P (Y \mid \mathbf {X}) P (\mathbf {C} \mid d \mathbf {X} d Y d \mathbf {C}) \log \hat {P} ^ {\phi} (Y \mid \mathbf {C}) d \mathbf {X} d Y d \mathbf {C}
$$
Then, consider the term  $\lambda I(\mathbf{C},\mathbf{X})$  , it is decomposed by following equation:
$$
\begin{array}{l} I (\mathbf {C}, \mathbf {X}) = \int \hat {P} ^ {\phi} (\mathbf {X}, \mathbf {C}) \log \frac {\hat {P} ^ {\phi} (\mathbf {C} \mid \mathbf {X})}{P (\mathbf {C})} d \mathbf {C} d \mathbf {X} \tag {32} \\ = \int \hat {P} ^ {\phi} (\mathbf {X}, \mathbf {C}) \log \hat {P} ^ {\phi} (\mathbf {C} \mid \mathbf {X}) d \mathbf {C} d \mathbf {X} - \int P (\mathbf {C}) \log P (\mathbf {C}) d \mathbf {C} d \mathbf {X}. \\ \end{array}
$$
It is upper bounded by
$$
I (\mathbf {C}, \mathbf {X}) \leq \int P (\mathbf {X}) \hat {P} ^ {\phi} (\mathbf {C} \mid \mathbf {X}) \log \frac {\hat {P} ^ {\phi} (\mathbf {C} \mid \mathbf {X})}{P (\mathbf {C})} d \mathbf {X} d \mathbf {C}
$$
where  $P(\mathbf{C})$  is prior of  $\mathbf{C}$ . Combining the upper bound of  $I(\mathbf{C},\mathbf{X})$  and the lower bound of  $I(\mathbf{C},Y)$  together, the lower bound of IB objective is
$$
\begin{array}{l} I (\mathbf {C}, Y) - \lambda I (\mathbf {C}, \mathbf {X}) \geq \int P (\mathbf {X}) P (Y \mid \mathbf {X}) P (\mathbf {C} \mid \mathbf {x}) \log \hat {P} (Y \mid \mathbf {C}) d \mathbf {X} d Y d \mathbf {C} \tag {33} \\ - \lambda \int P (\mathbf {X}) \hat {P} ^ {\phi} (\mathbf {C} \mid \mathbf {X}) \log \frac {\hat {P} ^ {\phi} (\mathbf {C} \mid \mathbf {X})}{P (\mathbf {C})} d \mathbf {X} d \mathbf {C} \\ \end{array}
$$
From the results, we can draw a conclusion that maximizing the IB objective  $I(\mathbf{C}, Y) - \lambda I(\mathbf{C}, \mathbf{X})$  is equivalent to minimizing the objective Eq.(9) in Theorem 4.3.