SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Henceforth, we will refer to the above estimate of causal effect as Direct. Often in practice, the backdoor adjustment variable  $X$  is high dimensional (e.g., vector encoding of text or images), and as a result, it becomes difficult to estimate the above quantity. Thus, several methods have been proposed to efficiently estimate and debias the causal effect for high dimensional data [6, 33]. In this paper, we use one of the recently proposed methods [6], henceforth denoted by Riesz, that uses insights from the Riesz representation theorem to create a multitasking neural network-based method to get a debiased estimate of the causal effect. Since this method uses a neural network to perform estimation, it is desirable for many data modalities such as text and image that require a neural network to get a good representation of the input. Let  $g(X,A) \coloneqq \mathbb{E}[Y|X,A]$ , then the estimator used by Riesz is given by:
$$
T E _ {A} = \mathbb {E} \left[ \alpha_ {0} (X, A) \cdot g (X, A) \right] \tag {6}
$$
where  $\alpha_0(X,A)$  is called a Riesz representative (RR), whose existence is guaranteed by the Riesz representation theorem. Lemma 3.1 in Chernozhukov et al. [6] states that in order to estimate  $g(X,A) = \mathbb{E}[Y|X,A]$ , it suffices to estimate  $g(X,A) = \mathbb{E}(Y|\alpha_0(X,A))$ . Since  $\alpha_0(X,A)$  is a scalar quantity, it is more efficient to estimate  $\mathbb{E}(Y|\alpha_0(X,A))$  than to condition on high dimensional  $(X,A)$ . Thus, Riesz uses a multi-tasking objective to learn both  $\alpha_0(X,A)$  and  $g(X,A)$  jointly. First, a neural network is used to encode the input  $(X,A)$  to a latent representation  $Z$ . Then, the architecture branches out into two heads; the first head trains the Riesz representative  $\alpha_0(X,A)$  and the second branch is used to train  $g(X,A)$  (see Figure 1 and Section 3 in Chernozhukov et al. [6] for details). Let the loss used to train the Riesz representative be called as  $RRloss$  and the loss used to train  $g(X,A)$  be called as REGloss and L2Reg be the 12 regularization loss (see Section 3 in [6] for the exact form of loss). Therefore, the final loss can be written as
$$
\operatorname {R i e s z L o s s} = \operatorname {R E G l o s s} + R _ {1} \cdot \operatorname {R R l o s s} + R _ {2} \cdot \operatorname {L 2 R e g L o s s}, \tag {7}
$$
with  $R_{1}$  and  $R_{2}$  as the regularization hyper parameters. Chernozhukov et al. [6] also suggests using another term in the overall loss (TMLEloss) which we do not consider in the current setup. After training the whole neural network jointly with the above loss objective, the causal effect estimate could be calculated using:
$$
\operatorname {R i e s z} := \mathbb {E} _ {X} [ g (X, A = 1) - g (X, A = 0) ] \tag {8}
$$
Chernozhukov et al. [6] also proposes another debiased measure of the causal effect given by:
$$
D e b i a s e d R i e s z := \mathbb {E} _ {X} \left[ (g (X, A = 1) - g (X, A = 0)) + \alpha (X, A) (Y - g (X, A)) \right] \tag {9}
$$
We observe that this measure gives a worse estimate than Riesz on Syn-Text dataset where the ground truth causal effect is known. Thus, in all our experiments we only use the Riesz estimator and leave the exploration of Debiased Riesz as future work. Please see §E.4 for the setup we use in our experiments to estimate the causal effect for different datasets.
A.2 Max Margin Objective
Taking inspiration from the description of the max-margin classifier from Kumar et al. [19] and Nagarajan et al. [26], we give a brief introduction to the max-margin training objective that is used to train the classifier we consider in Theorem 3.1. Consider an encoder  $h: X \to Z$  mapping
the input  $\pmb{x}$  to latent space representation  $\pmb{z}$ , and a classifier  $c: \mathbf{Z} \to Y$  that uses the latent space representation  $\mathbf{Z}$  to predict the task label  $y$  as  $c(\pmb{z}) = \pmb{w} \cdot \pmb{z}$ . The hyperplane  $c(\pmb{z}) = 0$  is called the decision boundary of the classifier. Let the task label be binary, taking values 1 or -1, without loss of generality (otherwise, the labels could be relabelled to conform to this notation). Then the points falling on one side of decision boundary  $c(\pmb{z} < 0$  are assigned one predicted label (say -1) and the ones falling on the other side the other predicted label (say +1). Then the distance of a point  $\pmb{z}$ , having task label  $y$ , from the decision boundary is given by:
$$
\mathcal {M} _ {c} (\boldsymbol {z}) = \frac {\boldsymbol {y} \cdot c (\boldsymbol {z})}{\| \boldsymbol {w} \| _ {2}} \tag {10}
$$
where  $\| w\| _2$  is the L2 norm of the classifier. The margin of a classifier is the distance of the closest latent representation  $z$  from the decision boundary. Thus, the goal of the max-margin objective is then to train a classifier that has the maximum margin. Equivalently, we want to minimize the following loss for training a classifier that has the largest margin.
$$
\mathcal {L} _ {m m} (c (Z), Y) = (- 1) \min  _ {i} \frac {y ^ {i} c \left(\boldsymbol {z} ^ {i}\right)}{\| c (\boldsymbol {z}) \| _ {2}} \tag {11}
$$
where the minimum is over all the training data points indexed by  $i$ .
B Proof of Prop 3.1
Proof. First Claim: To identify the causal effect of attribute  $A$  on the label  $Y$ , we need access to the interventional distribution  $P(Y|do(A))$ . For example, in the case when  $A$  is binary random variable the average causal effect of  $A$  on  $Y$  is given by  $TE_A = \mathbb{E}[Y|do(A) = 1] - \mathbb{E}[Y|do(A) = 0]$ . We can write  $P(Y|do(A))$  as:
$$
\begin{array}{l} P (Y | d o (A)) = \sum_ {X} \sum_ {U} P (Y, X, U | d o (A)) \\ = \sum_ {X} \sum_ {U} \left\{P (U) P (X | U, d o (A)) P (Y | X) \right\} \tag {12} \\ = \sum_ {X} P (Y | X) \left\{\sum_ {U} P (U) P (X | U, d o (A)) \right\} \\ = \sum_ {X} P (Y | X) P (X | d o (A)) \\ \end{array}
$$
Since we have access to the interventional distribution  $P(X|do(A))$  and access to observational distribution  $P(Y,X,A)$ , we can also estimate  $P(Y|do(A))$  using the above identity completing the first part of the proof.
Second Claim: We are given the attributes  $\mathcal{V} = \{C, S\}$ , but we do not know the distinction between the causal and spurious attribute beforehand. We will show that we can identify the interventional
distribution  $P(Y|do(A))$  where  $A \in \mathcal{V}$ . If  $A = C$ , then we have:
$$
\begin{array}{l} P (Y | d o (C)) = \sum_ {S} P (Y, S | d o (C)) \\ = \sum_ {S} P (S | d o (C)) P (Y | S, C) = \sum_ {S} P (S) P (Y | S, C) \tag {13} \\ = \sum_ {\mathcal {V} \backslash A} P (\mathcal {V} \backslash A) P (Y | \mathcal {V}) \\ \end{array}
$$
Then, when we have  $A = S$ , we have:
$$
\begin{array}{l} P (Y | d o (S)) = \sum_ {C} P (Y, C | d o (S)) = \sum_ {C} P (C | d o (S)) P (Y | C, d o (S)) \\ = \sum_ {C} P (C) P (Y | C, S) \quad \because (Y \perp S | C) \tag {14} \\ = \sum_ {\mathcal {V} \backslash A} P (\mathcal {V} \backslash A) P (Y | \mathcal {V}) \\ \end{array}
$$
Thus,  $P(Y|do(A)) = \sum_{\mathcal{V} \setminus A} P(\mathcal{V} \setminus A)P(Y|\mathcal{V})$  for all  $A \in \mathcal{V}$  and could be estimated from pure observational data.
C Failure of Mouli on DGP-2 and DGP3
Mouli and Ribeiro [25] define a particular DGP for their task and propose a score that identifies the invariant transformations. DGP-2 and DGP-3 in Fig. 2 adapt the causal graph taken in their work to our setting, where the unobserved variable associated with every transformation is replaced with an observed attribute. In addition, we add an additional level of complexity to the DGP by introducing the unobserved confounding variable  $U$  that introduces a spurious correlation between different attributes in DGP-2 and between an attribute and task label in DGP-3. The graph in DGP-1 is different from their setting and thus cannot use their method. This shows that the method proposed in their work doesn't generalize to different DGPs. Below, we show that their method will be able to identify the spurious attribute in the DGP-2 but fail to do so in DGP-3.
Corollary C.0.1. Let the observed input  $X$  be defined as the concatenation of the  $C, S$  and  $x$  in DGP-2 i.e.  $X = [C, S, x]$  and  $A$  and  $x$  in DGP-3 i.e.  $X = [A, x]$ . Then, Theorem 1 in Mouli and Ribeiro [25] will correctly identify the spurious attribute in DGP-2. For DGP-3, it will incorrectly claim that there is an edge between  $A$  and  $Y$  even if it is non-existent in the original graph.
Proof Sketch. For a attribute  $A$  to be spurious, the score proposed in Theorem 1 in Mouli and Ribeiro [25] requires conditional independence of the  $A$  with the task label  $Y$  given the rest of the observed attributes and core input feature  $x$ . For DGP-2, we show that both the spurious attribute  $S$  correctly satisfy that condition whereas the causal attribute  $C$  correctly doesn't satisfy satisfy the condition. Then for DGP-3, we show that even if the attribute  $A$  is spurious, it cannot be conditionally independent to  $Y$  given  $x$  due to unblocked path  $A \to U \to Y$  in the graph. See the complete proof below.
Proof. Theorem 1 in Mouli and Ribeiro [25] defines a score, henceforth called mouli's score, to identify whether there is an edge between any node  $N$  in the graph and the main task label  $Y$ . There is no edge between  $N$  and  $Y$  iff for all  $X$ :
$$
| P (Y | \Gamma_ {N} (X), U _ {Y}) - P (Y | X, U _ {Y}) | _ {T V} = 0 \tag {15}
$$
where  $\Gamma_N$  is the most-expressive representation that is invariant with respect to the node  $N$ .
Case 1 - DGP-2: For DGP-2 when the node  $N$  is causal  $(C)$ , we have  $X = [x, C, S], \Gamma_{C}(X) = [x, S]$  and  $U_{Y} = \phi$ . Thus, the mouli's score becomes:
$$
\left| P (Y \mid x, S) - P (Y \mid x, C, S) \right| _ {T V} \neq 0 \tag {16}
$$
since for at least one combination of  $x, C, S$  we have  $P(Y|x, S) \neq P(Y|x, C, S)$  since  $C \not\perp Y|S, x$  in DGP-2. Thus, mouli's score doesn't incorrectly mark causal node  $C$  as spurious. Next for a spurious node  $S$ , we have  $\Gamma_S(X) = [x, C]$ ; thus the mouli's score becomes:
$$
| P (Y | x, C) - P (Y | x, C, S) | _ {T V} = 0 \tag {17}
$$
for all  $x, C, S$  we have  $P(Y|x, C) = P(Y|x, C, S)$  since  $S \perp Y|C, x$ . Thus the mouli's score correctly identifies the spurious node  $S$ .
Case 2 - DGP-3: For DGP-3, when the node  $N = A$ , we have  $X = [x,A], \Gamma_A(X) = [x]$ , and if is spurious i.e. there is no edge from  $A$  to  $Y$  in the actual graph then the mouli's score becomes:
$$
\left| P (Y | x) - P (Y | x, A) \right| _ {T V} \neq 0 \tag {18}
$$
for all  $x, A$  since  $A \nsubseteq Y|x \Rightarrow P(Y|x) \neq P(Y|x, A)$  for at least one setting of  $(x, A, Y)$ . Thus the mouli's score will be non-zero and will fail to identify  $A$  as a spurious node.
We start by formalizing the assumption we made about the disentangled latent space in §3.3.
Assumption D.1 (Disentangled latent space). Let  $\mathcal{A} = \{ca_1, \dots, ca_K, sp_1, \dots, sp_J\}$  be the set of attribute given to us. The latent representation  $\mathbf{z}$  is disentangled and is of form  $[z_{ca_1}, \dots, z_{ca_K}, z_{sp_1}, \dots, z_{sp_J}]$ , where  $z_{ca_k} \in \mathbb{R}^{d_k}$  is the features corresponding to causal attribute "ca_k" and  $z_{sp_j} \in \mathbb{R}^{d_j}$  are the features corresponding to spurious attribute "sp_j". Here  $d_k$  and  $d_j$  are the dimensions of  $z_{ca_k}$  and  $z_{sp_j}$  respectively.
Now we are ready to formally state the main theorem and its proof that shows that it is sufficient to have the correct ranking of the causal effect of causal and spurious attributes to learn a classifier invariant to the spurious attributes using one instantiation of our regularization objective (3). Our proof goes in two steps:
1. First Claim: To prove that the desired classifier will be preferred by our loss objective over the classifier learned by the max-margin objective (henceforth denoted as undesired), we compare the loss incurred by both the classifier w.r.t to our training objective. Next, we show that given the ranking of the causal effect of causal and spurious attributes follows the relation mentioned in the theorem statement, there always exists a regularization strength when the desired classifier has a lower loss than the undesired one w.r.t to our training objective.
2. Second Claim: Using our first claim we show that the desired classifier will have a lower loss than any other classifier that uses the spurious attribute. Thus the desired classifier is preferred over any classifier that uses the spurious attribute using our loss objective. Then we show that among all the classifier that only uses the causal attribute for prediction, again our loss objective will select the desired classifier.
Theorem 3.1. Let the latent space be frozen and disentangled such that  $\mathbf{z} = [z_{ca_1},\ldots ,z_{ca_K},z_{sp_1},\ldots ,z_{sp_J}]$  (Assm D.1). Let the desired classifier  $c^{des}(\mathbf{z}) = \sum_{k = 1}^{K}\pmb{w}_{ca_k}^{des}\cdot \pmb{z}_{ca_k}$  be the max-margin classifier among all linear classifiers that use only the causal features  $z_{ca_k}$ 's for prediction. Let  $c^{mm}(z) = \sum_{k = 1}^{K}\pmb{w}_{ca_k}^{mm}\cdot \pmb{z}_{ca_k} + \sum_{j = 1}^{J}\pmb{w}_{sp_j}^{mm}\cdot \pmb{z}_{sp_j}$  be the max-margin classifier that uses both the causal and the spurious features, and let  $w_{sp_j}^{mm}\neq 0, \forall j\in [J]$ . We assume  $w_{sp_j}^{mm}\neq 0, \forall j\in [J]$ , without loss of generality because otherwise, we can restrict our attention only to those  $j\in [J]$  that have  $w_{sp_j}^{mm}\neq 0$ . Let the norm of the parameters of both the classifier be set to 1 i.e  $\sum_{k = 1}^{K}\| \pmb{w}_{ca_k}^{mm}\| _p^2 +\sum_{j = 1}^{J}\| \pmb{w}_{sp_j}^{mm}\| _p^2 = \sum_{k = 1}^{K}\| \pmb{w}_{ca_k}^{des}\| _p^2 = 1$ . Then if regularization coefficients are related s.t. mean  $\left(\left\{\frac{\lambda_{ca_k}}{\lambda_{sp_j}}\cdot \eta_{k,j}\right\}_{k\in [K],j\in [J]}\right) < \frac{J}{K}$  where  $\eta_{k,j} = \frac{\|\pmb{w}_{ca_k}^{des}\| - \|\pmb{w}_{ca_k}^{mm}\|_p}{\|\pmb{w}_{sp_j}^{mm}\|_p}$ , then
1. Preference:  $\mathcal{L}_{\text{AutoACER}}(c^{des}(z)) < \mathcal{L}_{\text{AutoACER}}(c^{mm}(z))$ . Thus, our causal effect regularization objective (Def 3) will choose the  $c^{des}(z)$  classifier over the max-margin classifier  $c^{mm}(z)$  which uses the spuriously correlated feature.
2. Global Optimum: The desired classifier  $c^{des}(z)$  is the global optimum of our loss function  $\mathcal{L}_{AutoACER}$  when  $J = 1$ ,  $K = 1$ ,  $p = 2$ , the regularization strength are related s.t.  $\lambda_{ca_1} < \lambda_{sp_1} \Rightarrow |\hat{T} E_{ca_1}| > |\hat{T} E_{sp_1}|$  and search space of linear classifiers  $c(z)$  are restricted to have the norm of parameters equal to 1.
Proof. Proof of Part 1: Since we are training our task classifier using the max-margin objective (Eq. 11) thus over training objective becomes:
$$
\mathcal {L} _ {\text {A u t o A C E R}} \left(c (h (\boldsymbol {x}))\right) := \mathcal {L} _ {m m} \left(c (h (\boldsymbol {x})), y\right) + R \cdot \sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \| \boldsymbol {w} _ {c a _ {k}} \| _ {p} + \sum_ {j = 1} ^ {J} \lambda_ {s p _ {j}} \| \boldsymbol {w} _ {s p _ {j}} \| _ {p} \tag {19}
$$
where  $\| \cdot \| _p$  is the  $L_{p}$  norm. For ease of exposition, we will denote  $c(h(\pmb {x}))$  as  $c(z)$  where  $\pmb{z}$  is the latent space representation of  $\pmb{x}$  given by  $\pmb {z} = h(\pmb {x})$ . Next, we will show that there always exists a regularization strength  $R\geq 0$  s.t.  $\mathcal{L}_{AutoACER}(c^{des}(\pmb {z})) < \mathcal{L}_{AutoACER}(c^{mm}(\pmb {z}))$  given mean  $\left(\left\{\frac{\lambda_{ca_k}}{\lambda_{sp_j}}\cdot \eta_{k,j}\right\}_{k\in [K],j\in [J]}\right) < \frac{J}{K}$  (from the main statement of Theorem 3.1). To have  $\mathcal{L}_{AutoACER}(c^{des}(\pmb {z})) < \mathcal{L}_{AutoACER}(c^{mm}(\pmb {z}))$  we need to select the regularization strength  $R\geq 0$  s.t:
$$
\begin{array}{l} \mathcal {L} _ {m m} \left(c ^ {d e s} (\boldsymbol {z}), y\right) + R \cdot \left\{\sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} \right\} <  \\ \mathcal {L} _ {m m} \left(c ^ {m m} (\boldsymbol {z}), y\right) + R \cdot \left\{\sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p} + \sum_ {j = 1} ^ {J} \lambda_ {s p _ {j}} \| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p} \right\} \tag {20} \\ \end{array}
$$
The max-margin objective (see §A.2 for background) given is given by  $\mathcal{L}_{mm}\big(c(z),y\big) = (-1)\cdot \min_i\left\{\frac{y^ic(z^i)}{\|c(z)\|_2}\right\}$  where  $z^{i}$  is the latent space representation of the input  $\pmb{x}^i$  with label  $y^{i}$  and  $\| c(z)\| _2$  is the L2-norm of the weight vector of the classifier  $c(z)$ . The norm of both the classifier  $c^{des}(z)$  and  $c^{mm}(z)$  is 1 (from the statement of this Theorem 3.1). Substituting the max-margin loss for both the classifier in Eq. 20 we get:
$$
\begin{array}{l} (- 1) \cdot \min  _ {i} \left\{y ^ {i} e ^ {d e s} \left(\boldsymbol {z} ^ {i}\right) \right\} + R \cdot \left\{\sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} \right\} <  \\ (- 1) \cdot \min  _ {j} \left\{y ^ {j} c ^ {m m} \left(\boldsymbol {z} ^ {j}\right) \right\} + R \cdot \left\{\sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p} + \sum_ {j = 1} ^ {J} \lambda_ {s p _ {j}} \| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p} \right\} \tag {21} \\ \end{array}
$$
Rearranging the above equation we get:
$$
\begin{array}{l} R \cdot \underbrace {\left\{\sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \left[ \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p} - \| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} \right] + \sum_ {j = 1} ^ {J} \lambda_ {s p _ {j}} \| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p} \right\}} _ {\text {L H S - t e r m 1}} > \tag {22} \\ \min _ {j} \left\{y ^ {j} c ^ {m m} (\boldsymbol {z} ^ {j}) \right\} - \min _ {i} \left\{y ^ {i} c ^ {d e s} (\boldsymbol {z} ^ {i}) \right\} \\ \end{array}
$$
If the LHS-term 1 in the above equation is greater than 0, then we can always select a regularization strength  $R > 0$  s.t. the above inequality is always satisfied, whenever
$$
R > \frac {\operatorname* {m i n} _ {j} \left\{y ^ {j} c ^ {m m} \left(\boldsymbol {z} ^ {j}\right) \right\} - \operatorname* {m i n} _ {i} \left\{y ^ {i} c ^ {d e s} \left(\boldsymbol {z} ^ {i}\right) \right\}}{\left\{\sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \left[ \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p} - \| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} \right] + \sum_ {j = 1} ^ {J} \lambda_ {s p _ {j}} \| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p} \right\}} \tag {23}
$$
Next we will show that given mean  $\left(\left\{\frac{\lambda_{ca_k}}{\lambda_{sp_j}}\cdot \eta_{k,j}\right\}_{k\in [K],j\in [J]}\right) < \frac{J}{K}$ , the LHS-term1 in Eq. 22 is always greater than 0. For LHS-term1 to be greater than 0 we need:
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \left[ \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p} - \| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} \right] > (- 1) \cdot \sum_ {j = 1} ^ {J} \lambda_ {s p _ {j}} \| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p} \\ \Longrightarrow \sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \left[ \| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} - \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p} \right] <   \sum_ {j = 1} ^ {J} \lambda_ {s p _ {j}} \| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p} \tag {24} \\ \end{array}
$$
Since  $\lambda_{(\cdot)} = 1 / |\hat{T}E_{(\cdot)}|$  and  $|\hat{T}E_{(\cdot)}| \in [0,1]$  we have  $\lambda_{(\cdot)} > 0$  (see §3.3 for discussion). From the main statement of this theorem we have  $\mathbf{w}_{sp_j}^{mm} \neq \mathbf{0} \Rightarrow \| \mathbf{w}_{sp_j}^{mm} \|_p > 0$  for all value of " $p$ ". Next, we have the following two cases:
Case 1  $\left(\sum_{k=1}^{K} \lambda_{ca_k}\left[\| \boldsymbol{w}_{ca_k}^{des}\|_p - \| \boldsymbol{w}_{ca_k}^{mm}\|_p\right] \leq 0\right)$ : For this case the above Eq. 24 is trivially satisfied since  $\sum_{j=1}^{J} \lambda_{sp_j} \| \boldsymbol{w}_{sp_j}^{mm}\|_p > 0$  as  $\forall j \in [J]$ ,  $\lambda_{sp_j} > 0$  and  $\| \boldsymbol{w}_{sp_j}^{mm}\|_p > 0$ .
Case 2  $\left(\sum_{k=1}^{K} \lambda_{ca_k}\left[\|\boldsymbol{w}_{ca_k}^{des}\|_p - \|\boldsymbol{w}_{ca_k}^{mm}\|_p\right] > 0\right)$ : Since for all  $j \in [J]$  we have  $\lambda_{sp_j} > 0$  and  $\|\boldsymbol{w}_{sp_j}^{mm}\|_p > 0$ , rearranging we get:
$$
\sum_ {j = 1} ^ {J} \frac {\lambda_ {s p _ {j}} \| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p}}{\sum_ {k = 1} ^ {K} \lambda_ {c a _ {k}} \left[ \| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} - \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p} \right]} > 1
$$
$$
\sum_ {j = 1} ^ {J} \frac {1}{\sum_ {k = 1} ^ {K} \left(\frac {\lambda_ {c a _ {k}}}{\lambda_ {s p _ {j}}}\right) \left(\frac {\| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} - \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p}}{\| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p}}\right)} > 1 \tag {25}
$$
$$
\frac {1}{\sum_ {j = 1} ^ {J} \left\{\underbrace {\frac {1}{\sum_ {k = 1} ^ {K} \left(\frac {\lambda_ {c a _ {k}}}{\lambda_ {s p _ {j}}}\right) \left(\frac {\| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} - \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p}}{\| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p}}\right)} _ {\text {L H S - t e r m 3}} \right\}} \cdot J <   J
$$
The above equation is the harmonic mean of the LHS-term 3 and LHS-term 3 is  $>0$  for all values of  $j$ . We know that the harmonic mean is always less than or equal to the arithmetic mean for a set of positive numbers (Inequalities book by GH Hardy [34]). Thus the above inequality is satisfied if the following inequality is satisfied:
$$
\sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {K} \frac {\left(\frac {\lambda_ {c a _ {k}}}{\lambda_ {s p _ {j}}}\right) \left(\frac {\| \boldsymbol {w} _ {c a _ {k}} ^ {d e s} \| _ {p} - \| \boldsymbol {w} _ {c a _ {k}} ^ {m m} \| _ {p}}{\| \boldsymbol {w} _ {s p _ {j}} ^ {m m} \| _ {p}}\right)}{J \cdot K} <   \frac {J}{K} \tag {26}
$$
Since the above condition on regularization strength of features is satisfied (given in the main statement of this theorem), we have  $\mathcal{L}_{AutoACER}(c^{des}(\boldsymbol{z})) < \mathcal{L}_{AutoACER}(c^{mm}(\boldsymbol{z}))$  thus completing the proof. The above Eq. 26 states that the mean of  $\left(\frac{\lambda_{ca_k}}{\lambda_{sp_j}}\right)\left(\frac{\|\boldsymbol{w}_{ca_k}^{des}\|_p - \|\boldsymbol{w}_{ca_k}^{mm}\|_p}{\|\boldsymbol{w}_{sp_j}^{mm}\|_p}\right)$  should be less than  $J / K$ . Thus the above equation is also satisfied if individually all the terms considered when taking the mean are individually less than  $J / K$ . Thus, a stricter but more intuitive condition on  $\lambda$ 's s.t Eq. 26 is satisfied is given by:
$$
\left(\frac {\lambda_ {c a _ {k}}}{\lambda_ {s p _ {j}}}\right) \eta_ {k, j} <   \frac {J}{K} \tag {27}
$$
$$
\hat {T E} _ {s p _ {j}} \left(\frac {K}{J} \eta_ {k, j}\right) <   \hat {T E} _ {c a _ {k}}
$$
where  $\eta_{k,j} = \big(\frac{\|\pmb{w}_{ca_k}^{des}\|_p - \|\pmb{w}_{ca_k}^{mm}\|_p}{\|\pmb{w}_{sp_j}^{m m}\|_p}\big)$
Proof of Part 2: Given  $p = 2$  and since  $K = 1$  and  $J = 1$ , the desired classifier takes form  $c^{des}(z) = w_{ca,1}^{des} \cdot z_{ca,1} = w_{ca}^{des} \cdot z_{ca}$ . For ease of exposition, we will drop "1" from the subscript which denotes
Table 2: Conditional Probability Distribution for Syn-Text dataset ( $P(Y|\text{casual, confound})$ ): The rows represent the different settings of the parent attributes—causal and confound and the columns represent different values the task label  $Y$  can take. Each cell represents the probability of observing the task label given the parent's setting.
Y = 0	Y = 1
causal=0, confound=0	0.99	0.01
causal=0, confound=1	0.30	0.70
causal=1, confound=0	0.70	0.30
causal=1, confound=1	0.01	0.99