SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 1. The projection of  $\mathbb{E}[\pmb{x}_{d}^{\top}\pmb{v}_{i}^{d}]$  on each basis corresponding to feature is non zero, i.e.
$$
\left| \mathbb {E} ^ {e} \left[ \boldsymbol {x} _ {\hat {d}} ^ {\intercal} \boldsymbol {v} _ {i} ^ {\hat {d}} \right] \right| > 0, \forall e \in \left\{e _ {\mathrm {I D}}, e _ {\mathrm {O O D}} \right\}, i \in [ d ].
$$
This ensures that coefficients of a feature can not be always 0, otherwise we can simply remove it.
Proof. Let  $\beta^{\mathrm{ID}}$  and  $\beta^{\mathrm{OOD}}$  denote the solution of linear regression in the ID and OOD domains, i.e.,
$$
\beta_ {\hat {d}} ^ {\mathrm {I D}} = \underset {\beta} {\arg \min } \mathbb {E} _ {\beta} ^ {\mathrm {I D}} (y - \boldsymbol {x} _ {\hat {d}} ^ {\top} \beta) ^ {2} \tag {2}
$$
$$
\beta_ {\hat {d}} ^ {\mathrm {O O D}} = \underset {\beta} {\arg \min } \mathbb {E} ^ {\mathrm {O O D}} (y - \boldsymbol {x} _ {\hat {d}} ^ {\top} \beta) ^ {2} \tag {3}
$$
Now let us compare the OOD loss after we include  $\boldsymbol{x}_{\mathrm{spu},\hat{d}_{\mathrm{spu}} + 1}$ . In practice, we can only obtain  $\beta^{\mathrm{ID}}$  and then apply it on both the ID and OOD domains, which elicits the following errors:
$$
\mathcal {L} _ {\mathrm {I D}} \left(\Phi_ {\hat {d}}\right) = \mathbb {E} ^ {\mathrm {I D}} \left(y - \boldsymbol {x} _ {\hat {d}} ^ {\top} \beta^ {\mathrm {I D}}\right) \tag {4}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {O O D}} \left(\Phi_ {\hat {d}}\right) = \mathbb {E} ^ {\mathrm {O O D}} \left(y - \boldsymbol {x} _ {\hat {d}} ^ {\top} \beta^ {\mathrm {I D}}\right) \\ = \underbrace {\mathbb {E} ^ {\text {O O D}} \left(y - \boldsymbol {x} _ {\hat {d}} ^ {\top} \beta^ {\mathrm {I D}}\right) - \mathbb {E} ^ {\text {O O D}} \left(y - \boldsymbol {x} _ {\hat {d}} ^ {\top} \beta^ {\text {O O D}}\right)} _ {\xi_ {1} ^ {\hat {d}}} + \underbrace {\mathbb {E} ^ {\text {O O D}} \left(y - \boldsymbol {x} _ {\hat {d}} ^ {\top} \beta^ {\text {O O D}}\right)} _ {\xi_ {2} ^ {\hat {d}}} \tag {5} \\ \end{array}
$$
It is well known that the residual of the linear fitting  $y$  by  $\pmb{x}_{d}$  on the ID domain is
$$
\mathcal {L} _ {\mathrm {I D}} \left(\Phi_ {\hat {d}}\right) = \mathbb {E} ^ {\mathrm {I D}} \left[ y - \boldsymbol {x} _ {\hat {d}} \mathbb {E} ^ {\mathrm {I D}} \left[ \boldsymbol {x} _ {\hat {d}} ^ {\top} \boldsymbol {x} _ {\hat {d}} \right] ^ {- 1} \mathbb {E} ^ {\mathrm {I D}} \left[ \boldsymbol {x} _ {\hat {d}} y \right] \right] ^ {2} = \mathbb {E} ^ {\mathrm {I D}} \left[ y - \Phi_ {\hat {d}} (\boldsymbol {x}) ^ {\top} \beta_ {\hat {d}} ^ {\mathrm {I D}} \right] ^ {2}, \tag {6}
$$
$$
\mathcal {L} _ {\mathrm {I D}} \left(\Phi_ {\hat {d} + 1}\right) = \mathbb {E} ^ {\mathrm {I D}} \left[ y - \boldsymbol {x} _ {\hat {d} + 1} ^ {\top} \beta_ {\hat {d} + 1} ^ {\mathrm {I D}} \right] ^ {2}. \tag {7}
$$
Since  $\pmb{x}_{\mathrm{spu},\hat{d}_{\mathrm{spu} + 1}}$  does not lie in the space spanned by  $\pmb{x}_{\hat{d}}$ , so the space spanned by  $\pmb{x}_{\hat{d} +1}$  is strictly larger than  $\pmb{x}_{\hat{d}}$ .
Together with Assumption 1, we have
$$
\mathcal {L} _ {\mathrm {I D}} \left(\Phi_ {\hat {d}}\right) - \mathcal {L} _ {\mathrm {I D}} \left(\Phi_ {\hat {d} + 1}\right) = \mathbb {E} ^ {\mathrm {I D}} \left[ y - \boldsymbol {x} _ {\hat {d}} ^ {\top} \beta_ {\hat {d}} ^ {\mathrm {I D}} \right] ^ {2} - \mathbb {E} ^ {\mathrm {I D}} \left[ y - \boldsymbol {x} _ {\hat {d} + 1} ^ {\top} \beta_ {\hat {d} + 1} ^ {\mathrm {I D}} \right] ^ {2} > 0, \tag {8}
$$
$$
\xi_ {2} ^ {\hat {d}} - \xi_ {2} ^ {\hat {d} + 1} = \mathbb {E} ^ {\mathrm {O O D}} [ y - \boldsymbol {x} _ {\hat {d}} ^ {\top} \beta_ {\hat {d}} ^ {\mathrm {O O D}} ] ^ {2} - \mathbb {E} ^ {\mathrm {O O D}} [ y - \boldsymbol {x} _ {\hat {d} + 1} ^ {\top} \beta_ {\hat {d} + 1} ^ {\mathrm {O O D}} ] ^ {2} > 0. \tag {9}
$$
By the proof in Appendix B.6.3 (above Eq. 29) in [61], we have
$$
\xi_ {1} ^ {\hat {d}} = \sum_ {i} ^ {\hat {d}} \left(\mathbb {E} ^ {\mathrm {O O D}} \left[ \boldsymbol {x} _ {\hat {d}} ^ {\top} y \right] ^ {\top} \boldsymbol {v} _ {i} ^ {\mathrm {O O D}, \hat {d}}\right) ^ {2} \lambda_ {i} ^ {\mathrm {O O D}} \left(\frac {1}{\lambda_ {i} ^ {\mathrm {I I D} , \hat {d}}} - \frac {1}{\lambda_ {i} ^ {\mathrm {O O D} , \hat {d}}}\right) ^ {2}. \tag {10}
$$
By Eq. (20) in [61], we have
$$
\lambda_ {i} ^ {\mathrm {I I D}, \hat {d}} - \lambda_ {i} ^ {\mathrm {O O D}, \hat {d}} = \left(\alpha_ {i} ^ {\mathrm {I I D}}\right) ^ {2} - \left(\alpha_ {i} ^ {\mathrm {O O D}}\right) ^ {2}. \tag {11}
$$
$$
\begin{array}{l} \xi_ {1} ^ {\hat {d} + 1} - \xi_ {1} ^ {\hat {d} + 1} = \sum_ {i = 1} ^ {\hat {d}} \left[ \left(\mathbb {E} ^ {\mathrm {O O D}} [ \pmb {x} _ {\hat {d}} y ] ^ {\top} \pmb {v} _ {i} ^ {\mathrm {O O D}, \hat {d}}\right) ^ {2} \left(\lambda_ {i} ^ {\mathrm {O O D}, \hat {d}}\right) \left(\frac {1}{\lambda_ {i} ^ {\mathrm {I D} , \hat {d}}} - \frac {1}{\lambda_ {i} ^ {\mathrm {O O D} , \hat {d}}}\right) ^ {2} \right. \\ \left. - \left(\mathbb {E} ^ {\mathrm {O O D}} \left[ \boldsymbol {x} _ {\hat {d} + 1} y \right] ^ {\top} \boldsymbol {v} _ {i} ^ {\mathrm {O O D}, \hat {d} + 1}\right) ^ {2} \left(\lambda_ {i} ^ {\mathrm {O O D}, \hat {d} + 1}\right) \left(\frac {1}{\lambda_ {i} ^ {\mathrm {I D} , \hat {d} + 1}} - \frac {1}{\lambda_ {i} ^ {\mathrm {O O D} , \hat {d} + 1}}\right) ^ {2} \right] \\ + \left(\mathbb {E} ^ {\mathrm {O O D}} \left[ \boldsymbol {x} _ {\hat {d} + 1} y \right] ^ {\top} \boldsymbol {v} _ {\hat {d} + 1} ^ {\mathrm {O O D}, \hat {d} + 1}\right) ^ {2} \frac {\left(\left(\alpha_ {\hat {d} + 1} ^ {\mathrm {I D}}\right) ^ {2} - \left(\alpha_ {\hat {d} + 1} ^ {\mathrm {O O D}}\right) ^ {2}\right) ^ {2}}{\left(\lambda_ {\hat {d} + 1} ^ {\mathrm {I D}, \hat {d} + 1}\right) ^ {2} \lambda_ {\hat {d} + 1} ^ {\mathrm {O O D}, \hat {d} + 1}}. \tag {12} \\ \end{array}
$$
From Eq. 10 and 12, we have the desired result.