SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem C.2. Let  $\mathrm{X}$  and  $\mathrm{Z}$  be defined as above. Then every linear predictor  $f(\mathbf{x}) = \mathbf{b} + \mathbf{W}\mathbf{x}$  exhibits statistical parity w.r.t.  $\mathrm{Z}$  when evaluated on  $\mathrm{X}$  if and only if each class-conditional mean  $\mathbb{E}\big[\mathrm{X}|\mathrm{Z} = z\big]$  is equal to  $\mathbb{E}\big[\mathrm{X}\big]$ .
Proof. Suppose each class-conditional mean  $\mathbb{E}\big[\mathrm{X}|\mathrm{Z} = z\big]$  is equal to  $\mathbb{E}\big[\mathrm{X}\big]$ . Then by the linearity of expectation, we have for all  $z \in \mathcal{Z}$ :
$$
\mathbb {E} [ f (\mathrm {X}) | \mathrm {Z} = z ] = \mathbb {E} [ \mathbf {W X} + \mathbf {b} | \mathrm {Z} = z ] = \mathbf {W E} [ \mathrm {X} | \mathrm {Z} = z ] + \mathbf {b} = \mathbf {W E} [ \mathrm {X} ] + \mathbf {b} = \mathbb {E} [ f (\mathrm {X}) ].
$$
This matches the definition of statistical parity provided in Definition C.1.
Conversely, suppose every linear predictor  $f(\mathbf{x}) = \mathbf{b} + \mathbf{W}\mathbf{x}$  exhibits statistical parity w.r.t. Z when evaluated on X. Then this holds for the identity function  $\mathrm{id}(\mathbf{x}) = \mathbf{x}$ , and thus for all  $z \in \mathcal{Z}$ :
$$
\mathbb {E} [ \mathrm {X} | \mathrm {Z} = z ] = \mathbb {E} [ \mathrm {i d} (\mathrm {X}) | \mathrm {Z} = z ] = \mathbb {E} [ \mathrm {i d} (\mathrm {X}) ] = \mathbb {E} [ \mathrm {X} ].
$$
D Implications for Prior Work
In this section we discuss the implications of Theorem 4.1, which characterizes the necessary and sufficient conditions for an affine erasure function to yield a perfectly linearly guarded dataset, for methods proposed in prior work.
Spectral Attribute RemovaL (SAL) [37] uses the top  $n$  left singular vectors of  $\Sigma_{\mathrm{XZ}}$  to construct an orthogonal projection matrix  $\mathbf{Q}_{\mathrm{SAL}} = \mathbf{I} - \mathbf{U}_{:n}\mathbf{U}_{:n}^T$  which is then applied to X. Notably, while  $n$  is presented as a free parameter in their method, all of their experiments involve binary classification problems where Z is a one-hot vector, and  $n$  is set to a value no greater than 2. We'll call the version of SAL where  $n = \mathrm{rank}(\Sigma_{\mathrm{XZ}})$ , "full-rank SAL." Since these left singular vectors are an orthonormal basis for  $\Sigma_{\mathrm{XZ}}$ 's column space, Theorem 4.1 implies that full-rank SAL guarantees linear guardedness.
Mean Projection (MP) [17] orthogonally projects  $\mathrm{X}$  onto the orthogonal complement of the span of the difference in class centroids  $\mathbb{E}[\mathrm{X}|\mathrm{Z} = 1] - \mathbb{E}[\mathrm{X}|\mathrm{Z} = 0]$ , where  $\mathrm{Z}$  is assumed to be binary. Since the centroids are equal after the projection, this method guarantees linear guardedness by Theorem 3.1. In fact, by Theorem 3.4, MP is mathematically equivalent to SAL when  $\mathrm{Z}$  is a one-dimensional random vector taking one of two possible values.
E Derivation of LEACE
Below are two independent proofs of Theorem 4.2.
E.1 Algebraic Proof
Proof. We shall first show that, in any orthonormal basis, $^7$  each row  $\mathbf{P_i}$  constitutes an independent optimization problem, and then select a basis in which we can easily show that the corresponding component  $X_{i}$  of  $X$  can be almost surely decomposed into a linear combination of mutually uncorrelated components in the whitened random vector  $\mathbf{WX}$ , some of which correlate with  $Z$  and some of which do not. The solution  $(\mathbf{PX})_i$  is then that same linear combination, restricted to those components which do not correlate with  $Z$ .
Consider first an orthonormal basis diagonalizing the inner product  $\mathbf{M}$ , so that  $\langle \mathbf{x},\mathbf{y}\rangle_{\mathbf{M}} = \sum_{i = 1}^{d}\alpha_{i}x_{i}y_{i}$  for fixed  $\alpha_{1},\ldots ,\alpha_{d}\geq 0$ . This allows us to treat each row  $\mathbf{P_i}\in \mathbb{R}^d$  of  $\mathbf{P}$  as a separate optimization problem,
$$
\underset {\mathbf {P _ {i} \in \mathbb {R} ^ {d}}} {\operatorname {a r g m i n}} \mathbb {E} \Big [ \alpha_ {i} \big (\mathbf {P _ {i}} ^ {T} \mathrm {X} - \mathrm {X} _ {i} \big) ^ {2} \Big ] \quad \text {s u b j e c t t o} \operatorname {C o v} (\mathbf {P _ {i}} ^ {T} \mathrm {X}, \mathrm {Z}) = \mathbf {0},
$$
at which point the weights  $\alpha_{i}$  of each subproblem become irrelevant, and our objective may as well be Euclidean, allowing us to view each row as an independent optimization problem not just in this basis, but from any convenient one.
So now let  $\ell = \mathrm{rank}(\pmb{\Sigma}_{\mathrm{XZ}}) = \mathrm{rank}(\pmb{\Sigma}_{\mathbf{WX},\mathbf{Z}})$  and  $m = \mathrm{rank}(\pmb{\Sigma}_{\mathrm{XX}}) = \mathrm{rank}(\pmb{\Sigma}_{\mathbf{WX},\mathbf{WX}})$ , and consider a (new) orthonormal basis whose first  $m$  coordinates span the column (and row) space of  $\mathbf{W}$  (i.e. the subspace of  $\mathbb{R}^d$  in which  $\mathbf{X}$  and  $\mathbf{WX}$  have nonzero variance), and whose first  $\ell \leq m$  coordinates span the column space of  $\pmb{\Sigma}_{\mathbf{WX},\mathbf{Z}}$  (i.e. the subspace of  $\mathbb{R}^d$  in which  $\mathbf{WX}$  has nonzero covariance with  $\mathbf{Z}$ ).
Any component of X can be (almost surely) written as a fixed linear combination of the nontrivial components of its whitening WX:
$$
\mathrm {X} _ {i} = (\mathbf {W} ^ {+} \mathbf {W X}) _ {i} = \sum_ {j = 1} ^ {m} W _ {i j} ^ {+} (\mathbf {W X}) _ {j}. \quad \text {(a l m o s t s u r e l y)}
$$
Meanwhile, any component of  $\mathbf{PX}$  can be (always) written as a fixed linear combination of the nontrivial components of  $\mathbf{WX}$  and the almost surely zero components of  $\mathbf{X}$ :
$$
(\mathbf {P X}) _ {i} = \sum_ {j = 1} ^ {m} A _ {i j} (\mathbf {W X}) _ {j} + \sum_ {j = m + 1} ^ {d} B _ {i j} \mathrm {X} _ {j},
$$
i.e.  $\mathbf{P} = \mathbf{A}\mathbf{W} + \mathbf{B}\mathbf{V}$ , where  $\mathbf{V} = \mathbf{I} - \mathbf{W}^{+}\mathbf{W}$  is the orthogonal projection onto X's almost surely zero components.
7Throughout this proof, we abuse the notations  $\mathbf{X}_i,\mathbf{P_i}$  , etc. to refer to the  $i^{\mathrm{th}}$  component in the specified basis, not necessarily the standard one.
The  $i^{\mathrm{th}}$  sub-objective is then:
$$
\mathbb {E} \big (\mathbf {P _ {i}} ^ {T} \mathbf {X} - \mathbf {X} _ {i} \big) ^ {2} = \mathbb {E} \bigg [ \sum_ {j = 1} ^ {m} (A _ {i j} - W _ {i j} ^ {+}) (\mathbf {W X}) _ {j} \bigg ] ^ {2} = \sum_ {j = 1} ^ {m} (A _ {i j} - W _ {i j} ^ {+}) ^ {2},
$$
where we have safely ignored the almost surely zero terms  $B_{ij}X_j$  ( $j > m$ ), and used the fact that the first  $m$  components of  $\mathbf{WX}$  have identity covariance matrix.
$\mathbf{PX}$  is almost surely equal to  $\mathbf{AWX}$ , so our constraint  $\mathrm{Cov}(\mathbf{PX},\mathbf{Z}) = \mathbf{0}$  is equivalent to  $\mathbf{A}\Sigma_{\mathbf{WX},\mathbf{Z}} = \mathrm{Cov}(\mathbf{AWX},\mathbf{Z}) = \mathbf{0}$ , i.e.  $A_{ij} = 0$  when  $j\leq \ell$ , since the first  $\ell$  components are those for which  $\mathbf{WX}$  correlates with  $\mathbf{Z}$ . Subject to this, the objective is minimized for  $A_{ij} = W_{ij}^{+}$  when  $j > \ell$ , i.e.  $\mathbf{A} = \mathbf{W}^{+}(\mathbf{I} - \mathbf{P}_{\mathbf{W}\Sigma_{\mathbf{XZ}}})$ .
The particular choice  $\mathbf{B} = \mathbf{I}$  gives our solution  $\mathbf{P}^{*} = \mathbf{I} - \mathbf{W}^{+}\mathbf{P}_{\mathbf{W}\Sigma_{\mathrm{xZ}}}\mathbf{W}$ , leaving the non-varying components of X intact (see Fig. 1 for a visualization).
The solution is unique except for columns corresponding to the components of  $X$  with zero variance, and rows corresponding to the zero-weighted components of the (pseudo) inner product  $M$ .
E.2 Covector Proof
Proof. We assume without loss of generality that vectors in  $\mathbb{R}^d$  are represented in a basis diagonalizing the inner product  $\mathbf{M}$ , so that  $\langle \mathbf{x},\mathbf{y}\rangle_{\mathbf{M}} = \sum_{i = 1}^{d}m_{i}x_{i}y_{i}$  for fixed  $m_1,\ldots ,m_d\geq 0$ . This allows us to treat each row  $\mathbf{P_i}\in \mathbb{R}^d$  of  $\mathbf{P}$  as a separate optimization problem,
$$
\underset {\mathbf {P _ {i} \in \mathbb {R} ^ {d}}} {\operatorname {a r g m i n}} \mathbb {E} \Big [ m _ {i} \big (\mathbf {P _ {i} ^ {T} X - X _ {i}} \big) ^ {2} \Big ] \quad \mathrm {s u b j e c t t o} \operatorname {C o v} (\mathbf {P _ {i} ^ {T} X}, Z) = \mathbf {0}.
$$
Our objective only depends on  $\mathbf{P_i}$  through its effect on the scalar random variable  $\xi = \mathbf{P_i}^T\mathbf{X}$ . All random variables of the form  $\zeta = \mathbf{u}_{\zeta}^{T}\mathbf{X}$  for some covector  $\mathbf{u}_{\zeta}^{T}\in \mathbb{R}^{d}$  form a vector space  $U$ , which we equip with the covariance inner product  $\langle \xi ,\zeta \rangle_{\mathrm{Cov}} = \mathrm{Cov}(\xi ,\zeta) = \mathbb{E}[\xi \zeta ] = \mathbf{u}_{\xi}^{T}\pmb{\Sigma}_{\mathrm{XX}}\mathbf{u}_{\zeta}$ .
By the linearity of covariance, the elements of  $U$  uncorrelated with  $Z$  form a subspace  $Z^{\perp} \subseteq U$ . Note also that  $\xi \in Z^{\perp}$  if and only if  $\xi$ 's covector  $\mathbf{u}_{\xi}^{T}$  satisfies  $\mathrm{Cov}(\mathbf{u}_{\xi}^{T}\mathrm{X}, Z) = \mathbf{u}_{\xi}^{T}\boldsymbol{\Sigma}_{\mathrm{XZ}} = \mathbf{0}_{k}$ , and that these covectors themselves form the subspace  $\operatorname{colsp}(\boldsymbol{\Sigma}_{\mathrm{XZ}})^{\perp}$  of  $\mathbb{R}^{d}$ .
Our objective now reduces to finding a covector  $\mathbf{P}_i^T$  that defines the orthogonal projection of  $\mathrm{X}_i$  onto  $Z^\perp$ . The difficulty is that orthogonality of elements in  $U$  is not equivalent to orthogonality of the corresponding covectors. We can fix this by changing the basis in which covectors are represented. Since  $\mathrm{X} \in \operatorname{colsp}(\mathbf{W})$  a.s., we can write any element of  $U$  as a linear form in  $\mathbf{W}\mathbf{X}$  rather than  $\mathrm{X}$  by applying the change-of-basis  $\mathbf{u}_{\xi}' = \mathbf{W}^{+}\mathbf{u}_{\xi}$  to every covector:  $\xi = (\mathbf{u}_{\xi}')^T\mathbf{W}\mathbf{X} = \mathbf{u}_{\xi}'^T\mathbf{W}^{+}\mathbf{W}\mathbf{X}$  a.s.
In this new basis, which is orthonormal under our covariance inner product, each component of  $\mathbf{X}$  is written  $\mathbf{X}_i = (\mathbf{W}^+)_i^T\mathbf{W}\mathbf{X}$  and the inner product of any two elements of  $U$  is simply the Euclidean inner product of the corresponding covectors:
$$
\langle \xi , \zeta \rangle_ {\mathrm {C o v}} = \operatorname {C o v} \left(\mathbf {u} _ {\xi} ^ {\prime T} \mathbf {W X}, \mathbf {u} _ {\zeta} ^ {\prime T} \mathbf {W X}\right) = \mathbf {u} _ {\xi} ^ {\prime T} \mathbf {W} \boldsymbol {\Sigma} _ {\mathrm {X X}} \mathbf {W} \mathbf {u} _ {\zeta} ^ {\prime} = \mathbf {u} _ {\xi} ^ {\prime T} \mathbf {u} _ {\zeta} ^ {\prime}.
$$
Since the two inner products are now equivalent, and  $Z^{\perp}$  is precisely those random variables with covector  $\mathbf{u}' \in \mathrm{colsp}(\mathbf{W}\boldsymbol{\Sigma}_{\mathrm{XZ}})^{\perp}$ , the orthogonal projection of  $\mathrm{X}_i$  onto  $Z^{\perp}$  is also an orthogonal projection of its covector  $(\mathbf{W}^{+})_{i}^{T}$  onto  $\mathrm{colsp}(\mathbf{W}\boldsymbol{\Sigma}_{\mathrm{XZ}})^{\perp}$ :
$$
\hat {\mathbf {X}} _ {i} = \left(\mathbf {W} ^ {+}\right) _ {i} ^ {T} \left(\mathbf {I} - \mathbf {P} _ {\mathbf {W} \boldsymbol {\Sigma} _ {\mathrm {X Z}}}\right) (\mathbf {W X}) \tag {6}
$$
Putting all the components of  $\mathrm{X}$  together, we have our final solution,
$$
\hat {\mathbf {X}} = (\mathbf {I} - \mathbf {W} ^ {+} \mathbf {P} _ {\mathbf {W} \boldsymbol {\Sigma} _ {\mathrm {X Z}}} \mathbf {W}) \mathbf {X},
$$
which is almost surely equivalent to Eq. 6, but keeps the non-varying components of X intact.
8Strictly speaking, equivalence classes of almost surely equal random variables.
If  $\Sigma_{\mathrm{XX}}$  is full rank, there is a one-to-one correspondence between random variables in  $U$  and covectors. In the singular case, we may choose the component of the covector inside  $\ker (\Sigma_{\mathrm{XX}})$  arbitrarily, since it will make no difference to the inner product.
F The Optimality of Oblique Projections
As noted in subsection 4.2, the optimal affine erasure function  $r(\mathbf{x}) = \mathbf{b} + \mathbf{P}\mathbf{x}$  does not in general use an orthogonal projection for the matrix  $\mathbf{P}$ . A simple example illustrates why. Let  $d = 2$ ,  $k = 1$  so that  $\mathbf{X}$  takes values in  $\mathbb{R}^2$  and  $\mathbf{Z}$  takes values in  $\mathbb{R}$ , with the first feature  $\mathrm{X}_1$  and the label  $\mathbf{Z}$  each independently and uniformly distributed in  $\{-1, +1\}$ , and the second feature  $\mathrm{X}_2$  simply equal to the sum  $\mathrm{X}_2 = \mathrm{X}_1 + \mathrm{Z}$ . A dataset reflecting such a distribution has four  $(\mathbf{x}, \mathbf{y})$  pairs:
$$
\left([ 1, 2 ] ^ {T}, 1\right), \quad \left([ 1, 0 ] ^ {T}, - 1\right), \quad \left([ - 1, 0 ] ^ {T}, 1\right), \quad \left([ - 1, - 2 ] ^ {T}, - 1\right)
$$
In this case, all of the information  $\mathrm{X}$  has about  $\mathrm{Z}$  resides in  $\mathrm{X}_2$ , so the minimally disruptive orthogonal projection which guards  $\mathrm{Z}$  will nullify that component:
$$
\mathbf {P} _ {\text {o r t h o}} = \left[ \begin{array}{c c} 1 & 0 \\ 0 & 0 \end{array} \right]
$$
On the other hand,  $\mathrm{X}_1$  contains some information about  $\mathrm{X}_2$  (despite having no information about  $Z$ ), allowing a partial reconstruction of  $\mathrm{X}_2$  while preserving full concept erasure:
$$
\mathbf {P} _ {\text {o b l i q u e}} = \left[ \begin{array}{c c} 1 & 0 \\ 1 & 0 \end{array} \right]
$$
Both methods fully erase the ability to predict  $Z$  from the data, however a simple calculation shows the second, oblique method to perform better as measured by mean squared edit distance:
$$
\mathbb {E} \| \mathbf {P} _ {\text {o r t h o}} \mathrm {X} - \mathrm {X} \| ^ {2} = 2, \quad \mathbb {E} \| \mathbf {P} _ {\text {o b l i q u e}} \mathrm {X} - \mathrm {X} \| ^ {2} = 1
$$
G Equivalence of Guardedness Definitions
Xu et al. [43] define the conditional  $\nu$ -entropy of  $Z$  given  $X$  as the lowest achievable cross-entropy loss predicting  $Z$  with a function of  $X$  in the predictor class  $\nu$ . In our notation:
$$
H _ {\mathcal {V}} (\mathrm {Z} \mid \mathrm {X}) = \inf  _ {\theta \in \Theta} \mathbb {E} [ \mathcal {L} (\eta (\mathrm {X}; \boldsymbol {\theta}), \mathrm {Z}) ],
$$
where  $\mathcal{L}(\eta, z) = -\log \frac{\exp(\eta_z)}{\sum_{i=1}^{k} \exp(\eta_i)}$  is the cross-entropy loss function.
They then define the (unconditional)  $\nu$ -entropy  $H_{\mathcal{V}}(Z) = H_{\mathcal{V}}(Z \mid \mathbf{0})$  to be the lowest achievable cross-entropy loss in the case of a constantly null random data variable. This is exactly our trivially attainable loss  $L_{\tau}$  (Definition 2.2).
Finally, they define the  $\nu$ -information from X to Z as the reduction in  $\nu$ -entropy as compared to using such a null random data variable:
$$
I _ {\mathcal {V}} (\mathrm {X} \to \mathrm {Z}) = H _ {\mathcal {V}} (\mathrm {Z}) - H _ {\mathcal {V}} (\mathrm {Z} \mid \mathrm {X}).
$$
Using these notions, Ravfogel et al. [31] say that  $\mathrm{X}$  is  $\epsilon$ -guaraded with respect to  $\mathcal{V}$  if  $I_{\mathcal{V}}(\mathrm{X} \to \mathrm{Z}) < \epsilon$ .
In Appendix B, we showed the equivalence of guardedness (as we have defined it in Definition 2.1) to the optimality of the trivially attainable loss. That is,  $\mathrm{X}(\mathcal{V},\mathfrak{L})$ -guards  $\mathrm{Z}$  when  $H_{\mathcal{V}}(\mathrm{Z}\mid \mathrm{X}) = L_{\tau} = H_{\mathcal{V}}(\mathrm{Z})$ , in the case where  $\mathfrak{L}$  is the singleton class consisting solely of the cross-entropy loss function. In the language of [31],  $\mathrm{X}$  is  $\epsilon$ -guarded with respect to  $\mathcal{V}$  for all  $\epsilon >0$ .
H Constraining Norm Growth
In early concept scrubbing experiments (Sec. 6), we found that at specific layers in some models, concept scrubbing with LEACE would cause the norm of the representation to diverge, leading to NaN outputs. By contrast, SAL never caused divergence, even though it causes a larger disruption to model performance on average (Table 1). This is because SAL uses an orthogonal projection  $\mathbf{Q}$ , whose eigenvalues are thus all in  $\{0,1\}$ , so the norm of the hidden state can never increase after erasure, while LEACE's oblique projection matrix  $\mathbf{P}$  does generally have singular values greater than 1. To combine the superior average-case MSE of LEACE with the stability of SAL, we adopt a simple regularization heuristic. After constructing  $\mathbf{P}$ , we analytically compute the
trace of the covariance matrix of the hidden states after applying  $\mathbf{P}$ . If  $\mathrm{tr}(\mathbf{P}\boldsymbol{\Sigma}_{\mathrm{XX}}\mathbf{P}^{\mathrm{T}}) > \mathrm{tr}(\boldsymbol{\Sigma}_{\mathrm{XX}})$ , we solve a quadratic equation to find the convex combination  $\mathbf{P}' = \alpha \mathbf{P} + (1 - \alpha)\mathbf{Q}$  such that  $\mathrm{tr}(\boldsymbol{\Sigma}_{\mathrm{XX}}) = \mathrm{tr}(\mathbf{P}'\boldsymbol{\Sigma}_{\mathrm{XX}}(\mathbf{P}')^{\mathrm{T}})$ . By Theorem 4.1, the set of matrices which ensure linear guardedness is convex, $^{10}$  so  $\mathbf{P}'$  is guaranteed to be in the feasible set. Furthermore, since our mean squared error objective is convex,  $\mathbf{P}'$  is guaranteed to have no worse MSE than  $\mathbf{Q}$ . We find this solves the divergence issue in practice.
I Oracle LEACE
The concept erasure method derived in Section 4 does not require access to concept labels at inference time. That is, we can fit an erasure function on a labeled training dataset, then apply the function to unlabeled datapoints. If we have oracle access to the label  $z$  for each  $x$ , we can achieve an even more surgical edit. In Theorem I.1 below, we derive Oracle LEACE, a closed-form formula for the the nearest  $X'$  to any  $X$  such that  $\mathrm{Cov}(X', Z) = 0$ .
Like in Sec. 4, the resulting  $\mathrm{X}_{\mathrm{LEACE}}^{\prime}$  is "nearest" to X with respect to all p.s.d. inner products  $\mathbf{a}^T\mathbf{M}\mathbf{b}$  defined on  $\mathbb{R}^d$  simultaneously. This is because, by expressing X in a basis that diagonalizes M, we can decompose the problem into  $d$  independent subproblems, one for each component of  $\mathrm{X}_i$ . Each subproblem can then be viewed as an orthogonal projection, not in  $\mathbb{R}^d$ ,
Figure 5: Orthogonal projection of  $i^{\mathrm{th}}$  component of X, itself a vector in the random variable Hilbert space  $\mathcal{H}$ , onto the span of the components of Z. The residual  $\mathrm{X}_i - \operatorname{proj}_{\mathcal{Z}}\mathrm{X}_i$  is the closest vector to  $\mathrm{X}_i$  orthogonal to, and hence uncorrelated with,  $\mathcal{Z} = \operatorname{span}(\{\mathrm{Z}_1,\mathrm{Z}_2\})$ .
but in an abstract vector space of real-valued random variables. For geometric intuition, see Figure 5.
Prior work has noted that computing an orthogonal projection in a random variable Hilbert space is equivalent to solving an ordinary least squares regression problem [1]. Our theorem is a natural extension of this work: we find that  $\mathrm{X}_{\mathrm{LEACE}}^{\prime}$  is equal to the OLS residual from regressing  $\mathrm{X}$  on  $\mathrm{Z}$ , plus a constant shift needed to ensure that erasing  $\mathrm{Z}$  does not change the mean of  $\mathrm{X}$ .
Theorem I.1 (Oracle Concept Erasure). Let  $\mathcal{H}$  be the Hilbert space of square-integrable real-valued random variables equipped with the inner product  $\langle \xi, \zeta \rangle_{\mathcal{H}} \coloneqq \mathbb{E}[\xi \zeta]$ . Let  $(X,Z)$  be random vectors in  $\mathcal{H}^d$  and  $\mathcal{H}^k$  respectively. Then for every p.s.d. inner product  $\langle \mathbf{a}, \mathbf{b} \rangle_{\mathbf{M}} = \mathbf{a}^T \mathbf{M} \mathbf{b}$  on  $\mathbb{R}^d$ , the objective
$$
\operatorname * {a r g m i n} _ {\mathrm {X} ^ {\prime} \in \mathcal {H} ^ {d}} \mathbb {E} \big \| \mathrm {X} ^ {\prime} - \mathrm {X} \big \| _ {\mathbf {M}} ^ {2} \quad \text {s u b j e c t t o} \operatorname {C o v} (\mathrm {X} ^ {\prime}, \mathrm {Z}) = \mathbf {0}
$$
is minimized by the (appropriately shifted) ordinary least squares residuals from regressing X on Z:
$$
\mathrm {X} _ {\text {L E A C E}} ^ {\prime} = \mathrm {X} - \boldsymbol {\Sigma} _ {\mathrm {X Z}} \boldsymbol {\Sigma} _ {\mathrm {Z Z}} ^ {+} (\mathrm {Z} - \mathbb {E} [ \mathrm {Z} ]).
$$
Proof. Assume w.l.o.g. that  $\mathbf{X}$  and  $\mathbf{X}'$  are represented in a basis diagonalizing  $\mathbf{M}$ , so we may write
$$
\mathbb {E} \big \| \mathrm {X} ^ {\prime} - \mathrm {X} \big \| _ {\mathbf {M}} ^ {2} = \sum_ {i = 1} ^ {d} m _ {i} \mathbb {E} \big [ (\mathrm {X} _ {i} ^ {\prime} - \mathrm {X} _ {i}) ^ {2} \big ],
$$
where  $m_1, \ldots, m_d \geq 0$  are eigenvalues of  $\mathbf{M}$ . Crucially, each term in this sum is independent from the others, allowing us to decompose the primal problem into  $d$  separate subproblems of the form  $\| \mathrm{X}_i' - \mathrm{X}_i \|_{\mathcal{H}}^2$ , one for each component  $i$  of  $(\mathrm{X}, \mathrm{X}')$ .
Factoring out constants. Now consider the subspace  $\mathcal{C} = \operatorname{span}(1) \subset \mathcal{H}$  consisting of all constant (i.e. zero variance) random variables. Orthogonally decomposing  $\mathrm{X}_i$  along  $\mathcal{C}$  yields  $\mathrm{X}_i = \tilde{\mathrm{X}}_i + \mu_i$ , where  $\mu_i = \mathbb{E}[\mathrm{X}_i] \in \mathcal{C}$  and  $\tilde{\mathrm{X}}_i = \mathrm{X} - \mathbb{E}[\mathrm{X}]_i \in \mathcal{C}^\perp$ , and likewise for  $\mathrm{X}_i'$ . Our objective is now
$$
\left\| \mathrm {X} _ {i} ^ {\prime} - \mathrm {X} _ {i} \right\| _ {\mathcal {H}} ^ {2} = \left\| \mu_ {i} ^ {\prime} - \mu_ {i} \right\| _ {\mathcal {H}} ^ {2} + \left\| \tilde {\mathrm {X}} _ {i} ^ {\prime} - \tilde {\mathrm {X}} _ {i} \right\| _ {\mathcal {H}} ^ {2}. \tag {7}
$$
<sup>10</sup>In fact, it is a subspace of  $\mathbb{R}^{d\times d}$ . For any matrices  $\mathbf{A}$ ,  $\mathbf{B} \in \mathbb{R}^{d\times d}$  such that  $\mathbf{A}\pmb{\Sigma}_{\mathrm{XZ}} = \mathbf{0}$  and  $\mathbf{B}\pmb{\Sigma}_{\mathrm{XZ}} = \mathbf{0}$ , we have by linearity  $(\alpha \mathbf{A} + \beta \mathbf{B})\pmb{\Sigma}_{\mathrm{XZ}} = \alpha \mathbf{A}\pmb{\Sigma}_{\mathrm{XZ}} + \beta \mathbf{B}\pmb{\Sigma}_{\mathrm{XZ}} = \alpha \mathbf{0} + \beta \mathbf{0} = \mathbf{0}$  for any scalars  $\alpha$  and  $\beta$ .
Since  $\mu_i'$  and  $\mu_i$  are orthogonal to  $\tilde{\mathrm{X}}_i'$  and  $\tilde{\mathrm{X}}_i$ , and the constraint  $\operatorname{Cov}(\mathrm{X}', \mathrm{Z}) = 0$  is invariant to constant shifts, we can optimize the two terms in Eq. 7 independently. The first term is trivial: it is minimized when  $\mu_i' = \mu_i$ , and hence  $\mathrm{X}_i' = \tilde{\mathrm{X}}_i' + \mathbb{E}[\mathrm{X}_i]$ .
Orthogonal projection. We can now rewrite the zero covariance condition as an orthogonality constraint on  $\tilde{\mathbf{X}}_i$ . Specifically, for every  $i \in \{1 \ldots d\}$  we have
$$
\underset {\tilde {\mathrm {X}} _ {i} ^ {\prime} \in \mathcal {H}} {\operatorname {a r g m i n}} \| \tilde {\mathrm {X}} _ {i} ^ {\prime} - \tilde {\mathrm {X}} _ {i} \| _ {\mathcal {H}} ^ {2} \quad \text {s . t .} \forall j \in \{1 \dots k \}: \langle \tilde {\mathrm {X}} _ {i} ^ {\prime}, \tilde {\mathrm {Z}} _ {j} \rangle_ {\mathcal {H}} = 0, \tag {8}
$$
where  $\tilde{\mathbf{Z}} = \mathbf{Z} - \mathbb{E}[\mathbf{Z}]$ . In other words, we seek the nearest  $\tilde{\mathrm{X}}_i'$  to  $\tilde{\mathrm{X}}_i$  orthogonal to  $\mathcal{Z} = \operatorname{span}(\{\tilde{\mathrm{Z}}_1, \ldots, \tilde{\mathrm{Z}}_k\})$ , which is simply the orthogonal projection of  $\tilde{\mathrm{X}}_i$  onto  $\mathcal{Z}^\perp$ . This in turn is equal to the ordinary least squares residual from regressing  $\tilde{\mathrm{X}}$  on  $\tilde{\mathrm{Z}}$ :