SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nora Belrose<sup>1</sup> David Schneider-Joseph<sup>1</sup> Shauli Ravfogel<sup>2</sup> Ryan Cotterell<sup>3</sup> Edward Raff<sup>4</sup> Stella Biderman<sup>1,4</sup>
$^{1}$ EleutherAI  $^{2}$ Bar-Ilan University  $^{3}$ ETH Zürich  $^{4}$ Booz Allen Hamilton
{nora,stella}@eleuther.ai david@davidsj.com
Concept erasure aims to remove specified features from a representation. It can improve fairness (e.g. preventing a classifier from using gender or race) and interpretability (e.g. removing a concept to observe changes in model behavior). We introduce LEAst-squares Concept Erasure (LEACE), a closed-form method which provably prevents all linear classifiers from detecting a concept while changing the representation as little as possible, as measured by a broad class of norms. We apply LEACE to large language models with a novel procedure called concept scrubbing, which erases target concept information from every layer in the network. We demonstrate our method on two tasks: measuring the reliance of language models on part-of-speech information, and reducing gender bias in BERT embeddings. Our code is available at https://github.com/EleutherAI/concept-erasure.
The ability to prevent a machine learning system from using a specified concept is important for fairness and interpretability. Popular notions of fairness require that protected attributes should not causally affect predictions [22, 26], and interpretability research often estimates the causal effect of a concept by attempting to remove it from a model's internal representations [10, 30, 25, 5, 18].
What it means for a model  $\mathcal{M}$  to "use" a concept  $Z$  is often vague and application-specific, but a necessary condition is that its outputs—and therefore its inputs and hidden states—should have significant mutual information with  $Z$ .<sup>1</sup> Concept erasure leverages this fact to limit  $\mathcal{M}$ 's use of  $Z$  without finetuning or inspecting its parameters. Instead, we edit the input or hidden states  $X$  used by  $\mathcal{M}$  to minimize the predictive  $\mathcal{V}$ -information  $I_{\mathcal{V}}(X \to Z)$  [43], a tractable lower bound on the mutual information  $I(X; Z)$  which measures the degree to which classifiers from the family  $\mathcal{V}$  can predict  $Z$ . Intuitively, if no classifier in  $\mathcal{V}$  can outperform a constant function at predicting  $Z$ —a condition known as guardedness—then  $\mathcal{M}$  can't use  $Z$  either, at least if  $\mathcal{V}$  is expressive enough relative to  $\mathcal{M}$ .
In this work, we improve upon existing concept erasure techniques using a theory-driven approach. We focus on the case where  $\mathcal{V}$  is the set of linear classifiers, and prove a previously unnoticed equivalence: a classification task is linearly guarded if and only if every class has exactly the same mean feature vector (§ 3). Leveraging this equivalence, we derive a simple necessary and sufficient condition for an affine transformation to produce linearly guarded features. We then identify the unique surgical transformation in this family—the one that minimizes the mean squared distance from the original features with respect to all norms induced by inner products, including the popular Euclidean and Mahalanobis norms. We name it LEAst-squares Concept Erasure (LEACE) (§ 4).
While prior work has focused on preventing linear models from leveraging Z, we aim to erase concepts from deep neural networks as well. Interpretability research has shown that networks
This follows from the fact that causal dependence is a special kind of statistical dependence [28]. By the data processing inequality,  $\mathcal{M}$ 's output can't have any more information about  $Z$  than its input or hidden states.
can be usefully described as encoding features in linear subspaces [11, 24, 41], suggesting that fundamentally nonlinear methods may not be necessary for successful erasure in DNNs. In light of this, we introduce a simple procedure called concept scrubbing (§ 6), which sequentially applies LEACE to the intermediate representations at each layer of a deep network.
We empirically validate our proposals, demonstrating the superiority of LEACE for erasing gender bias from BERT representations (§ 5.2), and using concept scrubbing to measure the extent to which large language models use part-of-speech information (§ 6).
Consider a  $k$ -class classification task over jointly defined random vectors  $\mathrm{X}$  (the input data) and  $\mathrm{Z}$  (the one-hot labels), with  $\mathrm{X}$  of finite first moment and taking values in  $\mathbb{R}^d$ , and  $\mathrm{Z}$  taking values in  $\mathcal{Z} = \{\mathbf{z} \in \{0,1\}^k \mid \| \mathbf{z} \|_1 = 1\}^2$  with each  $\mathbb{P}(\mathrm{Z} = j) > 0$ . Let  $\eta(\cdot; \pmb{\theta}) : \mathbb{R}^d \to \mathbb{R}^k$  be a predictor chosen from a function class  $\mathcal{V} = \{\eta(\cdot; \pmb{\theta}) \mid \pmb{\theta} \in \Theta\}$  (presumed to contain all constant functions) so as to minimize the expectation  $\mathbb{E}\big[\mathcal{L}(\eta(\mathrm{X}), \mathrm{Z})\big]$  of some  $\mathcal{L} : \mathbb{R}^k \times \mathcal{Z} \to [0, \infty)$  in a class  $\mathfrak{L}$  of loss functions.
We borrow the concept of guardedness from Ravfogel et al. [31], who define it in terms of  $\nu$ -information [43]. We opt for a slightly more general definition here, which is equivalent to theirs in the case of cross-entropy loss (see Appendix G).
Definition 2.1 (Guardedness). Let  $X, Z, \mathcal{V},$  and  $\mathfrak{L}$  be as defined above, and let  $\chi$  be the set of all random vectors of finite first moment taking values in  $\mathbb{R}^d$ , jointly defined with  $Z$ .
We say  $\mathrm{X}(\mathcal{V},\mathfrak{L})$  guards  $\mathrm{Z}$  if, for all losses  $\mathcal{L}\in \mathfrak{L}$ , it maximizes the minimum expected loss:
$$
\mathrm{X}\in \operatorname *{argmax}_{\mathrm{X}^{\prime}\in \chi}\inf_{\boldsymbol {\theta}\in \Theta}\mathbb{E}\Bigl[\mathcal{L}(\eta (\mathrm{X}^{\prime};\boldsymbol {\theta}),\mathrm{Z})\Bigr ].
$$
In other words, its conditional distribution  $\mathbb{P}(\mathrm{X}\mid \mathrm{Z} = \cdot)$  is among the worst possible distributions for predicting Z from X using a predictor of the form  $\eta (\cdot ;\pmb {\theta})\in \mathcal{V}$  and a loss function in  $\mathfrak{L}$ .
Definition 2.2 (Trivially Attainable Loss). The trivially attainable loss for labels  $Z$  and loss  $\mathcal{L}$  is the lowest possible expected loss available to a constant predictor  $\eta(\mathbf{x}) = \mathbf{b}$ :
$$
L _ {\tau} = \inf  _ {\mathbf {b} \in \mathbb {R} ^ {k}} \mathbb {E} [ \mathcal {L} (\mathbf {b}, \mathrm {Z}) ]
$$
We will sometimes write it  $L_{\tau}^{(\mathsf{Z}, \mathcal{L})}$  in cases of possible ambiguity. If there is a specific constant predictor actually achieving this loss, we call it the trivial predictor  $\eta_{\tau} = \eta_{\tau}^{(\mathsf{Z}, \mathcal{L})}$ .
We examine this problem in the important case of loss functions  $\mathcal{L}:\mathbb{R}^k\times \mathcal{Z}\to [0,\infty)$  which are convex in the prediction  $\eta (\mathbf{x})$ , and linear predictors that take the functional form  $\eta (\mathbf{x};\mathbf{b},\mathbf{W}) = \mathbf{b} + \mathbf{W}\mathbf{x}$ , for some bias  $\mathbf{b}\in \mathbb{R}^k$  and weight matrix  $\mathbf{W}\in \mathbb{R}^{k\times d}$ .
Definition 2.3 (Linear Guardedness). If  $\mathrm{X}(\mathcal{V},\mathfrak{L})$ -guards  $\mathbf{Z}$ , where  $\mathfrak{L}$  is the class of nonnegative loss functions which are convex in their first argument, and  $\mathcal{V}$  is the class of linear predictors  $\eta (\mathbf{x}) = \mathbf{b} + \mathbf{W}\mathbf{x}$ , we say that  $\mathrm{X}$  linearly guards  $\mathbf{Z}$ .
Our primary theoretical result is that the following conditions are all equivalent:
1. The data X linearly guards the labels Z. (Definition 2.3)
2. For all convex losses  $\mathcal{L}$ , the trivially attainable loss is optimal on  $(\mathrm{X},\mathrm{Z})$ . (Definition 2.2)
3. The class-conditional mean vectors  $\mathbb{E}[\mathrm{X}\mid \mathrm{Z} = i]$  are equal to the unconditional mean  $\mathbb{E}[\mathrm{X}]$ .
4. Every component of  $\mathrm{X}$  has zero covariance with every component of  $\mathrm{Z}$ .
5. Every linear classifier evaluated on X exhibits statistical parity w.r.t. Z. (App. C)
The equivalence of conditions 1, 2, and 5 is relatively straightforward to show, and the relevant theorems can be found in Appendices B and C. The other equivalences are proven below (cond.  $3 \leftrightarrow$  cond. 2 in § 3.1 and § 3.2); cond.  $3 \leftrightarrow 4$  in § 3.3).
2We frequently use the integer  $j\leq k$  to refer to the element of  $\mathcal{Z}$  which is 1 at the  $j^{\mathrm{th}}$  index and 0 elsewhere.
3.1 Equality of Class Centroids Implies Linear Guardedness
The following result establishes the implication from condition 3 to condition 2.
Theorem 3.1. Suppose  $\mathcal{L}$  is convex in the linear prediction  $\eta$ . Then if each class-conditional mean  $\mathbb{E}[\mathrm{X}|\mathrm{Z} = i]$  is equal to  $\mathbb{E}[\mathrm{X}]$ , the trivially attainable loss cannot be improved upon.
Proof. Let  $\eta(\mathbf{x}) = \mathbf{b} + \mathbf{W}\mathbf{x}$  be any linear predictor. By Jensen's inequality, the loss with  $\eta$  evaluated on  $\mathbf{X}$  is lower bounded by the loss with  $\eta$  evaluated on the unconditional mean of the data  $\mathbb{E}[\mathbf{X}]$ :
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} (\eta , Z) \right] = \mathbb {E} _ {Z} \left[ \mathbb {E} \left[ \mathcal {L} (\eta , Z) | Z \right] \right] \\ \geq \mathbb {E} _ {\mathrm {Z}} \left[ \mathcal {L} \left(\mathbb {E} \left[ \eta | \mathrm {Z} \right], \mathrm {Z}\right) \right] \quad (\text {J e s s e n ' s i n e q u a l i t y}) \\ = \mathbb {E} _ {\mathrm {Z}} \left[ \mathcal {L} \left(\mathbf {b} + \mathbf {W} \mathbb {E} [ \mathrm {X} | \mathrm {Z} ], \mathrm {Z}\right) \right] \quad (\text {l i n e a r i t y o f} \eta) \\ = \mathbb {E} _ {\mathrm {Z}} \left[ \mathcal {L} \left(\mathbf {b} + \mathbf {W} \mathbb {E} [ \mathrm {X} ], \mathrm {Z}\right) \right]. \quad (\text {b y}) \\ \end{array}
$$
This in turn is the loss of the constant predictor  $\eta^{\prime}(\mathbf{x}) = \mathbf{b} + \mathbf{W}\mathbb{E}[\mathbf{X}]$ . Since the trivially attainable loss is the best that can be achieved by a constant predictor, and every predictor's loss is lower bounded by that of some constant predictor, we cannot improve upon the trivially attainable loss.
Intuitively, this shows that the classifier's expected loss is lower-bounded by the loss it would receive if each data point were replaced with the centroid of its class. But, if these centroids are all equal, the loss can't be any lower than what we'd get if every data point were replaced with the global mean  $\mathbb{E}[\mathrm{X}]$ . In that case, the data points are indistinguishable and we can't do better than  $\mathbf{W} = \mathbf{0}$ .
3.2 Linear Guardedness Implies Equality of Class Centroids
We now prove the implication from condition 2 to condition 3. Condition 2 applies when the trivially attainable loss is optimal for all convex losses, including cross-entropy loss in particular. And if it holds for cross-entropy loss, we now show that condition 3—the class centroids are equal—must follow. First a more general lemma:
Lemma 3.2. Suppose  $\mathcal{L}$  has bounded partial derivatives, which when off-category never vanish and do not depend on the category, i.e.  $\partial \mathcal{L}(\eta ,z_1) / \partial \eta_i = \partial \mathcal{L}(\eta ,z_2) / \partial \eta_i\neq 0$  for all categories  $z_{1},z_{2}\neq i$ . If  $\mathbb{E}\big[\mathcal{L}(\eta ,Z)\big]$  is minimized among linear predictors by the constant predictor  $\eta (\mathbf{x}) = \mathbf{b}^{*} + \mathbf{W}^{*}\mathbf{x}$  with  $\mathbf{W}^{*} = \mathbf{0}$ , then each class-conditional mean  $\mathbb{E}\big[\mathrm{X}|\mathrm{Z} = i\big]$  is equal to  $\mathbb{E}\big[\mathrm{X}\big]$ .
Proof. The first-order optimality condition on the  $i^{\text{th}}$  component of our parameters  $\mathbf{b}$  and  $\mathbf{W}$  yields the equations:
$$
\mathbb {E} \left[ \frac {\partial \mathcal {L} (\eta , Z)}{\partial \eta_ {i}} \cdot \frac {\partial \eta_ {i}}{\partial b _ {i}} \right] = 0 \quad \text {a n d} \quad \mathbb {E} \left[ \frac {\partial \mathcal {L} (\eta , Z)}{\partial \eta_ {i}} \cdot \frac {\partial \eta_ {i}}{\partial \mathbf {W} _ {i}} \right] = \mathbf {0}, \tag {1}
$$
where we have used the boundedness of  $\mathcal{L}$  's partial derivative and the finite first moment of  $\frac{\partial\eta_i}{\partial b_i} = 1$  and  $\frac{\partial\eta_i}{\partial\mathbf{W}_i} = \mathbf{X}$  to justify (via the Dominated Convergence Theorem) interchanging the derivative with the expectation.
Since  $\eta$  is constant over all values of  $X$ , and  $\frac{\partial \eta_i}{\partial b_i} = 1$ , the first equation in (1) reduces to:
$$
\mathbb {P} (Z = i) \frac {\partial \mathcal {L} (\eta , i)}{\partial \eta_ {i}} + \mathbb {P} (Z \neq i) \frac {\partial \mathcal {L} (\eta , \neq i)}{\partial \eta_ {i}} = 0, \tag {2}
$$
where  $\frac{\partial\mathcal{L}(\eta,\neq i)}{\partial\eta_i}$  is an abuse of notation denoting the off-category partial derivative, emphasizing its independence of the category  $Z$ .
3 Specifically, its generalization to convex functions over  $\mathbb{R}^k$ . See [12] p. 76.
Similarly, the constancy of  $\eta$  and the fact that  $\frac{\partial\eta_i}{\partial\mathbf{W}_i} = \mathbf{X}$  reduces the second equation in (1) to:
$$
\mathbb {P} (\mathrm {Z} = i) \frac {\partial \mathcal {L} (\eta , i)}{\partial \eta_ {i}} \cdot \mathbb {E} [ \mathrm {X} | \mathrm {Z} = i ] + \mathbb {P} (\mathrm {Z} \neq i) \frac {\partial \mathcal {L} (\eta , \neq i)}{\partial \eta_ {i}} \cdot \mathbb {E} [ \mathrm {X} | \mathrm {Z} \neq i ] = \mathbf {0}. \tag {3}
$$
Solving for  $\mathbb{P}(\mathrm{Z} = i)\frac{\partial\mathcal{L}(\eta,i)}{\partial\eta_i}$  in (2) and substituting in (3) gives us:
$$
\mathbb {P} (Z \neq i) \frac {\partial \mathcal {L} (\eta , \neq i)}{\partial \eta_ {i}} \cdot \left(\mathbb {E} [ X | Z \neq i ] - \mathbb {E} [ X | Z = i ]\right) = 0.
$$
If  $\mathbb{P}(\mathrm{Z}\neq i) = 0$ , then  $\mathbb{E}[\mathrm{X}] = \mathbb{E}[\mathrm{X}|Z = i]$  is trivially true. Otherwise, using the non-vanishingness of the off-category partial derivative  $\frac{\partial\mathcal{L}(\eta,\neq i)}{\partial\eta_i}$ , division yields the equivalence of  $\mathbb{E}\big[\mathrm{X}|\mathrm{Z} = i\big]$  to  $\mathbb{E}\big[\mathrm{X}|\mathrm{Z}\neq i\big]$ , and hence to the unconditional mean  $\mathbb{E}[\mathrm{X}]$ .
We now show that Lemma 3.2 applies to the widely used cross entropy loss:
Theorem 3.3. If the class probabilities  $\mathbb{P}(\mathrm{Z} = j)$  are all nonzero, and the trivially obtainable loss is optimal when  $\mathcal{L}(\eta ,z) = -\log \frac{\exp(\eta_z)}{\sum_{i = 1}^{k}\exp(\eta_i)}$ , then each class has the same mean  $\mathbb{E}\big[\mathrm{X}|\mathrm{Z} = z\big]$ .
Proof. In this case, the trivial predictor  $\eta_{\tau}(\mathbf{Z})_j = \log (\mathbb{P}(\mathbf{Z} = j))$  exists, achieving the trivially obtainable loss, which we have assumed optimal. Furthermore,  $\mathcal{L}$  has on-category partial derivative  $\partial \mathcal{L}(\eta ,i) / \partial \eta_i = \exp (\eta_i) / \sum_{j = 1}^k\exp (\eta_j) - 1\in (-1,0]$ , and nonvanishing off-category partial derivative  $\partial \mathcal{L}(\eta ,\neq i) / \partial \eta_i = \exp (\eta_i) / \sum_{j = 1}^k\exp (\eta_j)\in (0,1)$ , both bounded, so the conditions of Lemma 3.2 apply.
3.3 Linearly Guarded Labels Have Zero Covariance with the Features
The next theorem establishes the equivalence of conditions 3 and 4.
Theorem 3.4. Let  $X$  be a random vector taking values in  $\mathbb{R}^d$  with finite first moment, and  $Z$  a random vector taking values in  $\{0,1\}^k$  with one-hot encoding, with each class probability  $\mathbb{P}(Z = j)$  being nonzero. Then the class-conditional means  $\mathbb{E}[X|Z = j]$  are all equal to the unconditional mean  $\mathbb{E}[X]$  if and only if every component of  $X$  has zero covariance with every component of  $Z$ , i.e., the cross-covariance matrix  $\Sigma_{XZ}$ , whose  $(i,j)^{th}$  entry is  $\operatorname{Cov}(X_i,Z_j)$ , is the zero matrix.
Proof. Since  $\mathbf{Z}$  is one-hot, we can rewrite the  $(i,j)^{\mathrm{th}}$  entry of  $\Sigma_{\mathrm{XZ}}$  as:
$$
\mathbb {E} \left[ \mathrm {X} _ {i} \mathrm {Z} _ {j} \right] - \mathbb {E} \left[ \mathrm {X} _ {i} \right] \mathbb {E} \left[ \mathrm {Z} _ {j} \right] = \mathbb {P} (\mathrm {Z} = j) \Big (\mathbb {E} \left[ \mathrm {X} _ {i} \mid \mathrm {Z} = j \right] - \mathbb {E} \left[ \mathrm {X} _ {i} \right] \Big).
$$
As  $\mathbb{P}(\mathrm{Z} = j) > 0$ , it follows that  $\mathbb{E}[\mathrm{X}_i|\mathrm{Z} = j] = \mathbb{E}[\mathrm{X}_i]$  if and only if  $\operatorname{Cov}(\mathrm{X}_i, \mathrm{Z}_j) = 0$ .
We have thus established the equivalence of the first four conditions stated earlier. See Appendix C for the last one, on statistical parity.
4 Least-Squares Concept Erasure
In Section 3 we saw that  $\mathrm{X}$  linearly guards  $\mathbf{Z}$  if and only if each component of  $\mathrm{X}$  has zero covariance with each component of  $\mathbf{Z}$ . We will now characterize the set of affine transformations  $r(\mathbf{x}) = \mathbf{P}\mathbf{x} + \mathbf{b}$  such that  $r(\mathrm{X})$  linearly guards  $\mathbf{Z}$ .
Theorem 4.1. Let  $X$  and  $Z$  be random vectors taking values in  $\mathbb{R}^d$  and  $\mathbb{R}^k$  respectively, with  $X$  of finite first moment. Then given some affine function  $r(\pmb{x}) = \mathbf{P}\pmb{x} + \mathbf{b}$ , the modified random vector  $r(X)$  linearly guards  $Z$  if and only if the columns of the cross-covariance matrix  $\pmb{\Sigma}_{\mathrm{XZ}}$  are contained in the null space of  $\mathbf{P}$ .
Proof. From Theorem 3.4 we know that  $r(\mathrm{X})$  linearly guards  $\mathrm{Z}$  if and only if  $\operatorname{Cov}(r(\mathrm{X}), \mathrm{Z})$  is the zero matrix. By the linearity property of cross-covariance, we have:
$$
\operatorname {C o v} (r (\mathrm {X}), \mathrm {Z}) = \operatorname {C o v} (\mathrm {P X} + \mathrm {b}, \mathrm {Z}) = \mathrm {P C o v} (\mathrm {X}, \mathrm {Z}) = \mathrm {P} \Sigma_ {\mathrm {X Z}}.
$$
Therefore,  $r(\mathrm{X})$  linearly guards  $\mathbf{Z}$  if and only if  $\ker (\mathbf{P})\supseteq \operatorname {colsp}(\Sigma_{\mathrm{XZ}})$
Implications for prior work. Notably, the above theorems imply that three previously proposed methods in the literature, Spectral Attribute Removal (SAL) [37], Mean Projection [17], and Fair PCA [20], are guaranteed to achieve linear guardedness given suitable hyperparameters. See Appendix D for further discussion.
4.1 Derivation of LEACE
Theorem 4.1 is a very weak condition, which is far from identifying unique values for  $\mathbf{P}$  and  $\mathbf{b}$ . In most applications, however, we'd like to make a "small" edit to  $\mathrm{X}$  so that useful information contained in  $\mathrm{X}$  is maximally preserved. We operationalize the notion of a small edit in terms of the mean squared norm  $\mathbb{E}\| r(\mathrm{X}) - \mathrm{X}\|_{\mathbf{M}}^2$  defined by some positive-definite inner product  $\mathbf{M}$ ,<sup>4</sup> which can be thought of as a local quadratic approximation to any measure of divergence between  $\mathrm{X}$  and  $r(\mathrm{X})$  (such as Kullback-Leibler divergence, for example). While we are primarily interested in the Euclidean  $(\mathbf{M} = \mathbf{I})$  and Mahalanobis  $(\mathbf{M} = \boldsymbol{\Sigma}_{\mathrm{XX}}^{+})$  norms, it will turn out that there is a single erasure function that minimizes all such norms simultaneously. We will see in Section 6 that ensuring edits are small in this sense provides substantial benefit to downstream task performance as compared to other methods which also guard the labels  $\mathbf{Z}$ .
Below, we derive the optimal eraser under the assumption that X and Z are centered.
Theorem 4.2. Let  $\mathrm{X}$  and  $\mathrm{Z}$  be centered random vectors taking values in  $\mathbb{R}^d$  and  $\mathbb{R}^k$  respectively, each of finite second moment. Let  $\mathbf{M} \in \mathbb{R}^{d \times d}$  be a p.s.d. matrix defining a (possibly degenerate) inner product on  $\mathbb{R}^d$ :  $\langle \mathbf{x}, \mathbf{y} \rangle_{\mathbf{M}} = \mathbf{x}^T \mathbf{M} \mathbf{y}$ . Let  $\boldsymbol{\Sigma}_{\mathrm{XX}} \in \mathbb{R}^{d \times d}$  be X's covariance matrix, and  $\boldsymbol{\Sigma}_{\mathrm{XZ}} \in \mathbb{R}^{d \times k}$  be the cross-covariance matrix of X and Z. Let  $\mathbf{A}^+$  denote the Moore-Penrose pseudoinverse of a matrix  $\mathbf{A}$ , and let  $\mathbf{A}^{1/2}$  be the p.s.d. square root of a p.s.d. matrix  $\mathbf{A}$ . Then the objective
$$
\underset {\mathbf {P} \in \mathbb {R} ^ {d \times d}} {\operatorname {a r g m i n}} \mathbb {E} \left[ \left\| \mathbf {P X} - \mathrm {X} \right\| _ {\mathbf {M}} ^ {2} \right] \quad \text {s u b j e c t t o} \operatorname {C o v} (\mathbf {P X}, \mathrm {Z}) = \mathbf {0}
$$
has the following solution:
$$
\mathbf {P} ^ {*} = \mathbf {I} - \mathbf {W} ^ {+} \mathbf {P} _ {\mathbf {W} \Sigma_ {\mathrm {X Z}}} \mathbf {W},
$$
where  $\mathbf{W}$  is the whitening transformation  $(\Sigma_{\mathrm{XX}}^{1 / 2})^+$  and  $\mathbf{P}_{\mathbf{W}\Sigma_{\mathrm{XZ}}} = (\mathbf{W}\Sigma_{\mathrm{XZ}})(\mathbf{W}\Sigma_{\mathrm{XZ}})^+$  is the orthogonal projection matrix onto  $\operatorname{colsp}(\mathbf{W}\Sigma_{\mathrm{XZ}})$ .
Proof. See Appendices E.1 and E.2 for two independent proofs of Theorem 4.2.
The above theorem assumes that the random vectors  $\mathrm{X}$  and  $\mathrm{Z}$  are centered, and does not include a bias term. Below we extend our results to the uncentered case, and derive the optimal bias  $\mathbf{b}^*$ .
Theorem 4.3. Let  $\mathrm{X}$  and  $\mathrm{Z}$  be random vectors taking values in  $\mathbb{R}^d$  and  $\mathbb{R}^k$  respectively, each of finite second moment. Define  $\mathbf{M}$  and  $\mathbf{P}^*$  as in Theorem 4.2 and  $\mathbf{b}^* = \mathbb{E}[\mathrm{X}] - \mathbf{P}^*\mathbb{E}[\mathrm{X}]$ . Then  $(\mathbf{P}^*,\mathbf{b}^*)$  minimizes  $\mathbb{E}\| \mathbf{P}\mathbf{X} + \mathbf{b} - \mathbf{X}\| ^2$ , subject to  $\operatorname {Cov}(\mathbf{P}\mathbf{X} + \mathbf{b},\mathbf{Z}) = \mathbf{0}$ .
Proof. Let  $\mathbf{P} \in \mathbb{R}^{d \times d}$  and define  $\tilde{\mathbf{X}} = \mathbf{X} - \mathbb{E}[\mathbf{X}]$  and  $\mathbf{c} = \mathbf{P} \mathbb{E}[\mathbf{X}] + \mathbf{b} - \mathbb{E}[\mathbf{X}]$ . Then,
$$
\begin{array}{l} \mathbb {E} \left\| \mathbf {P X} + \mathbf {b} - \mathrm {X} \right\| _ {\mathbf {M}} ^ {2} = \mathbb {E} \left\| (\mathbf {P} \tilde {\mathrm {X}} - \tilde {\mathrm {X}}) + \mathbf {c} \right\| _ {\mathbf {M}} ^ {2} \\ = \mathbb {E} \left\| \mathbf {P} \tilde {\mathrm {X}} - \tilde {\mathrm {X}} \right\| _ {\mathbf {M}} ^ {2} + 2 \mathbb {E} \left[ \mathbf {P} \tilde {\mathrm {X}} - \tilde {\mathrm {X}} \right] ^ {T} \mathbf {M} \mathbf {c} + \mathbf {c} ^ {T} \mathbf {M} \mathbf {c} \\ = \mathbb {E} \big \| \mathbf {P} \tilde {\mathrm {X}} - \tilde {\mathrm {X}} \big \| _ {\mathbf {M}} ^ {2} + \mathbf {c} ^ {T} \mathbf {M} \mathbf {c}, \\ \end{array}
$$
where we have eliminated the middle term because  $\mathbf{P}$  is linear and  $\mathbb{E}[\tilde{\mathbf{X}}] = 0$ . Since  $\mathbf{M}$  is p.s.d., our objective is minimized for  $\mathbf{c} = \mathbf{0}$ , i.e.  $\mathbf{b} = \mathbb{E}[\mathrm{X}] - \mathbf{P}\mathbb{E}[\mathrm{X}]$ . The problem thus reduces to choosing  $\mathbf{P}$  so as to minimize  $\mathbb{E}\left\| \mathbf{P}\tilde{\mathbf{X}} -\tilde{\mathbf{X}}\right\|_{\mathbf{M}}^{2}$  subject to  $\operatorname {Cov}(\mathbf{P}\mathbf{X} + \mathbf{b},\mathbf{Z}) = \operatorname {Cov}(\mathbf{P}\tilde{\mathbf{X}},\mathbf{Z}) = \mathbf{0}$ , which Theorem 4.2 shows occurs when  $\mathbf{P} = \mathbf{P}^*$ .
4Our proofs also include degenerate "inner products" where  $\mathbf{M}$  is singular, and the associated seminorms.