LEACE: Perfect linear concept erasure in closed form
Authors: Nora Belrose
David Schneider-Joseph
Shauli Ravfogel
Ryan Cotterell
Edward Raff
Stella Biderman
Venue: NeurIPS 2023 poster (2023)
Keywords: interpretability
fairness
concept erasure
representation
adversarial
robustness
TL;DR: We derive a simple, efficient method for surgically editing neural representations to erase unwanted concepts, with strong theoretical guarantees.

Abstract: Concept erasure aims to remove specified features from a representation. It can improve fairness (e.g. preventing a classifier from using gender or race) and interpretability (e.g. removing a concept to observe changes in model behavior). We introduce LEAst-squares Concept Erasure (LEACE), a closed-form method which provably prevents all linear classifiers from detecting a concept while changing the representation as little as possible, as measured by a broad class of norms. We apply LEACE to large language models with a novel procedure called concept scrubbing, which erases target concept information from _every_ layer in the network. We demonstrate our method on two tasks: measuring the reliance of language models on part-of-speech information, and reducing gender bias in BERT embeddings. Our code is available at https://github.com/EleutherAI/concept-erasure.

LEACE: Perfect linear concept erasure in closed form
Nora Belrose'
David Schneider-Joseph
Shauli Ravfogel
Ryan Cotterell
Edward Raf
Stella Biderman 1,4
EleutherAI
2 Blrn University
"ETH Zürich 4Booz Allen Hamilton
{nora, shola,stella@@@eeuther ai
davidedavidsj com
Abstract
Concept erasure aims to remove specified features from a representation. It can improve fairness (e.g. preventing classifier from using gender or race) and interpretability (e.g. removing concept to observe changes in model behavior). We introduce I.EAst-squares Concept Erasure (LFACEE), a closed-form method which provably prevents all linear classifiers from detecting concept while changing the representation as little as possible, as measured by a broad class norms. We apply LEACE to large language models with novel procedure called concept scrubbing, which erases target concept information from every layer in the network. We demonstrate method on part-of-speecl information, and reducing gender bias BERT embeddings. Our code is available at uthhtps:/
tasks: measuring the reliance of language models
com/ EleutherAI/conceeeee erasure
1 Introduction
The ability to prevent machine learning system from using specified concept is important for fairness and interpretability. Popular notions of fairness require that protected attributes should not causally affect predictions [22 26), and interpretability research often estimates the causal effect of concept by attempting to remove from a model's s internal representations 10
25 [8]
What means for a model M to "use" a concept Z is often vague and application-specific, but necessary condition is that its outputs- therefore its inputs and hidden states- have significant mutual information with z Concept erasure leverages this fact to limit M''s use of withou! finetuning or inspecting its parameters. Instead, edit the input or hidden states X used by M to minimize the predictive V-information [y + Z) 43), tractable lower bound on the mutual information / (X; Z) which measures the degree to which classifiers from the family V can predict Z. Intuitively, if no classifier in V can outperform a constant function at predicting Z-a condition known as guardedness then M can't use either, at least if expressive enough relative to M.
In this work, we improve upon existing concept erasure techniques using theory-driven approach. We focus the case where V is the set of linear classifiers, and prove previously unnoticed cquivalence: classification task is linearly guarded if and only if every class has exactly the same mean feature vector ($ 3). Leveraging this equivalence, condition for an affine transformation to produce linearly guarded features. We then identify the unique surgical transformation in this family- the one that minimizes the mean squared distance from the original features with respect to all norms induced by inner products, including the popular Euclidean and Mahalanobis norms. We name LEAst-squares Concept Erasure (LEACE)
derive simple necessary and sufficient
While prior work has focused on preventing linear models from leveraging Z, we aim to erase concepts from deep neural networks well. Interpretability research has shown that networks
'This follows from the fact that causal dependence is a special kind of statistical dependence 28 By the data processing inequality, M's output can't 't have any more information about Z than its input or hidden states.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

can be usefully described as encoding features in linear subspaces fundamentally nonlinear methods may not be necessary for successful erasure DNNs. In light of this, we introduce simple procedure called concept scrubbing (S 6). which sequentially applies LEACE to the intermediate representations at each layer of a deep network.
I||
suggesting that
We empirically validate our proposals, demonstrating the superiority of LEACE for erasing gender bias from BERT representations 5.2), and using concept scrubbing measure the extent which large language models use part-of-speech information (
Preliminaries
Consider a k-class classification task over jointly defined random vectors X (the input data) and 7. (the one-hot labels), with of finite first moment and taking values Rd and taking values in 2 (z E {0, 13 || D with each P(Z j) > O. Let n1: 0) IR + RR be a predictor chosen from a function class V (n(:;0) 0 E } (presumed to contain all constant functions) so as to minimize the expectation E [[[(n(X), 2)] of some L : IR x 2 [0, oc) in a class & of loss functions.
We borrow the concept of guardedness from Ravfogel et al. 31,. who define it in terms of V- information 43). We opt for slightly more general definition here, which is equivalent to theirs in the case of cross-entropy loss (see Appendix G).
Definition 2.1 (Guardedness). Let X, Z. V, and & be as defined above, and let x be the set of all random vectors of finite first moment taking values in R. jointly defined with Z.
E, it maximizes the minimum expected loss: We say X (V, 12)0grrrdsss 'z for all losses
X E argmax X'EX OE
inf E (C;((X);))))))
other words, its conditional distribution TP(X 7. = is among the worst possible distributions for predicting from X. using predictor of the form n(: V and loss function in L. Definition 2.2 (Trivially Attainable Loss). The trivially attainable loss for labels and loss L is the lowest possible expected loss available to constant predictor 1π(x) (xxx)=b. =b:
L inf [[C
bElk
We will sometimes write it I!Z,C) cases of possible ambiguity. If there specific constant predictor actually achieving this loss, we call the trivial predictor 1)r = f) (Z.L)
We examine this problem in the important case of loss functions L: x 2 - oo) which are convex in the prediction 7)(x), and linear predictors that take the functional form 17(x; b, W) b + Wx, for some bias b E _ and weight matrix W 1xd
Definition 2.3 (Linear Guardedness). If X (V, 2)-gardss Z, where & is the class of nonnegative loss functions which are convex in their first argument, and V is the class of linear predictors n(x) b Wx, we say that X. linearly guards Z.
3 Theoretical Results
Our primary theoretical result is that the following conditions are equivalent:
The data X linearly guards the labels Z. (Definition 2.3
2. For all convex losses L. the trivially attainable loss is optimal on XX,.). (Definition 2.2
3. The class-conditional mean vectors E[[ Z i| are equal to the unconditional mean [[[]]
4. Every component of X has zero covariance with every component of Z.
5. Every linear classifier evaluated on X exhibits statistical parity w.r.t. Z. (App. C
The equivalence conditions T 2 and is relatively straightforward to show, and the relevant theorems can be found in Appendices and The other equivalences are proven below (cond.
cond. 2 in 3.1 and $ h.2h: cond.
3.3).
We frequently the integer < k refer the element of Z which is the 1dh index and 0 elsewhere.

---

3.1 Equality of Class Centroids Implies Linear Guardedness
The following result establishes the implication from condition 3
condition
2
Theorem 3.1. Suppose C is convex the linear prediction n. Then if each class- caas-condittioal mean
E Z = i] is equal to E [[X], the trivially attainable loss cannol be improved upon.
Proof. Let b+ Wx be any linear predictor. By Jensen's 's inequality the loss with 7/ evaluated on X is lower bounded by the loss with evaluated on the unconditional mean of the data E X]:
C(n. 2)
Ez E
(n. 2) |
Ez
(Jensen's 's inequality)
(C((++E(((2)22)) E22[<(b+Wr) WE [ .2]]
(linearity of n)
(by assumption)
This in turn is the loss of the constant predictor (x) b WE[X]. Since the trivially attainable
loss is the best that can be achieved by a constant predictor, and every predictor's 's loss is lower bounded by that of some constant predictor, cannot improve upon the trivially attainable loss. c
Intuitively, this shows that the classifier's expected loss is lower-bounded by the loss it would receive if each data point were replaced with the centroid of its class. But, if these centroids all equal, the loss can't it be any lower than what we' 'd get if every data point were replaced with the global mean E[X]. In that case, the data points are indistinguishable and we can't 't do better than W 0.
3.2 Linear Guardedness Implies Equality of Class Centroids
3 Condition applies when the trivially attainable loss is optimal for all convex losses, including cross-entropy loss in particular. And if holds for cross-entropy loss, now show that condition the class centroids equal-must follow. First a more general lemma:
We now prove the implication from condition 2 to condition
Lemma 3.2. Suppose L has bounded partial derivatives, which when off-category never vanish and do not depend the category, i.e. ÖL(n, 21)2///))))) AL(n, z2)/0m; + 0 for all categories 21, 22 + i. If IE| [(n, Z)] is minimized among linear predictors by the constant predictor rf(x) b" + W with W 0, then each class-conditional mean E.. [ i] is equal E[X].
Proof. The first-order optimality condition on the thh component of our parameters b and W yields the equations:
aC(n, Z)
Om
ac(n, Z)
dy:
E
= 0 and E
0,
(1)
Om:
ab.
dm:
aw1
where we have used the boundedness of L's partial derivative and the finite first moment of On: 1
8b;
Brg and 8W; = to justify (via the Dominated Convergence Theorem) interchanging the derivative with the expectation.
Oy Since v is constant over values of X, and 2b, 1, the first equation
reduces to:
i Onp
ac(n. # On,
P(Z
IP + i)
O,
(2)
where @6(1F1) is abuse of notation denoting the off-category partial derivative, emphasizing its om: independence of the category Z.
Sppecifically, its generalization convex functions over R See 12 p. 76.

---

Similarly, the constancy of and the fact that
Deg OW.
X reduces the second equation in to:
IP(Z = =i) OC(n,
dc(n. + # L x = 0.
E
=
+ P(Z +
Solving for PP(Z. i) (..)):)) dm
in
and substituting in (3) gives us:
E E[ - E - -
ac(n. + t)
P(Z + i)
Om D
If P(Z + 0, then [[X] Z i] is trivially true. Otherwise, using the non non-vanishingness
of the off-category partial derivative o((g.z)) division yields the equivalence of EE [ Z = x] to E Z i]. and hence to the unconditional mean IE [x]
0%,
We now show that Lemma 3.2 applies to the widely used cross entropy loss: Theorem 3.3. If the class probabilities P(Z j) are all nonzero, and the trivially obtainable loss is optimal when [(n, 2) log Eis, exp(nx)
exp(n.) then each class has the same mean E [ z].
Proof. In this case, the trivial predictor Tor (Z); = log(P(Z = j)) exists, achieving the trivially obtainable loss, which we have assumed optimal. Furthermore, L has on-category partial deriva- tive aC(n, i)/0n; exp(n)/E exp(1;) 1 E (-1,0,, O), and nonvanishing off-category partial derivative * 0)/0m: exp(n./E, exp(n,) (0, 1), both bounded, so the conditions of Lemma 3.2 apply.
3.3 Linearly Guarded Labels Have Zero Covariance with the Features
The next theorem establishes the equivalence of conditions 3 and
Theorem 3.4. Let X be a random vector taking values in Rd with finite first moment, and Z a random vector taking values in {0, 1) with me-hot encoding. with each class probability IPZ j) being nonzero. Then the class class-conditional means EX = -] are all equal to the unconditional mean E[X] if and only if every component of X has zero covariane with every component 7, 7, i.e. the cross-covariance matrix Exz, whose ( entry is Cov Cov(X;, 7.j), is the zero matrix.
Proof. Since Z is one-hot, we can rewrite the (i. entry of Exz as:
E[X,Z,] - E|y = (Z = j) E = 51 E[X.1)
As P(Z = j) > 0, it follows that |E[, IZ = -] = B[X.] if and only if Cov(X Z,) = 0.
We have thus established the equivalence of the first four conditions stated earlier. See Appendix C for the last one, on statistical parity.
Least-Squares Concept Erasure
In Section 3 we that X linearly guards ZZ if and only if each component of X has zero covariance with each component of Z. We will now characterize the set of affine transformations r(x) = Pb such that r(X) linearly guards Z.
Theorem 4.1. Let X and Z be random vectors taking values in Idd and R respectively. with X of finite first moment. Then given some affine function r(x) Px + the modified random vector r(X) linearly guards Z if and only if the columns of the cross-covariance matrix Exz are contained the null space P.
Proof. From Theorem 3.4 we know that r(X) linearly guards Z if and only if Cov(r(X), Z)) is the zero matrix. By the linearity property of cross-covariance, we have:
Cov(r(X), 7.) Cov b, 7) PCov (X, Z) = PE
Therefore, r(X) linearly guards Z if and only if ker(P) 2 colsp Exz).

---

Implications for prior work. Notably, the above theorems imply that three previously proposed methods in the literature. Spectral Attribute Removal (SAL) [37]. Mean Projection [77. and Fair PCA
20, are guaranteed to achieve linear guardedness given suitable hyperparameters. See Appendix D
for further discussion.
4.1 Derivation of LEACE
Theorem [.1 is a very weak condition, which is far from identifying unique values for P and b. In most applications, however, we'd like to make "small" edit to X that useful information contained in maximally preserved. We operationalize the notion of a small edit in terms of the mean squared norm - defined by some positive-definite inner product M which can be thought of as a local quadratic approximation to any measure of divergence between X and r(X) (such Kullback-Leibler divergence, for example). While are primarily interested in the Fuclidean (M I) and Mahalanobis (M x norms, it will out that there is single erasure function that minimizes all such norms simultaneously. We will see in Section c that ensuring edits small in this sense provides substantial benefit to downstream task performance compared to other methods which also guard the labels Z.
Below, we derive the optimal eraser under the assumption that X and Z are centered.
Theorem 4.2. Let X and Z be centered random vectors taking values in Id and Ihth respectively, each of finite second moment. Let M E Raxd be a p.s.d. matrix defining a (possibly degenerate) inner product Rd: YM.y) = =x My. be the cross-covariacce matrix of X and Z. Let A+ denote the Moore-Penrose pseudoinverse of a matrix A, and let 1/2 be the p.s.d. square root of p.s.d. matrix A. Then the objective
My. Let Σxx E Raxd be X's covariance matrix, and Exz E Ba
argmin E |||PX- - X [ Pe
subject to Cov Cov(PX,2) 2) =0
has the following solution:
I-=I-W W,
where W the whitening transformation ( xxx) and PWexz = (WEx2)(WEx)) the
orthogonal projection matrix onto colsp Exzz)
Proof. See Appendices E. and B.2 for two independent proofs of Theorem 4.2
The above theorem assumes that the random vectors X and are centered, and does not include bias term. Below we extend our results to the uncentered case, and derive the optimal bias b*. Theorem 4.3. Let X and Z be random vectors taking values in Rd and R respectively, each of finite second moment. Define M and P as in Theorem 4.2 and b" [[ P=E[X] Then (P*, b*) minimizes E. P -x|p subject Cov(PX + b, 7) 0.
Proof. Let Rxdxd and define X X E[ and PIX[ +b B[X]. Then,
PX x = EE (PX x) M
= = X
M + 2E PX - Me
= E
PX
x
M + M"Mc,
where we have eliminated the middle term because P is linear and E[X] = 0. Since M is p.s.d., our objective minimized for i.e. b E| PE[X]. The problem thus reduces choosing so
as to minimize E PP x subject to Cov PX + b, 7) Cov 7) O. which Theorem 4.2 shows occurs when
P
*Our proof's also include degenerate "inner products" where M singular, and the associated seminorms.
5

---

Original Data
Whitening
Erasure
Unwhitening
Concept
Subspece
Orthogonal
Subspace
-2
Subspace
Class
Class
-5
Figure I: LEACE projection in 3 steps. First the data is whitened, ensuring equal variance in all directions. It is then orthogonally projected onto colsp( WExz) guaranteeing linear guardedness. Finally, we unwhiten the data so that its covariance structure mimics the original.
Putting together Theorems 4.2 and 4.3 and rearranging, we arrive at the LEACE formula:
TLEACE (æ) = x WtPWExa W( E[x])
(1)
Intuitively, LEACE de-means and whitens projects onto the subspace responsible for correlations between X and Z, then unwhitens the result. Finally, subtracts this value from æ, thereby surgically removing the linearly available information about Z.
4.2 Oblique Projections are Least-Squares Optimal
Prior work on linear concept erasure has assumed that erasure functions should be orthogonal projections 29 33 37 appealing the well-known fact that an orthogonal projection of a point onto a subspace U yields the nearest point in to æ. But even in the case where X is centered, "LEACE not an orthogonal projection in general. Orthogonal projection matrices are symmetric, and - W Pw Wwxx W is only symmetric in the special case where PWExz and W commute. It is an oblique projection however, since applying P twice yields the same result applying it once: 2WP wexz W + W PWWEx W W= = P
(P*)2 2
Orthogonal projections necessary and sufficient condition for linear guardedness, PExz = 0, is a constraint on the nullspace of P, and not on its range. We may freely choose the range of the projection
generally not least-squares optimal for concept erasure because the
minimize the mean squared distance, as long as we zero out colsp( Exx). In Figure [ an orthogonal projection would map all points onto the the dashed line, thereby preserving less of the variance of the original data than LEACE does (green line). See Appendix F for a concrete example.
4.3 Extension to Continuous Z
While not a focus of this work, it's worth noting that LEACE can also be applied to the setting where Z takes arbitrary values in RR as long
restrict ourselves the ordinary least squares regression loss L(n, z) = ||> 2I.2. In particular, the proofs of equivalence between conditions | and given in Appendix B make categorical assumption Z, and the equivalence between the optimality of zero weight matrix (condition 2 and zero croas-covariance (condition 4 is well known the OLS setting. We then apply Theorems 4.2 [.33] which also make categorical assumption, to derive the same optimal affine eraser as in the categorical case.
5 Evaluation
5.1 Intrinsic Evaluation
Following Ravfogel al. [32] we evaluate the ability of our method to remove gender information from the last hidden layer of a frozen BERT model. We use the biographies dataset of De-Arteaga [6). composed of short biographies annotated by both binary gender and profession. We embed each biography with the [CLS] representation in the last layer of BERT, enforce the same- conditional-mean constraint to remove gender information from the [CLS], and then evaluate the performance of the model, after the intervention, the main task of profession prediction. We

---

0.4
0.4
TPR-Gap
TPR-Gap
0.2
0.2
0.0
0.0
vilcigge
-0.2
-0.2
-0.4
-0.4
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
% women
% women
Figure 3: The correlation between GAP femate.y and the relative proportion of women in profession y. for BERT representation, before (left; R=0.867) and after (right; R=0.392) the projection.
compare our intervention with RLACE [32], which uses gradient -basee optimization to solve a linear concept-erasure adversarial game.
Concept erasure results. First, we evaluate the ability of logistic regression classifiers to recover the removed information. The results, presented in [Fgg. 2 show that method is the only to achieve random accuracy (perfect erasure) with a small edit, although RLACE (but not INLP) comes close. At the same time, method is around orders of magnitude faster, and does not require gradient-based optimization.
5.2 Downstream Fairness
How does intervention affect the behavior of the model the main classification task of profession prediction? We fit
logistic regression profession-rrediction classifier over the projected [CLS] representations.
Majority
0.9
Gender Accuracy
RLACE
To measure the bias in classifier, we follow De-Arteaga et al. 66) and use the TPR-GAP measure, which quantifies the bias in classifier by considering the difference (GAP) in the true positive rate (TPR) between individuals with different protected attributes (e.g. race or gender). We use the notation GAPTPR 2.y
0.8
0.7
0.6
denote the TPR-gap in main-class label y (e.g. "nurse" prediction) for some protected group (e.g. "female"), we also consider GAP TAPTPRRRMM the RMS of the TPR-gap across all professions for protected group z:
Mean Squared Error
Figure 2: Gender prediction ac- curacy after bias-removal projec- tion versus the mean squared dis- tance from original representa- tion for INLP, RLACE, and LEACE on BERT representations.
GAPTPRRRMS
Σ DEC
(GAPTTRJa
V
C
To calculate the relation between the bias the model exhibits and the bias in the data, we also calculate (GAAPTR C.CEAPPRR,,,mmeee 9[Women) the correlation between the TPR gap in given profession and the percentage of women in that profession.
The main-tass task classifier achieves profession- prreiictiooniiiiionnnn accuracy 77.3% on the pro- jected representations (compared with 79.3% over the original representations), indicating that the intervention minimally affects the ability to predict the profession of a person from the represen- tation of their biography. At the same time, the TPR gap drops significantly from 0.198 to 0.084, indicating sharp drop in the biased behavior of the profession classifier. Indeed, inspecting the correlation C(GAPTPM Women) between the gap (per profession) and the representation of women in
Results.
7

---

0.8
MUM Accuracy
0.6
Ours
MLM Loss
Random
03
INLP (20 iterations)
No Intervention
2
Layer of Intervention
Layer of Intervention
Figure Amnesic probing results on pert-base-uncant
this profession, see that this correlation plummets from 0.867 0.392 after erasure. Re-fitting the main-task logistic regression classifier over the projected representations yields slightly higher main-task accuracy 78.1% 16, the price of significantly increasing the TPR gap to 0.158
5.3 Revisiting Amnesic Probing
Elazar et al. [00] have introduced the idea of ammesic probing as a causal intervention that aims to test the importance of given concept (e.g. part-of-speech tag) modeling). They applied Iterative Nullspace Projection (INLP) to remove different concepts from the hidden representations of the model, and assessed the degree which its behavior changed when performing masked language modeling. Since INLP often requires dozens of iterations to completely erase the concept, its usage this context raises concerns of collateral damage due to magnitude of the intervention and the non-exhaustive nature of INLP removal. Here, we replicate their experiments on the bert-bar sast-bbas-uuncseed model with our interventions.
main task (e.g. language
Experimental setup. We use part-of-speech (POS) tags our concept of interest. We collect sentences and their coarse POS tags ("Noun", "Verb" etc.; 18 in total) from the English Universal Dependencies dataset (27). We tokenize the sentences with the BERT tokenizer and map each word- piece to the POS tag the word to which belongs. We collect the unmasked BERT representations for each layer, intervene to linearly erase the POS concept from that layer, and continue the forward pass until the last layer, from which we compute the distribution of the MLM over the vocabulary. Note that in each experiment we intervene following the intervention, as well as the increase in the loss. compare with a baseline intervention of random orthogonal projection whose null space has the same rank the label space (18). For INLP, we perform iterations. This needed because INLP does not effectively remove the concept; after 20 iterations, classification accuracy is above majority accuracy. As result, INLP reduces the rank of the representation by 360. By contrast, our method decreases the rank just by 17.
single layer. We quantify the decrease in accuracy
The results are shown in Fig. 4b Our intervention only mildly changes BERT LM accuracy and loss until layer 8, with the highest drop recorded layer 11. INLP, in contrast, shows maximum effect at layer 6. Since it removes hundreds of dimensions, is difficult to attribute this effect to the erasure of the concept. These results suggest that the causal effect of the POS concept on the language model is concentrated in layer 11. Interestingly. this stands in contrast with POS linear probing results, which are optimal at earlier layers (88. As Elazar et al. [10] have noted, probing does not generally correlate with intervention-based analysis techniques.
Results.
'The softmax probabilities of a multiclass logistic regression classifier can leak the removed information if another classitier is stacked top of it [3]], though this setup not linear.

---

Concept Scrubbing
Algorithm 1 Concept scrubbing
Unfortunately, Elazar al. [TO] were forced to limit their interventions to a single layer due to the limitations of INLP. INLP often requires the deletion of several dozen dimensions before linear guarding is achieved as demonstrated in Figure 2 Kumar et al. [21] show empiri- cally and theoretically that INLP causes need- less "collateral damage" to useful parts of the representation that are orthogonal to the concept being erased. Because of this collateral damage,
Require: Model with layers f ................................
Require: Design matrix E I
Require: Label matrix Z E R
Ensure: LEACE parameters for each layer in
I: HI Embed(()
2: L 4-list()
3: for I 1... do
4: Fit (P, b) on H/ and Z
5: Append (P, b) to L
6:
Hi + P(H: (H, + μn. (Eq. D
it's impossible to apply INLP to multiple layers of transformer without causing its outputs to collapse into gibberish.
7: H + (H,)
8: return
Instead, we would like to erase all linear information about concept in every intermediate rep- resentation, which we term concept serubbing. LEACE makes concept scrubbing possible and eminently practical. It causes minimal collateral damage, induces little computational overhead, and the covariance statistics relies on can be computed in streaming fashion, without ever storing all the hidden states in memory or on disk.
Algorithm. Any intervention on the model at layer & changes the distribution of hidden states layers f' > Because of this, the naive approach independently fitting LEACEE parameters (P, for all layers of the clean model, then applying them all at once, may fail to fully erase the target concept. Instead, we fit LEACE parameters sequentially, starting from the first layer and proceeding the final layer. After we compute (P, b) for layer, immediately use them scrub the hidden states for that layer, then feed these scrubbed representations to the next layer (Algorithm) D
LLaMA
Pythia
Condition
7B
13B
30B
160M
1.4B
6.9B
12B
No intervention
0.69
0.66
0.62
0.90
0.70
0.64
0.62
Random erasure
0.69
0.66
0.62
0.99
0.72
0.66
0.63
LEACE 1.73
1.84
1.96
2.79
2.25
3.57
3.20
SAL
3.24
3.26
3.16
3.53
3.44
4.17
4.69
unigram entropy
2.90
2.90
2.90
2.66
2.66
2.66
2.66
Table 1: Perplexity in autoregressive language models when removing linearly available part-of- speech information from the input to each transformer layer. Units are bits per UTF-8 byte. The unigram baseline assigns probabilities to tokens based only on their frequency and not on the context.
6.1 Experimental Details
Dataset. For each model family, we use sample from the respective pretraining distribution: the validation split of the Pile [13] for the Pythia models 22) and the RedPajama replication of the ILaMA pretraining corpus for the LLaMA family [39]. sampling a slice of 222 tokens for fitting the LEACE parameters and another slice of 22 tokens for evaluation. Since neither corpus comes
with part-of-speech tags, we use the model from the SpaCy library [19] to automatically generate
Universal Dependency tags [2
Baseline method. We also run concept scrubbing using full-rank SAL 37 which is similar to our method but lacks bias term and does not adjust for correlations between features (Appendix (D).
Architecture. We focus on autoregressive language models. evaluate our method on EleutherAl's Pythia 160M, 1.4B, 6.9B, and 12B models (2], and Meta's LLaMA 7B, 13B, and 30B (3 We apply concept erasure to the input of each transformer block, immediately after normalization is applied (LayerNorm RMSNorm).

---

Randomized erasure. Almost any intervention on neural network will cause its performance to degrade to some extent. Following Elazar al. [TO), we isolate the effect of the concept erasure by comparing it to control condition in which we orthogonally project onto random linear subspace of the same rank as the cross-covariance matrix. To reduce the variance of our results, we sample fresh subspace for each minibatch, and erase that subspace at each layer, reporting the cross-entropy loss averaged over subspaces.
Training efficiency. Algorithm I avoids redundant computation by caching the layer hidden states for every data point, then using them to run layer i + 1. This approach has the downside of requiring a large amount of memory or disk space during training (up to 500GB in our experiments). It's possible to avoid caching any hidden states and instead recompute them as needed, at the expense of increasing the total compute cost from (e) to (
6.2 Results
We find strong evidence that autoregressive language models heavily rely on linearly encoded part-of-speech information. While erasing randomly selected subspace has little no effect on language modeling performance, scrubbing away part-of-speech information induces large increase in perplexity across all models (Table
The specific numbers, however, depend on the erasure method used: SAL induces significantly larger increases in perplexity for all models we tested. We take this to mean that SAL inflicts more collateral damage on other useful features in the representation than LEACE does. In other words, interventions made with LEACE are more surgical than those made with prior work; they more closely approximate the ideal of perfect intervention which only erases the target concept and keeps everything else fixed 40 15 If this experiment were conducted with SAL alone, we would have overestimated the causal effect of part-of-speech.
Limitations and Future Work
Much work remains to be done to validate concept scrubbing. Specifically, we'd like to see ex- periments that target concepts much narrower than part-of-speech, and use behavioral metrics to determine whether scrubbing changes the network in the ways we'd intuitively expect. If these experiments succeed, an exciting next step would be the incorporation of concept scrubbing into the pretraining and/or finetuning process. This may make it possible to train deep neural networks subject to conceptual constraints. It remains to be seen if gradient- pradient-based optimizers will be able to "circumvent" such constraints by learning completely nonlinear representations of protected attributes.
In this work, we focused exclusively on linear concept erasure due to its simplicity and tractability. Some authors have proposed nonlinear concept erasure techniques based on kernel methods, but have found that erasure functions fit using one kernel do not generalize well to other kernels [33] We conjecture that it is intractable to nondestructively edit so as to prevent a general nonlinear adversary from recovering Z, unless the data generating process for X is known in detail.
major motivation of concept erasure is that it promises prevent models from using a concept in a post hoc, model-agnostic fashion. But if our concept scrubbing procedure turns out to yield unsatisfactory results in practical use cases, the most promising research direction might then be to improve model-specific techniques, such as those that modify the training procedure [ 14]
8 Acknowledgements
We are grateful to CoreWeave for providing the compute resources used in Section 6 grateful to be supported by the Bloomberg Data Science PhD Fellowship.
Shauli Ravfogel
"We suspect erasing concept is at least as hard as extracting it from the original representation. But in the worst case, information about Z could be encoded cryptographically in X, which would be intractable decode given standard computational complexity assumptions. If the data is generated by a known algorithm, however, it may be possible to efficiently eliminate mutual information between Z and X by simply breaking the links in the causal graph that connect them.
10

---

References
[1] UC Berkeley. The Hilbert space of random variables. Lecture Notes Electrical Engineering 126, 2018. URL.tttp::/hnst.eess .ew..ee.e.bbb.bb.e..e.....e..... wwtp::eddddded//eee/ee http:///gg..cccce/eeee/////////p
[2] Stella Biderman, Hailey Schoelkopf. Quentin Authony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: suite for analyzing large language models across training and scaling. arXiv
preprint 2304:23300.03333, 2023.
[3] Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer woman is to homemaker? De- biasing word embeddings. Advances Neural Information Processing Systems, 29:4349-4357, 2016. URL https://proceeddnggs httpa//ppp..cc/ppp//////////////
44886177744331777771224444666663 Paper.
[4] Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. Adversarial deep averaging networks for cross- linguul sentiment classification. Transuctions of the Association for Compulational Lingwistics, 6:557-570, 2018. doi: 10.1 10.1162/tac____aeee a_00939, URL https
/aclanthology.ortt www.:olooooy.gg/11-1--/ orggQQ8--033
[5] Verna Dankers, Christopher Lucas, and Ivan Titov. Can transformer be too compositional? Analysing idiom processing neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 36266 2022.
[6] Maria De- Dr-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexan- dra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency. FAT* '19, page 120-128, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978 01888155661255 doi: org/10. 10. 1141/1111771 2228766000277777 3287572
01111445227755032287755555
URL https:
[7] Sunipa Dev, Tao Li. Jeff M. Phillips, and Vivek Srikumar. OSCaR: Orthogonal subspace correction and rectification of biases in word embeddings. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 5034-5050, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Lin- guistics. doi: 10. 111813611/2021111.. 1e1.0211emmlpmmaaaae m11n.411. URL emnlp- 1aailppmain..111
2021
[8] Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In In- ternational Conference in Learning Representations, pages 1-14, May 2016. URL https / //arxiv. Marxiv.org/aat httg/aab/15511..555
[9] Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11-21, Brussels, Belgium, October-Novemtee 2018. Association for Computational Linguistics. doi: 10. (..11863/v1/DD 1002. URL http:://waaaaatoooooooooo///---/
[10] Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 175, 2021. doi: 2rg/2021.tacl1111.
00359. URL https: / pattps://accanan aclanthology
[11] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Hatfield Zac Dodds, Danny Hernandez, Andy Jones, Jackson Kermion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
[12] Thomas S. Ferguson. Mathematical Statistics. Academic Press, Cambridge, MA, 1967.

---

[13] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The Pile: 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2 100.0000000002,, 2020.
[14] Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In International Conference on Machine Learning, pages 7334-7338. PMLR, 2022.
[15] Christopher Grimsley, Elijah Mayfield, and Julia Bursten. Why attention not explanation: Surgical intervention and causal reasoning about neural models. 2020.
[16] Pantea Haghighatkhah, Wouter Meulemans, Bettina Speckmann, Jérôme Urhausen, and Kevin Verbeek. Obstructing classification via projection. In Filippo Bonchi and Simon J. Puglisi, editors, 46th International Symposium on Mathematical Foundations of Computer Science, Leibniz International Proceedings in Informatics, LIPIcs. Schloss Dagstuhl Leibniz-Zentrum für Informatik, August 2021. doi: AF2
[17] Pantea Haghighatkhah, Antske Fokkens, Pia Sommerauer, Bettina Speckmann, and Kevin Verbeek. Better hit the nail on the head than beat around the bush: Removing protected attributes with single projection. arXiv preprint urXiv:2212.04277. 2022.
[18] Evan Hernandez and Jacob Andreas. The low-dimensional linear geometry of contextualized word representations. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 82-93, 2021.
[19] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, Adriane Boyd, et al. spacy: Industrial- strength natural language processing python. 2020.
[20] Matthaus Kleindessner, Michele Donini, Chris Russell, and Muhammad Bilal Zafar. Efficient fair PCA for fair representation learning. In International Conference on Artificial Intelligence and Statistics, pages 1250-5270, PMLR, 2023.
[21] Abhinav Kumar, Chenhao Tan, and Amit Sharma. Probing classifiers are unreliable for concept removal and detection. arXiv preprint arXiv:2207.04153, 2022.
[22] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
[23] Ryan McDonald, Joakim Nivre, Yvonne
Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, et al. Universal dependency annotation for multilingual parsing. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92-97, 2013.
[24] Neel Nanda. Actually, Othello-GPT has a linear emergent world model, Mar 2023. URL htt.://nnnnttto..ooopoeeeeeeeeee
(25) Vassilina Nikoulina, Maxat Tezekbayev, Nuradil Kozhakhmet, Madina Babazhanova, Matthias Gallé, and Zhenisbek Assylbekov. The rediscovery hypothesis: Language models need to meet linguistics. Journal of Artificial Intelligence Research, 72: 12:3334-33,4 2021.
[26] Hamed Nilforoshan, Johann D Gaebler, Ravi Shroff, and Sharad Goel. Causal conceptions of fairness and their consequences. In International Conference on Machine Learning, pages 16848-16887. PMLR, 2022.
[27] Joakim Nivre,
de Marneffe, Filip Ginter, Jan Hajic, Christopher D Manning, Sampo Pyysalo. Sebastian Schuster, Francis Tyers, and Daniel Zeman. Universal dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4034-4043, 2020.
[28] Judea Pearl. Causality. Cambridge University Press, Cambridge, UK, 2 edition, 2009. ISBN 978-0-521-8958 doi: 10.1017/CB09780511888111
12

---

[29] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online, July 2020. Association for Computational Linguistics. doi: 10. 32653////20000allmmmmaa peattps://aclanthhote
org/2020..... 2020
647
[30] Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 194-209, 2021.
[31] Shauli Ravfogel, Yoav Goldberg, and Ryan Cotterell. Linear guardedness and its implications. arXiv preprint arXiv:2210. 10012, 2022.
[32] Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning, pages 18400-18421, PMLR, 2022.
[33] Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 6034-6055 6055, 2022.
[34] Bashir Sadeghi and Vishnu Boddeti. On the fundamental trade-offs in learning invariant representations. arXiv preprint arXiv:2109.03330 2021. URL https:://pperreiieeet net/
pdf
[35] Bashir Sadeghi, Runyi Yu, and Vishnu Boddeti. On the global optima of kernelized adversarial representation learning. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7970-7978. IEEE, 2019. URL http::/halll.... cse msu. www.assets/aaeeeeeeeeeeeeeee / papers/2009-Lcc--eerreel www.eeeaaaaaarrraaaaaaaaaaaaaaae wtg.aaaaaaaaaapppppeeeeaaaaaaaaa
påf
[36] Shun Shao, Yftah Ziser, and Shay Cohen. Erasure of unaligned attributes from neural represen- tations. arXiv preprint arXiv:2302.00 02997, 2023.
[37] Shun Shao, Yftah Ziser, and Shay B. Cohen. Gold doesn't always glitter: Spectral removal of linear and nonlinear guarded attribute information. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1611-1622 Dubrovnik, Croatia, May 2023, Association for Computational Linguistics. URL https /aaclanthology org/ 2023.0222222222
118
[38] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 4603-4601,220 2019.
[39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[40] James Francis Woodward. Making Things Happen: A Theory of Causal Explanation explanation. Oxford University Press, 2005.
[41] Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D Goodman. Interpretability at scale: Identifying causal mechanisms in Alpaca. arXiv preprint arXiv:2305.08809 2023.
[42] Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. In Advances in Neural Information Processing Systems, volume 30, pages 585-596. 2017. URL https: acm. /tt/////1/0..655297711111 3294827
[43] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint X1x:220022.1088 2020.
[44] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAT/ACM Conference on AI, Ethics, and Society, page 335-340, New York, NY, USA, 2018. Association for Computing Machinery, ISBN 978 7788140330122 28. doi: 10.1145/378872213278777 URL, httpp:://dot org/10...110. /0/ 1145/ 3278721. 32777779
13

---

A Additional Related Work
The problem of linear concept erasure is an instance of the general problem of information removal. Information removal methods generally divide into adversarial methods, which applied during training, and the post-hoc linear methods considered in this paper. Adversarial methods 8 42 9 44] gradient-reversal layer during training induce representations that do not encode the protected attribute. However, Elazar and Goldberg [ have shown that these methods fail in exhaustively removing all the information associated with the protected attribute: it is often possible to train new adversaries that successfully recover the removed information. Linear methods have heen proposed tractable alternative, where identifies linear subspace that captures the concept of interest, and neutralizes it using algebraic techniques. Different methods have been proposed for the identification of the subspace, e.g. PCA and variants thereof [3 20 orthogonal-rotation
classification-bass. [29]. spectral [37 36 and adversarial approaches 32
Few works theoretically characterize the condition of linear guardedness. Haghighatkhah et al. [16 extensively analyzed the problem of preventing linear classification, with the focus on decreasing accuracy. They provide constructive proof of optimal intervention for SVM classifier. Ravfogel et al. [31] have proposed a formal definition of linear guardedness based on V information, and characterized the fairness implications of guardedness; show the relations with definition above. Ravfogel et al. [32) provide an adversarial formulation of the problem, derive a closed-formed solution certain cases, and propose an SGD-based optimization for others. While they seek an orthogonal projection, we empirically showed that their solution is very close to ours. Sadeghi et al. 35 and Sadeghi and Boddeti [34] both study adversarial formulation of concept erasure for linear regression, and they trade-off with main-tas ttask performance. In contrast to Ravfogel et al. [2), they consider a general linear adversary, i.e. not necessarily projection matrix. Closest to our work Kleindessner et al. [20]. Haghighatkhah et al. [I7]. Shao et al. [37] As we showed above ( 4), those methods do achieve the goal of linear guardedness though they are unable to prove this fact. At the same time, they are not optimal in terms of damage to the original representation space.
B Equivalence of Guardedness with the Optimality of Constant Predictors
The following theorems establish the equivalence of conditions I and 2 (indeed, they do so in the general setting, with no assumption of convex loss or linear predictors).
Theorem B.1. Suppose X (V, -)-guards Z. Then for every loss E L, the corresponding
trivially attainable loss I (7..C) cannot be improved upon by any predictor n(:0) E V, i.e.
L, info ((;(n(X;0), ()).
Proof. Consider the null random vector ('(w) O. Since all predictors are constant on X', and the trivially attainable loss gives the best available expected loss among constant predictors, we must have:
inf E[f(
The right side of equation (4) is the best possible loss achievable by function 17(:: 0) on the joint distribution of (X', Z), which by the definition of guardedness is upper bounded by the best possible loss achievable on the joint distribution of (X, Z):
inf [L((((XX 0), 2) s inf [[C(n(X))) 0
0
Combining equations and 5 and the fact that all constant functions exist in our function class = {t((;;0) 0)}, we arrive at our desired result:
L = inf E[C(((X))))
9
Theorem B.2. Suppose that for every loss C E &, the corresponding trivially attainable loss (2.2) cannot be improved upon by any predictor 11 0) E V, i.e. L,p = infg E[L(n(X;0)) 7)). Then X (V, 1)-guardss2 Z.
14

---

Proof. Let X' : - Rd be any other random data vector with finite first moment.
Since all constant predictors exist in our predictor class V {nt( 0)}. ). the best loss achievable (X', 7)) by functions V must be at least as good as the trivially attainable loss (the best loss available
by such constant predictors):
inf E[Cn((X 0), S Lr
By assumption, the trivially attainable loss cannot be improved upon over (X, 7. by predictors in V:
Lr inf ())
Since our choice of X' was arbitrary, this shows that X maximizes the minimal achievable loss, so X (V,2)= -ugarrds Z.
C Linear Guardedness is Equivalent to Linear Statistical Parity
To measure the effect linear guardedness on main- task classifiers, we use the following minimal definition of "fairess" with respect to an attribute, adapted from Edwards and Storkey [8]
Definition C.1 (Statistical Parity). Let X and 7. be defined as above, and let f be function with domain Rdd Then f exhibits statistical parity with respect to when evaluated on if
Vz Z EZS(X) - [E[(()).
We now prove the equivalence of conditions 3 and
Theorem C.2. Let X and Z be defined as above. Then every linear predictor f(x) = b Wx exhibits statistical parity w.r.t. ZZ when evaluated on X. if and only if each class-conditional mean X = is equal to E XX.
Proof. Suppose each cass-condittioal mean E [ is equal to E[x]. Then by the lincarity of expectation, we have for all Z:
Z 2] = WX b|Z WE|X|Z 2] b WEX b E[f(X).
This matches the definition of statistical parity provided in Definition C.I
Conversely, suppose every linear predictor f(x) b Wx exhibits statistical parity w.r.t. Z when evaluated on X. Then this holds for the identity function id(x) = x, and thus for all 2 E 2:
E 2] = [Z = [()] E[
D Implications for Prior Work
which characterizes the necessary and sufficient conditions for an affine erasure function yield perfectly linearly guarded dataset, for methods proposed in prior work.
In this section we discuss the implications of Theorem 4.1
Spectral Attribute RemovaL (SAL) 37 uses the top left singular vectors of Exz construct an orthogonal projection matrix QSAL - U... U which is then applied to X. Notably, while n is presented as a free parameter their method, all of their experiments involve binary classification problems where Z is a one-hot vector, and n is set to a value no greater than 2. We'll call the version of SAL where n rank Σxz), "full-rank SAL." Since these left singular vectors are orthonormal basis for Σxz' column space, Theorem 4.1 implies that full-rank SAL guarantees linear guardedness.
Mean Projection (MP) 17 orthogonally projects X onto the orthogonal complement of the span of the difference in class centroids E =] E[X]Z = Oj, where is assumed to be binary. Since the centroids are equal after the projection, this method guarantees linear guardedness by Theorem In fact, by Theorem 3.4 MP is mathematically equivalent to SAL when Z is a one-dimensional random vector taking one of two possible values.
15

---

E Derivation of LEACE
Theorem 4.2. Let and ' be centered random vectors taking values in I& and IR respectively, each of finite second moment. Let M E Rdxd be a p.s.d. matrix defining a (possibly degenerate) inner product on Rd: (x.))M = x T My. Let Σxx E R be X's covariance matrix, and Σxz E Rd be the cross-covee cosarccveee matrix of X and Z. Let A denote the matrix A, and let 1/2 be the p.s.d. square root of a p.s.d. matrix A. Then the objective
pseudoinverse of
argmin E
||IXX - [
subject to Cov =0
PERexd
has the following solution:
P- W P PWEx2 W,
where W is the whitening transformation (22/2)+ and PwExz = (WExz)(WEx)) is the
orthogonal projection matrix onto (elsp(WExz)..
Below are two independent proofs of Theorem
E.1 Algebraic Proof
Proof. We shall first show that, in any orthonormal basis each row P; constitutes an independent optimization problem, and then select basis in which we can easily show that the corresponding component X; of X can be almost surely decomposed into linear combination of mutually un- correlated components in the whitened random vector WX, some of which correlate with Z and some of which do not. The solution (PX); is then that same linear combination, restricted to those components which do not correlate with 7.
Consider first an orthonormal basis diagonalizing the inner product M, so that (x, Y/M = E 0: for fixed aq...., Old 0. This allows to treat each P. Rd of P separate optimization problem.
argmin E (P, x PIER"
subject to Cov(P: 7 X, 2) 0,
at which point the weights as of each subproblem become irrelevant, and our objective may as well view each row independent optimization problem not just in this
be Fuclidean, allowing basis, but from any convenient one.
So now let é raak(Exzi consider a (new) orthonormal basis whose first m coordinates span the column (and row) space of
rank Ewxz)) and rank(Exx:
rank(Ewr wx), and
W (i.e. the subspace of d in which X and WX have nonzero variance), and whose first E S coordinates span the column space of Ewx,z (i.e. the subspace of Rd which WX has nonzero covariance with Z).
Any component X can be (almost surely) written as a fixed linear combination of the nontrivial components of its whitening WX:
m
X; = (W): Σ w (WX),.
(almost surely)
1=1
Meanwhile, any component of PX can (always) written as fixed linear combination of the nontrivial components of WX and the almost surely zero components of X:
(PX), = Σ Ag (X),
+ Σ B,y,,,
1=1
j=m+1
i.c. P AW BV, where V - W W is the orthogonal projection onto X's almost surely zero components.
TTroouhhout this proof. we abuse the notations X,, Pt. etc. to refer to the th component in the specified basis, not necessarily the standard one.
16

---

The 1th sub-objective is then:
2 Ag - (WX))
E 2 x,)
E
==1
where we have safely ignored the almost surely zero terms B, (j > m), and used the fact that the first components of WX have identity covariance matrix.
PX is almost surely equal to AWX, so our constraint Cox 2) equivalent to A A = Cov( (ArX, Z) = O, i.. As = O when j s E, since the first & components are those for which WX correlates with Z. Subject to this, the objective is minimized for Aij = W when j > l, i.c.
A = -= -
The particular choice B I gives our solution P I Pwxx. W, leaving the non-varying components of X intact (see Fig. for a visualization).
The solution is unique except for columns corresponding to the components of with zero variance, and rows corresponding to the zero-weighted components of the (pseudo) inner product M.
E.2 Covector Proof
Proof. We assume without loss of generality that vectors in Rd represented in a basis diagonalizing the inner product M, so that (x. y)M Ei=1 m, for fixed m...... md 2 This allows us to treat each row Pı E Rd of P separate optimization problem,
argmin E m: -x)x subject to Cov - x, 2) =0 0.
PIER
Our objective only depends on Pj through its effect on the sealar random variable & = P,7'.. All random variable of the form Ç uf X for some covector u I form a vector space U, which we equip with the covariance inner product (E, Cov Cov(g. E[çç] un Exxx..
By the linearity covariance, the elements of U uncorrelated with form subspace Z C U. Note also that Ç E Z if and only if E covector uz satisfies Cov(u? Z) uu Exz 0g, and that these covectors themselves form the subspace colsp(Exzi) of R
Our objective now reduces to finding a covector P7 that defines the orthogonal projection of X, onto z-. The difficulty is that orthogonality of elements in U is not equivalent to orthogonality of the corresponding covectors. We fix this by changing the basis in which covectors represented. Since X E colsp( W) a.s., we can write any element of U as a linear form in WX rather than X by applying the change-of- -asss u = W us to every covector: & == WX uf WXX a.s.
In this new basis, which is orthonormal under our covariance inner product, each component of X is W and the inner product of any two elements of U is simply the Euclidean inner product of the corresponding covectors
written X; = (W+), WW
(E, Cov = Cov(u WX, U WX) = u
IT XXEx = u (T u'.
Since the two inner products are now equivalent, and Z is precisely those random variables with covector u' E colsp( [ Exz) the orthogonal projection of X; onto - is also orthogonal projection of its covector (W+)? onto colsp (
X; = (W+); (I Pwe.,) (XX)
Putting all the components of X together, we have our final solution,
x = W WWwEx W)X,
which is almost surely equivalent to Eq.
6 but keeps the non-varying components of X intact.
Strictly speaking, equivalence classes of almost surely equal random variables. "If Σxx is full rank, there a one-to-one correspondence between random variables in U and covectors. In the singular case, we may choose the component of the covector inside ker (Exx) arbitrarily, since it will make no difference the inner product.
17

---

F The Optimality of Oblique Projections
As noted in subsection 4.2, the optimal affine erasure function r(x) b Px does not in general use an orthogonal projection for the matrix P. A simple example illustrates why. Let d = 2. k = so that X takes values in R2 and Z takes values in IR, with the first feature X₃ and the label each independently and uniformly distributed in {-1, 1. +1), and the second feature X2 simply equal to the sum X2 X, Z. dataset reflecting such distribution has four (x.y) pairs:
(1),),,)), (1100",-1)
((-1,07,, ,1,
(11,--2².-11)
In this case, all of the information X has about Z resides in X2, so the minimally disruptive orthogonal projection which guards Z will nullify that component:
1
0 0
P ortho
On the other hand, 11 contains some information about X₂ (despite having no information about Z), allowing a partial reconstruction of X2 while preserving full concept erasure:
] 0 0 0
P Poblique
Both methods fully erase the ability to predict from the data, however simple calculation shows the second, oblique method to perform better measured by mean squared edit distance:
Porbo X - x||2 = 2, E P Poblique
X||
G Equivalence of Guardedness Definitions
Xu et define the conditional V-entropy of 7 given as the lowest achievable cross-entropy loss predicting Z with a function of X in the predictor class V. In our notation:
(z X) = inf E(C(n(X;00,,) Z),.
#E
where -(n,=)= log
exp(
is the cross-entropy loss function.
Ei= exp(n=)
They then define the (unconditional) V-entropy (y(Z) = Hv(Z 0) to be the lowest achievable cross-entropy loss in the of constantly null random data variable. This is exactly trivially
attainable loss L, DDfinition 2.2).
Finally, they define the V-information from X to Z. as the reduction in V-entropy as compared to using such null random data variable:
Tv(X + Z) (Z) Hv(Z
Using these notions, Ravfogel et al. 31 say that X is e-guarded with respect to Iy + Z) < e.
In Appendix B. showed the equivalence of guardedness (as we have defined it in Definition 2.1 to the optimality of the trivially attainable loss. That is, X (V, -ugrrss when (Z = T,p (Z),,, in the case where & is the singleton class consisting solely of the cross-entropy loss function. In the language of [3]], X is e-guarded with respect to V for all > 0.
H Constraining Norm Growth
In early concept scrubbing experiments (Sec. 6). we found that specific layers some models, concept scrubbing with LEACE would cause the norm of the representation to diverge, leading to NaN outputs. By contrast, SAL never caused divergence, even though causes a larger disruption to model performance average (Table IT. This is because SAL uses an orthogonal projection Q. whose eigenvalues are thus all {0. T). so the norm of the hidden state can never increase after erasure, while LEACE's oblique projection matrix P does generally have singular values greater than To combine the superior average-case MSE of LEACE with the stability of SAL, adopt simple regularization heuristic. After constructing P. we analytically compute the
18

---

trace of the covariance matrix of the hidden states after applying P. If tr tr tr tr(Exx), we solve a quadratic equation to find the convex combination P aP + a such that tr(Exx) tr( (P''xx(PP))) By Theorem 4.1 the set of matrices which ensure linear guardedness so P' is guaranteed to be the feasible set. Furthermore, since our mean squared error objective is convex, P' is guaranteed to have no worse MSE than Q We find this solves the divergence issue in practice.
convex 10
I
Oracle LEACE
X;
The concept crasure method derived in Section 4 does not require access concept labels at infer- ence time. That is, we can fit an crasure function on a labeled training dataset, then apply the func- tion to unlabeled datapoints. If we have oracle access the label for each æ; we can achieve an even more surgical edit. In Theorem I.] be- low, we derive Oracle LEACE, a closed-form formula for the the nearest X' to any X such that Cov(X', 7.)
Z1
X; X, proi,x; X;
proj X,
span Z
Zz
Like in Sec. the resulting XLEACE is "nearest" with respect all p.s.d. inner products a Mb defined on Rd simultaneously. This is because, by expressing in basis that diago- nalizes M, we can decompose the problem into independent subproblems, one for each com- ponent of X; Each subproblem can then be viewed as an orthogonal projection, not in Rdd
Figure 5: Orthogonal projection of 7th component of X, itself a vector in the random variable Hilbert space H. onto the span of the components of Z. The residual X; proj is the closest vector to X, orthogonal to, and hence uncorrelated with, Z = span( [Zz, Zz)).
but in an abstract vector space of real-valued random variables. For geometric intuition, see Figure
Prior work has noted that computing an orthogonal projection in a random variable Hilbert space is equivalent to solving an ordinary least squares regression problem []]. Our theorem is a natural extension of this work: find that Xf,ACCR is equal to the OLS residual from regressing X Z. plus constant shift needed to ensure that erasing Z does not change the mean of X.
Theorem I.1 (Oracle Concept Erasure). Let H be the Hilbert space of square-integrable real-valued random variables equipped with the inner product (E, C) == 18[[C). Let (X, Z) be random vectors in Hd and 4² respectively. Then for every p.s.d. inner product (a, b) = a Mb on Rd, the objective
argmin E|| -| -X NER
subject to Cov (X', Z) 0
is minimized by the (appropriately shifted) ordinary least squares residuals from regressing X on Z: XLEACE = X Exz Eitz I[Z)).
Proof. Assume w.l.o.g. that and X' are represented in basis diagonalizing M, so we may write
E E
M Σ my [[X] x.) -X))1.
where mi,..., mi 2 0 eigenvalues of M. Crucially, each term in this sum is independent from the others, allowing us to decompose the primal problem into d separate subproblems of the form ||X 12 one for each component i of (X.
Factoring out constants. Now consider the subspace C = span (1) C H. consisting of all constant
(i.e. zero variance) random variables. Orthogonally decomposing X, along C yields X, = X, + pi. where = [X) E C and X, = X EE E C-1, and likewise for X/. Our objective is now
X x, 2 ||4
mall2 + | X1
In fact, it is subspace of pdd For any matrices A, B R add such that AExz 0 and Byxz 0. we have by linearity (aA + 3B) Exz aAExz + (BExz a0 30 for any scalars and B.
19

---

Since n and N4 are orthogonal to X and X,, and the constraint Cov(X', 2) 0 is invariant to constant shifts, we can optimize the two terms Eq. independently. The first term is trivial: is minimized when μ4 = pis and hence X X E
Orthogonal projection. We can now rewrite the zero covariance condition as an orthogonality constraint on X,. Specifically, for every i E (1...d} we have
argmin || - x s.t. Vj (1....k) (8, 2i) 0, X
(8)
where 7 Z E(Z]. In other words, we seek the nearest X4 X orthogonal
}),), which is simply the orthogonal projection of X, onto z+. This turn is equal to the ordinary least squares residual from regressing X on Z:
X X; -rooj (X, 2) = X; - (Exz): D zZ - E(Z) - E[
Putting it all together. vector form yields
Plugging Eq. 9 into X X E[X,] and combining all components into
XLEACE =X Exz (Z E[Z]),
(10)
which completes the proof.
J
Notation Key
The space of one-hot labels ((21.... {0, 13*h (treated interchangeably with the integers {1..... k} when convenient). Integrable (i.e. finite first moment) random vectors taking values in Rd and R respectively (or their realized values inside an expectation, e.g. in (X)(()). 7. is sometimes restricted to the one-hot labels 2, in which each P(Z. = j) > O.
=
X,Z
assume
X,, 2 The 7th and 1 components thereof, themselves scalar random variables (or their realized values inside an expectation).
Scalar random variables taking values R. predictor function Rd + Z (or its value rx)) when inside an expectation). A space of predictor functions {n(-; 0) I2 H 0 E θ), parameterized
77
V
by and containing all constant functions.
space of loss functions {C R x 2 - (0,
An erasure function Rd Rd, hopefully making minimal edit X that eliminates the ability to predict labels with predictors in V.
A
A matrix with entries in IR.
Agj
The entry thereof at the 2h row and jth column.
A
The Moore-Penrose pseudoinverse of A.
A column vector with entries in R.
U The th component thereof.
20


Reviews:
[{"review_id": "G9lAw2rS7g", "reviewer": "Reviewer_rafU", "summary": "This paper studies concept removal from features. For linear classifiers, the authors prove several equivalent characterizations of linear guardedness (reducing the accuracy of any linear classifiers to the trivial accuracy). In particular, the features that achieve linear guardedness have zero covariance with the labels.\n\nEquipped with the new characterizations on the linear guardedness, the authors propose a new formulation of concept removal. Specially, the formulation finds the linear map minimizing the Euclidean distance such that the features after the linear transformation are linearly guarded. The formulation yields a quadratic program with a single linear equality constraint, and thus enjoys a closed-form solution.\n\nThe author conducts three experiments to verify the effectiveness of the method. The experiments on removing gender information from BERT shows that the proposed method successfully removes gender bias while preserving the classification accuracy.\nThey also apply the proposed method removing part-of-speech tag information from language models to investigate the importance of POS tag on language modeling tasks.", "questions": "1. In Figure 1 left, why does random projection have even higher accuracy compared to no intervention? I would expect random projection to be the same or slightly worse than no intervention. Figure 1 right does show they have similar loss.\n2. The definition 2.1 might have an issue. The maximization right after Line 56 should be over all possible joint distributions over features and labels that has marginal distribution the same as $Z$.", "limitations": "This authors have listed their limitations in the paper which have been addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of minimizing the Euclidean distance seems natural. The characterization on the linear guardedness is sound and useful. Importantly, this paper reveals connections with previously published methods and thus may strengthen the understanding of concept erasure. For example, the theoretical characterization justifies that SAL, Mean Projection and Fair PCA all achieve linear guardedness. Moreover, it is interesting that RLACE has the same top eigenvector with the method proposed in this paper.\n", "weaknesses": "1. The writing overall is good but the structure of writing can be improved. Theorem 4.3 strictly includes Theorem 4.2 as a special case. There is little benefit to separately present Theorem 4.2. Thus, they should be merged. Additionally, I find most of the proofs are not that insightful and thus it's better to defer them to the appendix. The extra space can be used to expand the preliminaries section as I find the current presentation is not entirely clear for audience who's not familiar with the topic.\n2. One experiment that I am curious about is the part-of-speech tag prediction accuracy after concept scrubbing. As LEACE only guarantees linear guardedness, it is unclear from Section 5 and 6 whether non-linear classifiers still preserves the concept after concept scrubbing.\n\nMinor:\n- Line 325: lineally -> linearly", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback on the presentation of the paper, especially the proofs. We will revise the manuscript based on their feedback for the camera ready version.\n\n> One experiment that I am curious about is the part-of-speech tag prediction accuracy after concept scrubbing. As LEACE only guarantees linear guardedness, it is unclear from Section 5 and 6 whether non-linear classifiers still preserves the concept after concept scrubbing.\n\nIf the class-conditional distributions $\\mathcal{P}(\\mathrm{X} | \\mathrm{Z} = i) : i \\in 1 \\ldots k$ are all Gaussian with equal covariance matrices, then LEACE will indeed prevent non-linear classifiers from predicting $\\mathrm{Z}$ using the scrubbed representation $\\mathrm{X}'$. This is because the resulting distributions $\\mathcal{P}(\\mathrm{X}' | \\mathrm{Z} = i) : i \\in 1 \\ldots k$ will all be equal, and therefore $\\mathrm{X}'$ will not have any mutual information with $\\mathrm{Z}$. In most cases, however, concept scrubbing will be unable to prevent non-linear classifiers from extracting information about $\\mathrm{Z}$ at least to some extent.\n\n> In Figure 1 left, why does random projection have even higher accuracy compared to no intervention?\n\nWe thank the reviewer for bringing this issue to our attention. We found a typographical error in our plotting code, which caused the “No Intervention” accuracy to be reported as 0.892 instead of the correct value of 0.982. The corrected plot is included in the PDF attached to the general rebuttal as Figure 3.\n\n> The definition 2.1 might have an issue. The maximization right after Line 56 should be over all possible joint distributions over features and labels that has marginal distribution the same as Z.\n\nAlthough the current text does state X and Z are “jointly defined,” we agree with the reviewer that this should be made more clear. We intend to fix this issue in the camera-ready version."}, {"review_id": "6Az79jE7ck", "reviewer": "Reviewer_EBcB", "summary": "Suppose we are given a distribution of data points $(x,z)$, where $x \\in \\mathbb{R}^d$ and we have one-hot class labels $z \\in \\mathbb{R}^k$. This paper shows how to construct an affine transformation $\\phi(x) = Px + b$ of the data so that\n* $\\phi(x)$ has zero covariance with $z$\n* $\\\\|\\phi(x) - x\\\\|$ is minimized, subject to the above condition\n\nThe paper argues that this is the optimal affine transformation that ``erases'' the concept $z$ from the representation $x$. Namely, linear regression on $x$ with any convex loss function will be unable to reconstruct $z$ better than random guessing. The paper empirically shows that the proposed method outperforms previous methods (RLACE, INLP) for erasing concepts from a representation, in terms of computational efficiency/collateral damage of the concept erasure.\n\nApplications are demonstrated in: 1) fairness, and 2) probing language models to measure how much a certain concept matters to their performance.\n", "questions": "1. The authors state that \"prior work has focused solely on preventing linear models from leveraging Z\", whereas their work applies also to deep neural networks. How does this claim relate to the cited papers [22, 5, 3, 37], which also intervene on the representations of deep neural networks?\n\n2. Can the authors clarify the paragraph at the top of page 7, entitled \"Results\"? Why is 77.3% profession-prediction accuracy reported, and then 78.1% accuracy reported? What is the difference between these settings?\n\n\nTypos:\n* In Definition 2.1, \\mathfrak{L} is said to have been defined in Section 2.1, but it is not.\n* In Section 2.2, the equation between lines 56 and 57 is an argmax over X' but X' does not appear in the expression\n* Figure 3, \"ramdom\" -> \"random\"\n* Appendix E.1, \"a forteriori\" --> \"a fortiori\"", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper has excellent exposition: the proofs and definitions are clear and simple to follow, and the experiments are generally well explained. The applications to language models are compelling, and the paper demonstrates an improvement over previous approaches. The result is of broad interest to the community.", "weaknesses": "I am not an expert in the field of concept erasure, so I do not know if this method was previously known. However, my feeling is that the paper proposes a \"trivial\" solution: orthogonal projection of the random vector $x$ to the subspace of random vectors uncorrelated with $z$. So I find it surprising that this method was not previously known (maybe in a different field and under a different name). \nOf course, this is not actually a weakness of the paper (but rather of the reviewer). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  my feeling is that the paper proposes a \"trivial\" solution: orthogonal projection of the random vector x to the subspace of random vectors uncorrelated with z. So I find it surprising that this method was not previously known (maybe in a different field and under a different name).\n\nWe believe the simplicity of LEACE is one of its major strengths. After a thorough literature review, we were unable to find any prior work proposing an equivalent method, which suggests our approach may only seem “trivial” in hindsight.\n\nThe closest prior art is the Spectral Attribute RemovaL (SAL) method of Shao et al. (2022), who erase a concept $\\mathrm{Z}$ from a representation $\\mathrm{X}$ by orthogonally projecting $\\mathrm{X}$ onto $\\mathrm{colsp}(\\Sigma_{\\mathrm{XZ}})^\\perp$. Like SAL, LEACE also neutralizes the subspace $\\mathrm{colsp}(\\Sigma_{\\mathrm{XZ}})$ with a projection. But unlike SAL, the LEACE projection is oblique (i.e. non-orthogonal) in general.\n\nTo see the difference, we encourage the reviewer to examine Figure 1 in the PDF attached to our general rebuttal. SAL would orthogonally project the blue and orange points onto the dashed line, which does indeed achieve linear guardedness. But this is not the least-squares optimal solution, since it doesn’t respect the covariance structure of the data. Prior to applying SAL, the data has more variance along the x-axis than along the y-axis, but SAL inverts this structure. By contrast, LEACE preserves the original data distribution as best it can.\n\nWe note also that our theorems prove the sufficiency of these methods for linear guardedness w.r.t. all convex loss functions (Theorem 3.1). Previously this had only been established for specific loss functions, such as the L2 loss (Ravfogel et al. 2022, “Linear Adversarial Concept Erasure”).\n\n> The authors state that \"prior work has focused solely on preventing linear models from leveraging Z\", whereas their work applies also to deep neural networks. How does this claim relate to the cited papers [22, 5, 3, 37], which also intervene on the representations of deep neural networks?\n\nWe apologize for the misleading wording— it would be better to say that prior work has focused primarily, but not exclusively, on linear models. We will correct this mistake in the camera-ready version. That said, 3 of the 4 cited papers (Chowdhury et al. 2022, Celikkanat et al. 2020, and Subramanian et al. 2021) apply concept erasure to frozen contextualized embeddings extracted from the final layer of a language model, and not to intermediate activations. We would classify these papers as concerning linear models, since they aim to prevent a linear classifier trained on frozen embeddings from using a concept. The other paper (Lasri et al. 2022) does apply INLP to selected intermediate layers in BERT, but does not attempt to erase a concept from multiple layers simultaneously, as our concept scrubbing method does. We hope this clarifies the novelty of our concept scrubbing method.\n\n> Can the authors clarify the paragraph at the top of page 7, entitled \"Results\"? Why is 77.3% profession-prediction accuracy reported, and then 78.1% accuracy reported? What is the difference between these settings?\n\nWhen a profession predictor is fit on the original unedited training set, it achieves 77.3% accuracy on a LEACE’d validation set— that is, a validation set on which we have applied LEACE to remove the concept of gender. When the predictor is fit on a LEACE’d training set, it achieves 78.1% accuracy on the LEACE’d validation set. The gap between these two figures corresponds to the mild distribution shift created by LEACE.\n\nWe apologize that this was not clearer in our submission and will rephrase the paragraph for clarity in the camera-ready version."}, {"review_id": "ASHsGufp1A", "reviewer": "Reviewer_3Jip", "summary": "This paper introduces a novel method to guard pretrained representations of deep neural networks from linear recovery of sensitive attributes Z. The notion of linear guardedness comes from the previous literature, while this paper proposes a simple characterisation when it takes place, which allows to dramatically simplify the algorithm as well as improve the performance.\n\nUnlike the previous methods such as INLP, the LEACE projection matrix has rank d - k, where d is the dimension of the representation at hand, and k is the number of values Z can take (i.e. 2 in the binary case). This allows to guard the sensitive attribute with minimal losses. The authors confirm this with two experiments: 1) guarding gender in bios from De-Arteaga et al. 2) amnesic probing from Elazar et al.\n\npros:\n- simple and effective method that outperforms previous literature\n- well written, and the equivalence theorems are easy to follow\n- the experiment in Figure 3 is fascinating, in particular, the conclusion that the information is concentrated in layer 11\n\ncons:\n- lack of error bars. For example, in the experiment corresponding to Figure 1, what happens if we feed a limited data into the LEACE algorithm and test guardedness on the rest?\n- in the experiments the data to which LEACE is applied is the same as it is tested on. It is not tested whether the concept can be erased using dataset that is collected independent of the downstream task.\n- the conclusion in section 5.3 is relevant to interpretation methods such as Kim et al \"Interpretability beyond attribution: TCAV\", who are also concerned with what kinds of information a neural network extracts at each layer. On the contrary to your claim, they report that the amount of \"concept information\" does not reduce as data passes from layer to layer, see Figure 5. I wonder how it compares to you measurements in the case of POS concept. Their method is very simple and should be easy to implement.\n- it is also not entirely clear how it affects the remaining information. For example, if I look at two identical bios which differ only in the gender of a person, how would their representations compare after you have removed the gender concept with LEACE?\n\nminor comments:\n\n- Figure 3: ramdom\n- l306: which a we", "questions": "-", "limitations": "-", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-", "weaknesses": "-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> lack of error bars.\n\nIn Figure 1, the vertical lines crossing through each data point are 95% confidence intervals. We apologize that we did not make this clearer in the original submission and we intend to replace the error lines with a translucent error ribbon in the camera-ready version.\n\n> For example, in the experiment corresponding to Figure 1, what happens if we feed a limited data into the LEACE algorithm and test guardedness on the rest? …in the experiments the data to which LEACE is applied is the same as it is tested on. It is not tested whether the concept can be erased using dataset that is collected independent of the downstream task.\n\nWe agree that examining the generalization properties of LEACE is an important research direction. There are many different metrics that could be used to measure generalization performance, and many different types of distribution shift that could be tested. Given the richness of the topic, we decided that we could not do it justice given the time and space constraints of this paper. We are excited to see this issue addressed in future work.\n\n> the conclusion in section 5.3 is relevant to interpretation methods such as Kim et al \"Interpretability beyond attribution: TCAV\", who are also concerned with what kinds of information a neural network extracts at each layer.\n\nThank you for bringing this work to our attention. We would like to emphasize that, unlike Kim et al. (2018), the experiments in Sections 5 and 6 are not concerned with measuring the kinds of information extracted by the neural network at each layer. We are rather interested in the causal contribution of a query concept— in this case, part of speech— to the network’s performance. This is a distinct, albeit related quantity.\n\nSince the publication of Kim et al. (2018), several papers have found that the ease of extracting a concept from neural network activations is a poor proxy for the causal contribution of that concept to network behavior; see e.g. “Designing and Interpreting Probes with Control Tasks” by Liang et al. (2019), or “Probing classifiers: Promises, shortcomings, and advances” by Belinkov (2022) for a thorough literature review. We will clarify this issue in the camera-ready with appropriate citations. \n\n> it is also not entirely clear how it affects the remaining information. For example, if I look at two identical bios which differ only in the gender of a person, how would their representations compare after you have removed the gender concept with LEACE?\n\nWe encourage the reviewer to examine Figure 1 in the PDF attached to our general rebuttal, which visualizes the LEACE erasure method on a 2D toy dataset."}, {"review_id": "LLEnDAcHa2", "reviewer": "Reviewer_uqXm", "summary": "The paper presents a concept erasure procedure that also takes into account the distortion of the representations during the debiasing process. To this end, the paper builds on several works on iterative nullspace projection approaches and incorporates a regularization component. First, LEACE improves upon previous projection approaches by showing that E[X|Z]=E[X] is sufficient to protect information against a linear adversary. Using this result,  the paper presents a closed-form solution for the objective function. The paper shows experiments on debiasing and interoperability setups. The results showcase that LEACE can achieve similar levels of debiasing performance with the learned representations having much less distortion (measured in terms of its rank). ", "questions": "See the weakness section.", "limitations": "Line 45: “taking values in Rd”: what do values refer to here? I get that it refers to X but should be clearly stated. \n\nLine 53: could not find the 4th variable defined in Section 2.1. Is it referring to the loss function?\n\nLine 55: it's unclear why the family of loss functions needs to be used here. The guarding is defined w.r.t. to a single loss function. \n\nThe framing of different lemmas can be smoothened out to make it more straightforward.\n\nMore context about the functioning of the baseline approaches would be helpful.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strengths:\n\n1. The paper proves the interesting result that achieving the following condition: E[X|Z]=E[X] results in preventing any linear adversary from extracting information about the protected attribute from a representation set. \n2. The paper presents a closed-form optimal solution to linear debiasing with the L2 regularization constraint. \n3. The paper shows experiments in several debiasing and interpretability setups. The results show that LEACE is able to achieve debiasing performance at par with previous approaches while retaining significant information (in terms of rank).", "weaknesses": "Weakness:\n\n1. Is it unclear from the paper why the MSE reconstruction loss E[X-\\hat{X}] is a good measure of information being retained? Ideally, I believe that the goal of concept erasure is to retain within-class information for each of the protected class categories. For such a requirement, there can be a trivial baseline where E[X|Z]=E[X] is achieved by normalizing the data within each class to have the same mean. Does this baseline work at all in terms of debiasing? If yes, what are the advantages of LEACE over this baseline? If no, is E[X|Z]=E[X] the right condition to be achieved?\n2. The derived closed-form result looks very similar to the result in RLACE. It would be interesting to have an analytical discussion about the differences and their implications on debiasing performance. \n3. Although having minimal edits is one of the key focuses of this paper, reconstruction loss is not reported in the results section. Simply reporting the rank is not convincing enough to evaluate the efficacy of the proposed approach in the paper. How does the reconstruction loss compare with other linear debiasing methods? If rank is the right metric to look at instead of the MSE loss, the paper needs more justification about why that is the case.\n4. The fair representation learning framework proposed in this paper has been studied by various works previously. For example, LAFTR[1] and [2] have a similar reconstruction loss along with a classifier and adversary. The paper should have these baselines by simply removing the classifier component and having just the reconstruction and a linear adversary loss. This would be useful in understanding the efficacy of projection-based approaches over adversarial ones. \n5. The experimental section is fairly weak at the moment. For example, in Table 1, why is LEACE not compared with RLACE and INLP in this setup? The authors do not report the performance of word embedding debiasing, which is one of the most common benchmarks for evaluating concept erasure approaches. \n6. The organization/writing of the paper can be improved in several areas. Broadly, some of the theoretical results can be condensed and more context about the debiasing setup would be helpful for readers not too familiar with concept erasure literature. See some minor additional comments in the suggestions/limitations section.\n\n[1]  https://arxiv.org/pdf/1802.06309.pdf\n\n[2]  https://www.cs.toronto.edu/~toni/Papers/icml-final.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. Is it unclear from the paper why the MSE reconstruction loss E[X-\\hat{X}] is a good measure of information being retained?\n\nWe agree with the reviewer that mean squared Euclidean distance is a limited measure. Luckily, we have since proven that LEACE is highly robust to the specific distance function used. Specifically, we prove that LEACE simultaneously minimizes the entire family of squared (pseudo-)norms of the form $\\\\| x \\\\|_{\\mathbf M}^2 = x^T \\mathbf{Mx}$ for some p.s.d. matrix $\\mathbf M$. Intuitively, this means that the “weight” or “importance” assigned to each direction in $\\mathbb R^d$ does not matter. See our general rebuttal for a proof of this result.\n\nWe agree that retention of within-class information is necessary for practically useful concept erasure, but it is not sufficient for our purposes. This is because our primary aim is to minimize damage to the network’s input-output behavior, without retraining or fine tuning its parameters. Arguably a good proxy for this is the Fisher information metric, which locally approximates the KL divergence between the network’s final output before and after perturbing its activations. Since this metric is p.s.d., LEACE minimizes it “out of the box,” without any change to its formula. Thus, LEACE can indeed be viewed as approximately minimizing damage to the network’s behavior, even though it is architecture-agnostic.\n\n> For such a requirement, there can be a trivial baseline where E[X|Z]=E[X] is achieved by normalizing the data within each class to have the same mean. Does this baseline work at all in terms of debiasing? If yes, what are the advantages of LEACE over this baseline?\n\nTheorem 3.1 does indeed imply that the normalization technique you suggest ensures linear guardedness. The primary benefit of LEACE over mean-normalization is that mean-normalization requires access to ground truth concept labels at test time. This is because one needs to know the label in order to know which class centroid $\\mathbb E[\\mathrm X | \\mathrm Z = i]$ should be subtracted from a given data point $\\boldsymbol x$. We often lack ground-truth labels at test time, such as when de-biasing a classifier w.r.t. a protected attribute. Attempting to “guess” the label from observed attributes is likely to be noisy, and may itself be an instance of bias or unfairness.\n\n> 2. The derived closed-form result looks very similar to the result in RLACE. It would be interesting to have an analytical discussion about the differences and their implications on debiasing performance.\n\nWe presume the reviewer is referring to Proposition 3.1 in Ravfogel et al. (2022) Linear Adversarial Concept Erasure, which proves that the orthogonal projection matrix onto $\\mathrm{colsp}(\\Sigma_{XZ})^\\perp$ can be used to achieve guardedness w.r.t. linear regression models. Indeed, Theorem 4.1 implies that this formula achieves guardedness w.r.t. linear classifiers as well. But unlike LEACE, it is not surgical, since it does not take into account anisotropy in the covariance matrix of X (see Figure 1 in the rebuttal PDF), nor does it account for the possibility that X is not centered at the origin.\n\n> 3. Although having minimal edits is one of the key focuses of this paper, reconstruction loss is not reported in the results section.\n\nWe thank the reviewer for noting this omission. We have therefore included a plot in our rebuttal PDF (Figure 2). We plan to replace Figure 1 in the current submission with this new plot in the camera-ready version.\n\n> The fair representation learning framework proposed in this paper has been studied by various works previously. For example, LAFTR[1] and [2] have a similar reconstruction loss along with a classifier and adversary. The paper should have these baselines by simply removing the classifier component and having just the reconstruction and a linear adversary loss.\n\nWe don’t view our work as being directly comparable to the cited works “LAFTR” (Madras et al. 2018) and Zemel et al. (2013), which are both comprehensive proposals for learning fair representations. By contrast, LEACE and concept scrubbing are modular tools which can be applied in a post hoc manner to pre-trained models of all kinds.\n\n> The experimental section is fairly weak at the moment. For example, in Table 1, why is LEACE not compared with RLACE and INLP in this setup?\n\nWe exclusively compare to SAL in the concept scrubbing experiment because it is the only applicable prior art which achieves perfect linear guardedness (by Theorem 4.1), and because it is efficiently scalable to very large datasets, such as the Pile and RedPajama pre-training corpora. This scalability is due to the fact that, like LEACE, SAL only depends on covariance and cross-covariance statistics which can be computed in a streaming fashion. RLACE and INLP, by contrast, require expensive gradient-based optimization and would be much more computationally intensive and time-consuming to run for this large-scale experiment.\n\n> could not find the 4th variable defined in Section 2.1. Is it referring to the loss function?\n\n$\\mathfrak L$ is a family of loss functions $\\mathcal L$, and its definition was missing from Sec. 2.1. We will correct this in the camera-ready version.\n\n> it's unclear why the family of loss functions needs to be used here. The guarding is defined w.r.t. to a single loss function.\n\nIn our work, we define guardedness w.r.t. a family of loss functions, since our Theorem 3.1 applies to the family of all convex losses.\n\n> lack of experiments on uncontextualized word embeddings.\n\nWhile the suggestion is well taken, we believe this is beyond the scope of the current paper. We believe the current content and focus fully occupies a complete paper, and that further expansions to problems outside of the deep classification context would make the paper unwieldy and would require comparing against two different lines of literature simultaneously."}]

Author rebuttal:
_Please see the attached PDF for figures cited in reviewer-specific rebuttals._

We thank all the reviewers for their helpful feedback. We would like to present two simple extensions of the theoretical results from our original submission, which we believe will make the paper even more compelling and useful for practitioners.

**Stronger theoretical result on surgicality** First, we prove a new theorem which implies that LEACE is "surgical" in a much stronger sense than we previously claimed. Specifically, we show that LEACE minimizes the mean squared distance between the original and edited features, not only with respect to the Euclidean norm, but with respect to _all_ norms of the form $\\| \mathbf x \\|_{\mathbf M}^2 = \mathbf x^T \mathbf{Mx}$ for some positive-semidefinite $\mathbf{M}$.

The proof is rather simple and intuitive. We begin with the case where $\mathrm X$ is centered and hence the LEACE bias term is zero. Without loss of generality, assume $\mathrm X$ is written in an orthogonal basis that diagonalizes $\mathbf M$. We can then write the reconstruction error as a weighted sum of $d$ terms, each of which only depends on a single row of the erasure matrix $\mathbf P$:

1. $\mathcal L(\mathbf P) = \mathbb E \Big [ \big\\| \mathbf P \mathrm X - \mathrm X \big\\|^2_{\mathbf M} \Big ] = \sum_{i=1}^d w_i \mathbb E \big [ (\mathbf P_i \mathrm X - \mathrm X_i)^2 \big ] = \sum_{i=1}^d w_i \mathcal L_i(\mathbf P_i),\quad \forall i : w_i \ge 0$.
The weights $w_i$ in Eq. 1 can be viewed as expressing the "importance" we assign to preventing changes along each component. For example, we might want to weight directions in proportion to their effect on the network's final output.

However, we can immediately see that these weights do not affect the set of optimal solutions. This is because Equation 1 implies that $\mathcal L$ is _additively separable_ along the rows of $\mathbf P$, and since our erasure constraint $\mathrm{Cov}(\mathbf P \Sigma_{\mathrm{XZ}}, \mathrm{Z}) = \mathbf 0_{d \times k}$ can also be decomposed into a set of independent constraints $\mathrm{Cov}(\mathbf P_i \Sigma_{\mathrm{XZ}}, \mathrm{Z}) = \mathbf 0_{1 \times k}$ for each row $\mathbf P_i$, we may conclude that $\mathbf P$ is optimal _if and only if_ each of its rows are optimal for their respective subproblems $\mathcal L_i$.

But this means that the optimality conditions for $\mathbf P$ are independent of the weights $w_i$, and hence also the choice of norm. Since we have already proven that the LEACE projection matrix is optimal for the Euclidean norm (i.e. $\mathbf M = \mathbf I$), we can conclude that it is optimal for all p.s.d. $\mathbf M$.

The extension to uncentered $\mathrm X$ closely mirrors the proof of Theorem 4.3 in our submission. Define $\tilde{\mathrm X} = \mathrm X - \mathbb E[\mathrm X]$ and $\mathbf{c} = \mathbf{P}\mathbb E[\mathrm X] + \mathbf{b} - \mathbb E[\mathrm X]$. Then we have \begin{align*} \mathbb E \big\\|\mathbf{P}\mathrm X + \mathbf{b} - \mathrm X \big\\|^2_\mathbf{M} &= \mathbb E \big\\|(\mathbf{P}\tilde{\mathrm X} - \tilde{\mathrm X}) + \mathbf{c} \big\\|^2_\mathbf{M} \\\\ & = \mathbb E \big\\|\mathbf{P}\tilde{\mathrm X} - \tilde{\mathrm X} \big\\|^2_\mathbf{M} + 2\mathbb E \big[ \mathbf{P}\tilde{\mathrm X} - \tilde{\mathrm X} \big]^T \mathbf{M} \mathbf{c} + \mathbf{c}^T \mathbf{M} \mathbf{c} \\\\ & = \mathbb E \big\\|\mathbf{P}\tilde{\mathrm X} - \tilde{\mathrm X} \big\\|^2_\mathbf{M} + \mathbf{c}^T \mathbf{M} \mathbf{c}, \end{align*} where we have eliminated the middle term because $\mathbf{P}$ is linear and $\mathbb E[\tilde{\mathrm X}] = 0$. Since $\mathbf{M}$ is p.s.d., our objective is minimized for $\mathbf{c} = \mathbf{0}$, i.e. $\mathbf{b} = \mathbb E[\mathrm X] - \mathbf{P}\mathbb E[\mathrm X]$. The problem thus reduces to choosing $\mathbf{P}$ so as to minimize $\mathbb E \big\\|\mathbf{P}\tilde{\mathrm X} - \tilde{\mathrm X} \big\\|^2_\mathbf{M}$ subject to $\mathrm{Cov}(\mathbf{P}\mathrm X + \mathbf{b}, \mathrm Z) = \mathrm{Cov}(\mathbf{P}\tilde{\mathrm X}, \mathrm Z) = \mathbf{0}$, which occurs when $\mathbf{P}$ is the LEACE projection matrix.

We are excited about this result because it suggests that LEACE should be useful under a wide variety of assumptions about which parts of a representation are most important. The choice of norm is not an arbitrary free parameter in our method.

**More intuitive closed-form formula & visualization** We also would like to present a new, yet _mathematically equivalent_ formula for the LEACE projection matrix which we believe is significantly more intuitive than the one we previously reported:

2. $r_{\mathrm{LEACE}}(\boldsymbol x) = \boldsymbol x - \mathbf{W}^+ \mathrm{Proj}(\mathbf{W}\Sigma_{\mathrm{XZ}}) \mathbf{W}\big (\boldsymbol x - \mathbb E[\mathrm X] \big )$,

where $\mathbf{W} = (\Sigma_{\mathrm{XX}})^{-1/2}$ is a whitening matrix and $\mathrm{Proj}(\mathbf{W}\Sigma_{\mathrm{XZ}})$ is the orthogonal projection matrix onto $\mathrm{colsp}(\mathbf{W}\Sigma_{\mathrm{XZ}})$.

Intuitively, Equation 2 tells us that LEACE de-means and whitens $\boldsymbol x$, projects onto the subspace responsible for correlations between $\mathrm X$ and $\mathrm Z$, then unwhitens the result. Finally, it subtracts this value from $\boldsymbol x$, thereby surgically removing the linearly available information about $\mathrm Z$. We encourage the reviewers to examine Figure 1 in the attached PDF, which shows this three-step process in action on a toy dataset.

We plan to include the full derivation of this new formulation in the appendix of the camera-ready version, where there is sufficient space. We welcome the reviewers to confirm numerically that this formula is indeed equivalent to the one we reported in our original submission, as long as $\Sigma_{\mathrm{XX}}$ is full rank.

Meta-review: The reviewers have provided a comprehensive evaluation of the paper, highlighting its strengths and weaknesses. The paper's novel approach to concept erasure, its clear and simple proofs, and its compelling applications to language models have been appreciated. The paper's ability to improve upon previous projection approaches and provide a closed-form optimal solution to linear debiasing with the L2 regularization constraint is also commendable.

However, the reviewers have also pointed out areas for improvement, such as the need for more clarity on why the MSE reconstruction loss is a good measure of information being retained, the lack of comparison with other linear debiasing methods, and the need for more context about the debiasing setup for readers not too familiar with concept erasure literature.

Despite these concerns, the paper's strengths outweigh its weaknesses. The authors have made a significant contribution to the field of concept erasure, and their work is likely to have a high impact on at least one sub-area. Therefore, the paper is accepted. The authors are encouraged to address the reviewers' concerns in their final version of the paper.
Decision: Accept (poster)