Log-linear Guardedness and its Implications
Year: 2023

Abstract:
Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful. However, the impact of this removal on the behavior of downstream classifiers trained on the modified representations is not fully understood. In this work, we formally define the notion of log-linear guardedness as the inability of an adversary to predict the concept directly from the representation, and study its implications. We show that, in the binary case, under certain assumptions, a downstream log-linear model cannot recover the erased concept. However, we demonstrate that a multiclass log-linear model can be constructed that indirectly recovers the concept in some cases, pointing to the inherent limitations of log-linear guardedness as a downstream bias mitigation technique. These findings shed light on the theoretical limitations of linear erasure methods and highlight the need for further research on the connections between intrinsic and extrinsic bias in neural models. https://github.com/rycolab/ guardedness

heading: 1 Introduction
text: Neural models of text have been shown to represent human-interpretable concepts, e.g., those related to the linguistic notion of morphology (Vylomova et al., 2017), syntax (Linzen et al., 2016), semantics (Belinkov et al., 2017), as well as extra-linguistic notions, e.g., gender distinctions (Caliskan et al., 2017). Identifying and erasing such concepts from neural representations is known as concept erasure. Linear concept erasure in particular has gained popularity due to its potential for obtaining formal guarantees and its empirical effectiveness (Bolukbasi et al., 2016; Dev and Phillips, 2019; Ravfogel et al., 2020; Dev et al., 2021; Kaneko and Bollegala, 2021; Shao et al., 2023b,a; Kleindessner et al., 2023; Belrose et al., 2023).
A common instantiation of concept erasure is removing a concept (e.g., gender) from a
representation (e.g., the last hidden representation of a transformer-based language model) such that it cannot be predicted by a log-linear model. Then, one fits a secondary log-linear model for a downstream task over the erased representations. For example, one may fit a log-linear sentiment analyzer to predict sentiment from gender-erased representations. The hope behind such a pipeline is that, because the concept of gender was erased from the representations, the predictions made by the log-linear sentiment analyzer are oblivious to gender. Previous work (Ravfogel et al., 2020; Elazar et al., 2021; Jacovi et al., 2021; Ravfogel et al., 2022a) has implicitly or explicitly relied on this assumption that erasing concepts from representations would also result in a downstream classifier that was oblivious to the target concept.
In this paper, we formally analyze the effect concept erasure has on a downstream classifier. We start by formalizing concept erasure using Xu et al.’s (2020) V-information.1
We then spell out the related notion of guardedness as the inability to predict a given concept from concept-erased representations using a specific family of classifiers. Formally, if V is the family of distributions realizable by a log-linear model, then we say that the representations are guarded against gender with respect to V . The theoretical treatment in our paper specifically focuses on loglinear guardedness, which we take to mean the inability of a log-linear model to recover the erased concept from the representations. We are able to prove that when the downstream classifier is binary valued, such as a binary sentiment classifier, its prediction indeed cannot leak information about the erased concept (§ 3.2) under certain assumptions. On the contrary, in the case of multiclass classification with a log-linear model, we show that predictions can potentially leak a substantial amount of information about the removed concept, thereby recovering the guarded information completely. The theoretical analysis is supported by experiments on commonly used linear erasure techniques (§ 5).
1We also consider a definition based on accuracy.
9413
While previous authors (Goldfarb-Tarrant et al. 2021, Orgad et al. 2022, inter alia) have empirically studied concept erasure’s effect on downstream classifiers, to the best of our knowledge, we are the first to study it theoretically. Taken together, these findings suggest that log-linear guardedness may have limitations when it comes to preventing information leakage about concepts and should be assessed with extreme care, even when the downstream classifier is merely a log-linear model.

heading: 2 Information-Theoretic Guardedness
text: In this section, we present an information-theoretic approach to guardedness, which we couch in terms of V-information (Xu et al., 2020).

heading: 2.1 Preliminaries
text: We first explain the concept erasure paradigm (Ravfogel et al., 2022a), upon which our work is based. Let X be a representation-valued random variable. In our setup, we assume representations are realvalued, i.e., they live in RD. Next, let Z be a binaryvalued random variable that denotes a protected attribute, e.g., binary gender.2 We denote the two binary values of Z by Z def= {⊥,⊤}. We assume the existence of a guarding function h : RD → RD that, when applied to the representations, removes the ability to predict a concept Z given concept by a specific family of models. Furthermore, we define the random variable Ŷ = t(h(X)) where t : RD → Y def= {0, . . . , |Y|} is a function3 that corresponds to a linear classifier for a downstream task. For instance, t may correspond to a linear classifier that predicts the sentiment of a representation.
Our discussion in this paper focuses on the case when the function t is derived from the argmax of a log-linear model, i.e., in the binary case we define Ŷ’s conditional distribution given h(X) as
p(Ŷ = y | h(X) = h(x)) = { 1, if y = y∗
0, else (1)
where θ ∈ RD is a parameter column vector, ϕ ∈ R is a scalar bias term, and
y∗ = { 1, if θ⊤h(x) + ϕ > 0 0, else
(2)
2Not all concepts are binary, but our analysis in § 2 makes use of this simplifying assumption.
3The elements of Y are denoted y.
And, in the multivariate case we define Ŷ’s conditional distribution given h(X) as
p(Ŷ = y | h(X) = h(x)) = { 1, if y = y∗
0, else (3)
where y∗ = argmaxy′∈Y (Θ ⊤h(x) + ϕ)y′ and Θy ∈ RD denotes the yth column of Θ ∈ RD×K , a parameter matrix, and ϕ ∈ RK is the bias term. Note K is the number of classes.
2.2 V-Information Intuitively, a set of representations is guarded if it is not possible to predict a protected attribute z ∈ Z from a representation x ∈ RD using a specific predictive family. As a first attempt, we naturally formalize predictability in terms of mutual information. In this case, we say that Z is not predictable from X if and only if I(X; Z) = 0. However, the focus of this paper is on linear guardedness, and, thus, we need a weaker condition than simply having the mutual information I(X; Z) = 0. We fall back on Xu et al.’s (2020) framework of V-information, which introduces a generalized version of mutual information. In their framework, they restrict the predictor to a family of functions V , e.g., the set of all log-linear models.
We now develop the information-theoretic background to discuss V-information. The entropy of a random variable is defined as
H(Z) def = − E
z∼p(Z) log p(z) (4)
Xu et al. (2020) analogously define the conditional V-entropy as follows
HV(Z | X) def= − sup q∈V E (x,z)∼p(X,Z) log q(z | x) (5)
The V-entropy is a special case of Eq. (5) without conditioning on another random variable, i.e.,
HV (Z) def = − sup
q∈V E z∼p(Z) log q(z) (6)
Xu et al. (2020) further define the V-information, a generalization of mutual information, as follows
IV(X→ Z) def= HV (Z)−HV(Z | X) (7)
In words, Eq. (7) is the best approximation of the mutual information realizable by a classifier belonging to the predictive family V . Furthermore, in the case of log-linear models, Eq. (7) can be
approximated empirically by calculating the negative log-likelihood loss of the classifier on a given set of examples, as HV (Z) is the entropy of the label distribution, and HV(Z | X) is the minimum achievable value of the cross-entropy loss.

heading: 2.3 Guardedness
text: Having defined V-information, we can now formally define guardedness as the condition where the V-information is small. Definition 2.1 (V-Guardedness). Let X be a representation-valued random variable and let Z be an attribute-valued random variable. Moreover, let V be a predictive family. A guarding function h ε-guards X with respect to Z over V if IV(X→ Z) < ε. Definition 2.2 (Empirical V-Guardedness). Let D = {(xn, zn)}Nn=1 where (xn, zn) ∼ p(X,Z). Let X̃ and Z̃ be random variables over RD and Z , respectively, whose distribution corresponds to the marginals of the empirical distribution over D. We say that a function h(·) empirically ε-guards D with respect to the family V if IV(h(X̃)→ Z̃) < ε.
In words, according to Definition 2.2, a dataset is log-linearly guarded if no linear classifier can perform better than the trivial classifier that completely ignores X and always predicts Z according to the proportions of each label. The commonly used algorithms that have been proposed for linear subspace erasure can be seen as approximating the condition we call log-linear guardedness (Ravfogel et al., 2020, 2022a,b). Our experimental results focus on empirical guardedness, which pertains to practically measuring guardedness on a finite dataset. However, determining the precise bounds and guarantees of empirical guardedness is left as an avenue for future research.

heading: 3 Theoretical Analysis
text: In the following sections, we study the implications of guardedness on subsequent linear classifiers. Specifically, if we construct a third random variable Ŷ = t(h(X)) where t : RD → Y is a function, what is the degree to which Ŷ can reveal information about Z? As a practical instance of this problem, suppose we impose ε-guardedness on the last hidden representations of a transformer model, i.e., X in our formulation, and then fit a linear classifier t over the guarded representations h(X) to predict sentiment. Can the predictions of the sentiment classifier indirectly leak information on
gender? For expressive V , the data-processing inequality (Cover and Thomas, 2006, §2.8) applied to the Markov chain X→ Ŷ → Z tells us the answer is no. The reason is that, in this case, V-information is equivalent to mutual information and the data processing inequality tells us such leakage is not possible. However, the data processing inequality does not generally apply to V-information (Xu et al., 2020). Thus, it is possible to find such a predictor t for less expressive V . Surprisingly, when |Y| = 2, we are able to prove that constructing such a t that leaks information is impossible under a certain restriction on the family of log-linear models.

heading: 3.1 Problem Formulation
text: We first consider the case where |Y| = 2.

heading: 3.2 A Binary Downstream Classifier
text: We begin by asking whether the predictions of a binary log-linear model trained over the guarded set of representations can leak information on the protected attribute. Our analysis relies on the following simplified family of log-linear models.
Definition 3.1 (Discretized Log-Linear Models). The family of discretized binary log-linear models with parameter δ ∈ (0, 1) is defined as
Vδ def= { f ∣∣∣ f(0) = ρδ(σ(α
⊤x+ γ)) f(1) = ρδ(1− σ(α⊤x+ γ))
} (8)
with α ∈ RD, γ ∈ R, σ being the logistic function, and where we define the δ-discretization function as
ρδ(p) def = { δ, if p ≥ 12 1− δ, else
(9)
In words, ρδ is a function that maps the probability value to one of two possible values. Note that ties must be broken arbitrarily in the case that p = 12 to ensure a valid probability distribution.
Our analysis is based on the following simple observation (see Lemma A.1 in the Appendix) that the composition of two δ-discretized log-linear models is itself a δ-discretized log-linear model. Using this fact, we show that when |Y| = |Z| = 2, and the predictive family is the set of δ-discretized binary log-linear models, ε-guarded representations h(X) cannot leak information through a downstream classifier.
Theorem 3.2. Let Vδ be the family of δdiscretized log-linear models, and let X be a representation-valued random variable. Define
Ŷ as in Eq. (1), then IVδ(h(X)→ Z) < ε implies IVδ(Ŷ → Z) < ε. Proof. Define the hard thresholding function
τ(x) def = { 1, if x > 0 0, else
(10)
Assume, by contradiction, that IVδ(X → Z) < ε, but IVδ(Ŷ → Z) > ε. We start by algebraically manipulating IVδ(X→ Z):
IVδ(Ŷ → Z) = HVδ(Z)−HVδ(Z | Ŷ) (11) = HVδ(Z) + sup
q∈Vδ E (z,y)∼p(Z,Ŷ) log q(z | y)
= HVδ(Z) + E (z,y)∼p(Z,Ŷ)
log q∗(z | y)
Finally, making use of a change of variable in Eq. (1) gives us IVδ(Ŷ → Z) = HVδ(Z) + E(z,x)∼p log q∗(z | τ(θ⊤h(x) + ϕ)). where θ and ϕ stem from the definition of t in Eq. (1). This chain of equalities gives us that IVδ(Ŷ → Z) < ε =⇒ HVδ(Z) + E(z,y)∼p log q∗(z | y) < ε. Now, we note that there exists a classifier r ∈ Vδ such that r(z | h(x)) = q∗(z | τ(θ⊤h(x))) by Lemma A.1. The existence of r implies that IVδ(X → Z) > ε, contradicting the assumption. Thus, IVδ(Ŷ → Z) < ε. ■

heading: 3.3 A Multiclass Downstream Classifier
text: The above discussion shows that when both Z and Y are binary, ε-log-linear guardedness with respect to the family of discretized log-linear models (Definition 3.1) implies limited leakage of information about Z from Ŷ. It was previously implied (Ravfogel et al., 2020; Elazar et al., 2021) that linear
concept erasure prevents information leakage about Z through the labeling of a log-linear classifier Ŷ, i.e., it was assumed that Theorem 3.2 in § 3.2 can be generalized to the multiclass case. Specifically, it was argued that a subsequent linear layer, such as the linear language-modeling head, would not be able to recover the information because it is linear. In this paper, however, we note a key flaw in this argument. If the data is log-linearly guarded, then it is easy to see that the logits, which are a linear transformation of the guarded representation, cannot encode the information. However, multiclass classification is usually performed by a softmax classifier, which adds a non-linearity. Note that the decision boundary of the softmax classifier for every pair of labels is linear since class i will have higher softmax probability than class j if, and only if, (θi − θj)⊤x > 0.
Next, we demonstrate that this is enough to break guardedness. We start with an example. Consider the data in R2 presented in Fig. 1(a), where the distribution p(X,Z) has 4 distinct clusters, each with a different label from Z , corresponding to Voronoi regions (Voronoi, 1908) formed by the intersection of the axes. The red clusters correspond to Z = ⊤ and the blue clusters correspond to Z = ⊥. The data is taken to be log-linearly guarded with respect to Z.4 Importantly, we note that knowledge of the quadrant (i.e., the value of Y), renders Z recoverable by a 4-class log-linear model.
Assume the parameter matrix Θ ∈ R4×2 of this classifier is composed of four columns
4Information-theoretic guardedness depends on the density over p(X), which is not depicted in the illustrations in Fig. 1(a).
θ1,θ2,θ3,θ4 such that θ1 = α · [1, 1]⊤,θ2 = α·[−1, 1]⊤,θ3 = α·[−1,−1]⊤,θ4 = α·[1,−1]⊤, for some α > 0. These directions encode the quadrant of a point: When the norm of the parameter vectors is large enough, i.e., for a large enough α, the probability of class i under a log-linear model will be arbitrarily close to 1 if, and only if, the input is in the ith quadrant and arbitrarily close to 0 otherwise. Given the information on the quadrant, the data is rendered perfectly linearly separable. Thus, the labels Ŷ predicted by a multiclass softmax classifier can recover the linear separation according to Z.
This argument can be generalized to a separation that is not axis-aligned (Fig. 1(b)).
Definition 3.3. Let θ1, . . . ,θK be column vectors orthogonal to corresponding linear subspaces, and let R1, . . . , RM be the Voronoi regions formed by their intersection (Fig. 1(b)). Let p(X,Z) be any data distribution such that any two points in the same region have the same value of Z:
xi ∈ Rk ∧ xj ∈ Rk =⇒ zi = zj (12)
for all (xi, zi), (xj , zj) ∼ p(X,Z) and for all Voronoi regions Rk. We call such distribution a K-Voronoi distribution.
Theorem 3.4. Fix ε > 0. Let p(X,Z) be a K-Voronoi distribution, and let h linearly ε-guard X against Z with respect to the family V of log-linear models. Then, for every η > 0, there exists a K-class log-linear model such as IV(Ŷ → Z) > 1− η.5
Proof. By assumption, the support of p(X) is divided up into K Voronoi regions, each with a label from Z . See Fig. 1 for an illustrative example.
Define the region identifier ιk(i) for each region k as follows
ιk(i) def = { 1, if θ⊤i x > 0 for x ∈ Rk −1, if θ⊤i x < 0 for x ∈ Rk (13)
We make the simplifying assumption that points x that lie on line θ⊤i x for any i occur with probability zero. Consider a K-class log-linear model with a parameter matrix Θ⋆ ∈ RD×K that contains, in its jth column, the vector θ⋆j def = α ∑K k=0 ιj(k)θk, i.e., we sum over all θk and give positive weight to a vector θk if a positive dot product with it is
5In base two, log2(2) = 1 is the maximum achievable V-information value in the binary case.
a necessary condition for a point x to belong to the kth Voronoi region. Additionally, we scale the parameter vector by some α > 0. Let x ∈ Rj and let Rm be a Voronoi region such that j ̸= m. We next inspect the ratio
r(α) def = softmax(Θ⋆⊤x)j softmax(Θ⋆⊤x)m
(14a)
= e(θ ⋆ j−θ⋆m)⊤x (14b) = e(α ∑K k=0 ιj(k)θk−α ∑K k=0 ιm(k)θk) ⊤ x = eα( ∑K k=0(ιj(k)−ιm(k))θk) ⊤ x (14c)
We now show that α( ∑K
k=0(ιj(k) − ιm(k))θk)
⊤x > 0 through the consideration of the following three cases:
• Case 1: ιj(k) = ιm(k). In this case, the subspace θk is a necessary condition for belonging to both regions j and m. Thus, the summand is zero.
• Case 2: ιj(k) = 1, but ιm(k) = −1. In this case, ιj(k)−ιm(k) = 2. As x ∈ Rj , we know that θk⊤x > 0, and the summand is positive.
• Case 3: ιj(k) = −1, but ιm(k) = 1. In this case, ιj(k) − ιm(k) = −2. As x ∈ Rj , we know that θk⊤x < 0, and the summand is, again, positive.
Since j ̸= m, a summand corresponding to cases 2 and 3 must occur. Thus, the sum is strictly positive. It follows that limα→∞ r(α) = 1. Finally, for Ŷ defined as in Eq. (3), we have p(Ŷ = j | x ∈ Rj) = 1 for α large. Now, because all points in each Rj have a distinct label from Z , it is trivial to construct a binary log-linear model that places arbitrarily high probability on Rj’s label, which gives us IV(Ŷ → Z) > 1 − η for all η > 0 small. This completes the proof.
■
This construction demonstrates that one should be cautious when arguing about the implications of log-linear guardedness when multiclass softmax classifiers are applied over the guarded representations. When log-linear guardedness with respect to a binary Z is imposed, there may still exist a set of k > 2 linear separators that separate Z.

heading: 4 Accuracy-Based Guardedness
text: We now define an accuracy-based notion of guardedness and discuss its implications. Note that the information-theoretic notion of guardedness described above does not directly imply that the accuracy of the log-linear model is damaged. To see this, consider a binary log-linear model on balanced data that always assigns a probability of 12 + ∆ to the correct label and 1 2 − ∆ to the incorrect label. For small enough ∆, the cross-entropy loss of such a classifier will be arbitrarily close to the entropy log(2), even though it has perfect accuracy. This disparity motivates an accuracy-based notion of guardedness.
We first define the accuracy function ℓ as
ℓ(q,x, z) =    1, if argmax z′∈Z q(z′ | x) = z
0, else (15)
The conditional V-accuracy is defined as
AV (Z | X) def= sup q∈V E (x,z)∼p(X,Z) ℓ(q,x, z) (16)
The V-accuracy is a special case of Eq. (16) when no random variable is conditioned on
AV (Z) def = sup
q∈V E z∼p(Z) ℓ(q, z) (17)
where we have overloaded ℓ to take only two arguments. We can now define an analogue of Xu et al.’s (2020) V-information for accuracy as the difference between the unconditional V-accuracy and the conditional V-accuracy6
IAV (X→ Z) def = AV (Z | X)−AV (Z) (18)
Note that the V-accuracy is bounded below by 0 and above by 12 . Definition 4.1 (Accuracy-based V-guardedness). Let X be a representation-valued random variable and let Z be an attribute-valued random variable. Moreover, let V be a predictive family. A guarding function h ε-guards X against Z with respect to V if IAV (h(X)→ Z) < ε. Definition 4.2 (Accuracy-based Empirical V-guardedness). Let D = {(xn, zn)}Nn=1 where (xn, zn) ∼ p(X,Z). Let X̃ and Z̃ be random variables over RD and Z , respectively, whose
6Note the order of the terms is reversed in V-accuracy.
distribution corresponds to the marginals of the empirical distribution over D. A guarding function h empirically ε-guards D with respect to V if IAV (h(X̃)→ Z̃) < ε.
When focusing on accuracy in predicting Z, it is natural to consider the independence (also known as demographic parity) (Feldman et al., 2015) of the downstream classifiers that are trained over the representations.
Definition 4.3. The L1 independence gap measures the difference between the distribution of the model’s predictions on the examples for which Z = ⊥, and the examples for which Z = ⊤. It is formally defined as
GAPind(Ŷ → Z | X) (19) def = ∑
y∈Y
∣∣∣ E x∼p(X|Z=⊥) p(Ŷ = y | Z = ⊥,X = x)
− E x∼p(X|Z=⊤)
p(Ŷ = y | Z = ⊤,X = x) ∣∣∣
where p(X | Z) is the conditional distribution over representations given the protected attribute.
In Prop. 4.4, we prove that if the data is linearly ε-guarded and globally balanced with respect to Z, i.e., if p(Z = ⊥) = p(Z = ⊤) = 12 , then the prediction of any linear binary downstream classifier is 4ε independent of Z. Note that is true regardless of any imbalance of the protected attribute Z within each class y ∈ Y in the downstream task: the data only needs to be globally balanced.
Proposition 4.4. Let V be the family of binary log-linear models, and assume that p(X, Z) is globally balanced, i.e., p(Z = ⊥) = p(Z = ⊤) = 12 . Furthermore, let h be a guarding function that ε-guards X against Z with respect to V in terms of accuracy (Definition 4.2), i.e., IAV (h(X)→ Z) < ε. Let Ŷ be defined as in Eq. (1). Then, the L1 independence gap (Eq. (19)) satisfies GAPind(Ŷ → Z | h(X)) ≤ 4ε.
Proof. See App. A.2 for the proof. ■

heading: 5 Experimental Evaluation
text: In the empirical portion of our paper, we evaluate the extent to which our theory holds in practice.
Data. We perform experiments on gender bias mitigation on the Bias in Bios dataset (De-Arteaga et al., 2019), which is composed of short biographies annotated by both gender and profession.
We represent each biography with the [CLS] representation in the final hidden representation of pre-trained BERT, which creates our representation random variable X. We then try to guard against the protected attribute gender, which constitutes Z.
Approximating log-linear guardedness. To approximate the condition of log-linear guardedness, we use RLACE (Ravfogel et al., 2022a). The method is based on a minimax game between a loglinear predictor that aims to predict the concept of interest from the representation and an orthogonal projection matrix that aims to guard the representation against prediction. The process results in an orthogonal projection matrix P ∈ RD×D, which, empirically, prevents log-linear models from predicting gender after the linear transformation P is applied to the representations. This process constitutes our guarding function hR. Our theoretical result (Theorem 3.2) only holds for δ-discretized loglinear models. RLACE, however, guards against conventional log-linear models. Thus, we apply δ-discretization post hoc, i.e., after training.

heading: 5.1 Quantifying Empirical Guardedness
text: We test whether our theoretical analysis of leakage through binary and multiclass downstream classifiers holds in practice, on held-out data. Profession prediction serves as our downstream prediction task (i.e., our Ŷ), and we study binary and multiclass variants of this task. In both cases, we measure three V-information estimates:
• Evaluating IV(X → Z). To compute an empirical upper bound on information about the protected attribute which is linearly extractable from the representations, we train a log-linear model to predict zn from xn, i.e., from the unguarded representations. In other words, this is an upper bound on the information that could be leaked through the downstream classifier Ŷ.
• Evaluating IV(Ŷp → Z). We quantify leakage through a downstream classifier Ŷ by estimating IV(Ŷ → Z), for binary and multiclass Ŷ, via two different approaches. The first of these, denoted IV(Ŷp → Z), is computed by training two log-linear and stiching them together into a pipeline. First, we fit a log-linear model on top of the guarded representations hR(X) to yield predictions for a downstream task Ŷp = t(hR(X)) where
t : RD → Y is the function induced by the trained classifier. Subsequently, we train a second log-linear model Ẑ = r(Ŷp) with r : Y → {0, 1} to predict Z from the output of Ŷ In words, Ŷp represents the argmax from the distribution of profession labels (binary or multiclass). We approximate the V-information Ŷp leaks about Z through the cross-entropy loss of a second classifier trained to predict the protected attribute from Ŷp, i.e., we compute empirical guardedness (Definition 2.2) on held-out data.
• Evaluating IV(Ŷa → Z). In addition to the standard scenario estimated by IV(Ŷp → Z), we also ask: What is the maximum amount of information that a downstream classifier could leak about gender? IV(Ŷa → Z) estimates this quantity, with a variant of the setup of IV(Ŷp → Z). Namely, instead of training the two log-linear models separately, we train them together to find the Ŷa that is adversarially chosen to predict gender well. However, the argmax operation is not differentiable, so we remove it during training.
In practice, this means Ŷa does not predict profession, but instead predicts a latent distribution which is adversarially chosen so as to best enable the prediction of gender.7
While high IV(Ŷa → Z) indicates that there exists an adversarial log-linear model that leaks information about Z, it does not necessarily mean that common classifiers like those used to compute IV(Ŷp → Z) would leak that information. Across all of 3 conditions, we explore how different values of the thresholds δ (applied after training) affect the V-information. Refer to App. A.3 for comprehensive details regarding our experimental setup.

heading: 5.2 Binary Z and Y
text: We start by evaluating the case where both Z and Y take only two values each.
Experimental Setting. To create our set Y for the binary classification task, we randomly sample 15 pairs of professions from the Bias in Bios dataset; see App. A.3. We train a binary log-linear
7Only at inference time do we apply the argmax over the first log-linear model to get a prediction Ŷa = y. We find that the loss of the composition model is not increased by the argmax operation.
model to predict each profession from the representation after the application of the RLACE guarding function, hR(xn). Empirically, we observe that our log-linear models achieve no more than 2% above majority accuracy on the protected data. For each pair of professions, we estimate three forms of V-information.
Results. The results are presented in Fig. 2, for the 15 pairs of professions we experimented with (each curve is the mean over all pairs), the three quantities listed above, and different values of the threshold δ on the x-axis. Unsurprisingly, we observe that the V-information estimated from the original representations (the red curve) has high values for some thresholds, indicating that BERT representations do encode gender distinctions. The blue curve, corresponding to IV(Ŷa → Z), measures the ability of the adversarially constructed binary downstream classifier to recover the gender information. It is lower than the red curve but is nonzero, indicating that the solution found by RLACE does not generalize perfectly. Finally, the orange curve, corresponding to IV(Ŷp → Z), measures the amount of leakage we get in practice from downstream classifiers that are trained on profession prediction. In that case, the numbers are significantly lower, showing that RLACE does

heading: 5.3 Binary Z, Multiclass Y
text: Empirically, we have shown that RLACE provides good, albeit imperfect, protection against binary log-linear model adversaries. This finding is in line with the conclusions of Theorem 3.2. We now turn to experiments on multiclass classification, i.e., where |Y| > 2. According to § 3.3, to the extent that the K-Voronoi assumption holds, we expect guardedness to be broken with a large enough |Y|.
Experimental Setting. Note that, since |Y| > 2, Ŷ is a multiclass log-linear classifier over Y , but the logistic classifier that predicts gender from the argmax over these remains binary. We consider different values of |Y| = 2, 4, 8, 16, 32, 64.
Results. The results are shown in Fig. 3. For all professions, we find a log-linear model whose predicted labels are highly predictive of the protected attribute. Indeed, softmax classifiers with 4 to 8 entries (corresponding to hidden neurons in the network which is the composition of two log-linear models) perfectly recover the gender information. This indicates that there are labeling schemes of the data using 4 or 8 labels that recover almost all information about Z.

heading: 5.4 Discussion
text: Even if a set of representations is log-linearly guarded, one can still adversarially construct a multiclass softmax classifier that recovers the information. These results stem from the disparity between the manifold in which the concept resides, and the expressivity of the (linear) intervention we perform: softmax classifiers can access information that is
inaccessible to a purely linear classifier. Thus, interventions that are aimed at achieving guardedness should consider the specific adversary against which one aims to protect.

heading: 6 Related Work
text: Techniques for information removal are generally divided into adversarial methods and post-hoc linear methods. Adversarial methods (Edwards and Storkey, 2016; Xie et al., 2017; Chen et al., 2018; Elazar and Goldberg, 2018; Zhang et al., 2018) use a gradient-reversal layer during training to induce representations that do not encode the protected attribute. However, Elazar and Goldberg (2018) have shown that these methods fail to exhaustively remove all the information associated with the protected attribute. Linear methods have been proposed as a tractable alternative, where one identifies a linear subspace that captures the concept of interest, and neutralizes it using algebraic techniques. Different methods have been proposed for the identification of the subspace, e.g., PCA and variants thereof (Bolukbasi et al., 2016; Kleindessner et al., 2023), orthogonal rotation (Dev et al., 2021), classification-based (Ravfogel et al., 2020), spectral (Shao et al., 2023a,b) and adversarial approaches (Ravfogel et al., 2022a).
Different definitions have been proposed for fairness (Mehrabi et al., 2021), but they are mostly extrinsic—they concern themselves only with the predictions of the model, and not with its representation space. Intrinsic bias measures, which focus on the representation space of the model, have been also extensively studied. These measures quantify, for instance, the extent to which the word representation space encodes gender distinctions (Bolukbasi et al., 2016; Caliskan et al., 2017; Kurita et al., 2019; Zhao et al., 2019). The relation between extrinsic and intrinsic bias measures is understudied, but recent works have demonstrated empirically either a relatively weak or inconsistent correlation between the two (Goldfarb-Tarrant et al., 2021; Orgad et al., 2022; Cao et al., 2022; Orgad and Belinkov, 2022; Steed et al., 2022; Shen et al., 2022; Cabello et al., 2023).

heading: 7 Conclusion
text: We have formulated the notion of guardedness as the inability to directly predict a concept from the representation. We show that log-linear guardedness with respect to a binary protected attribute
does not prevent a subsequent multiclass linear classifier trained over the guarded representations from leaking information on the protected attribute. In contrast, when the main task is binary, we can bound that leakage. Altogether, our analysis suggests that the deployment of linear erasure methods should carefully take into account the manner in which the modified representations are being used later on, e.g., in classification tasks.
Limitations
Our theoretical analysis targets a specific notion of information leakage, and it is likely that it does not apply to alternative ones. While the V-informationbased approach seems natural, future work should consider alternative extrinsic bias measures as well as alternative notions of guardedness. Additionally, our focus is on the linear case, which is tractable and important—but limits the generality of our conclusions. We hope to extend this analysis to other predictive families in future work.
Ethical Considerations
The empirical experiments in this work involve the removal of binary gender information from a pretrained representation. Beyond the fact that gender is a non-binary concept, this task may have realworld applications, in particular such that relate to fairness. We would thus like to remind the readers to take the results with a grain of salt and be extra careful when attempting to deploy methods such as the one discussed here. Regardless of any theoretical result, care should be taken to measure the effectiveness of bias mitigation efforts in the context in which they are to be deployed, considering, among other things, the exact data to be used and the exact fairness metrics under consideration.

heading: Acknowledgements
text: We thank Afra Amini, Clément Guerner, David Schneider-Joseph, Nora Belrose and Stella Biderman for their thoughtful comments and revision of this paper. This project received funding from the Europoean Research Council (ERC) under the Europoean Union’s Horizon 2020 research and innovation program, grant agreement No. 802774 (iEXTRACT). Shauli Ravfogel is grateful to be supported by the Bloomberg Data Science Ph.D Fellowship. Ryan Cotterell acknowledges the Google Research Scholar program for supporting the pro-
posal “Controlling and Understanding Representations through Concept Erasure.”

heading: A Appendix
text: A.1 Composition of δ-discretized binary log-linear models Lemma A.1. Let Vδ be the family of discretized binary log-linear models (Definition 3.1). Let τ(θ⊤x+ϕ) be a linear decision rule where τ is defined as in Eq. (10), and, furthermore, assume θ⊤x+ ϕ ̸= 0 for all x. Then, for any α, β ∈ R, there exists a function r ∈ Vδ such that r(0) = ρδ(σ(α · τ(θ⊤x+ ϕ) + β)) where ρδ is defined as in Definition 3.1.
Proof. Consider the function composition σ(α · τ(θ⊤x + ϕ) + β). First, note that τ(θ⊤x + ϕ) is a step-function. And, thus, so, too, is σ(α · τ(θ⊤x+ ϕ) + β), i.e.,
ŷ(x) def = σ(α · τ(θ⊤x+ ϕ) + β) =
{ 1
1+e−β def = a, if θ⊤x+ ϕ ≤ 0
1 1+e−α−β def = b, else
(20)
This results in a classifier with the following properties, if θ⊤x+ ϕ ≤ 0, we have
p(Ŷ = 0 | X = x) = a (21) p(Ŷ = 1 | X = x) = 1− a (22)
Otherwise, if θ⊤x+ ϕ > 0, we have
p(Ŷ = 0 | X = x) = b (23) p(Ŷ = 1 | X = x) = 1− b (24)
By assumption, we have θ⊤x+ ϕ ̸= 0. Now, observe that a binary δ-discretized classifier can represent any distribution of the form
r(0) = { δ if θ⊤x+ ϕ > 0 1− δ else
(25)
r(1) = { 1− δ if θ⊤x+ ϕ > 0 δ else
(26)
We show how to represent r as a δ-discretized binary log-linear model in four cases:
• Case 1: a > 12 and b < 1 2 . In this case, we require a classifier that places probability δ on 0 if
θ⊤x+ ϕ < 0 and probability 1− δ on 0 if θ⊤x+ ϕ < 0.
• Case 2: a < 12 and b > 1 2 . In this case, we require a classifier that places probability δ on 0 if
θ⊤x+ ϕ < 0 and probability 1− δ on 0 if θ⊤x+ ϕ > 0.
• Case 3: a, b > 12 . In this case, we set θ = 0 and ϕ > 0.
• Case 4: a, b < 12 . In this case, we set θ = 0 and ϕ < 0.
This proves the result. ■
A.2 Accuracy-based Guardedness: the Balanced Case
Proposition 4.4. Let V be the family of binary log-linear models, and assume that p(X, Z) is globally balanced, i.e., p(Z = ⊥) = p(Z = ⊤) = 12 . Furthermore, let h be a guarding function that ε-guards X against Z with respect to V in terms of accuracy (Definition 4.2), i.e., IAV (h(X)→ Z) < ε. Let Ŷ be defined as in Eq. (1). Then, the L1 independence gap (Eq. (19)) satisfies GAPind(Ŷ → Z | h(X)) ≤ 4ε.
Proof. In the following proof, we use the notation X = x for the guarded variable h(X) = h(x) to avoid notational cutter. Assume, by way of contradiction, that the L1 independence gap (Eq. (19)), ∑
y ∣∣∣∣ Ex∼p(X|Z=⊥)p(Ŷ = y | Z = ⊥,X = x)− Ex∼p(X|Z=⊤)p(Ŷ = y | Z = ⊤,X = x) ∣∣∣∣ > 4ε.
Then, there exists a y ∈ Y such that
∣∣∣∣ Eh(x)∼p(h(X)|Z=⊥)p(Ŷ = y | Z = ⊥,X = x)− Ex∼p(X|Z=⊤)p(Ŷ = y | Z = ⊤,X = x) ∣∣∣∣ > 2ε. (27)
We will show that we can build a classifier q⋆ ∈ V that breaks the assumption IAV (h(X) → Z) < ε. Next, we define the random variable Ẑq for convenience as
Ẑq(z) def =    1, if z = argmax z′ q(z′ | x)
0, else (28)
In words, Ẑq is a random variable that ranges over possible predictions, derived from the argmax, of the binary log-linear model q. Now, consider the following two cases.
• Case 1: There exits a y such that E x∼p(X|Z=⊥) p(Ŷ = y | Z = ⊥,X = x)− E x∼p(X|Z) p(Ŷ = y | Z =
⊤,X = x) > 2ε. Let Ŷ be defined as in Eq. (1). Next, consider a random variable Ẑr defined as follows
p(Ẑr = ⊥ | Ŷ = y) def= { 1, if Ŷ = y 0, else
(29a)
p(Ẑr = ⊤ | Ŷ = y) def= { 1, if Ŷ ̸= y 0, else
(29b)
Now, note that we have
p(Ẑr = ⊥ | X = x) = ∑
y∈Y p(Ẑr = ⊥ | Ŷ = y)p(Ŷ = y | X = x) (30a)
= p(Ŷ = y | X = x) (30b)
and
p(Ẑr = ⊤ | X = x) = ∑
y∈Y p(Ẑr = ⊤ | Ŷ = y)p(Ŷ = y | X = x) (31a)
= p(Ŷ ̸= y | X = x) (31b)
We perform the algebra below where the step from Eq. (37c) to Eq. (37d) follows because of the fact that, despite the nuisance variable Ŷ, the decision boundary of p(Ẑr = ⊤ | X = x) is linear and,
thus, there exists a binary log-linear model in V which realizes it. Now, consider the following steps
AV (Z | X) def= sup q∈V E (x,z)∼p(Z,X) ℓ(q,x, z) (32a)
= sup q∈V E x∼p(X|Z) E z∼p(Z) ℓ(q,x, z) (32b)
= sup q∈V E x∼p(X|Z=⊥) p(Ẑq = ⊥ | Z = ⊥,X = x)p(Z = ⊥) (32c)
+ E x∼p(X|Z=⊤) p(Ẑq = ⊤ | Z = ⊤,X = x)p(Z = ⊤)
≥ E x∼p(X|Z=⊥) p(Ẑr = ⊥ | Z = ⊥,X = x)p(Z = ⊥) (32d)
+ E x∼p(X|Z=⊤) p(Ẑr = ⊤ | Z = ⊤,X = x)p(Z = ⊤)
= E x∼p(X|Z=⊥) p(Ŷ = y | Z = ⊥,X = x)p(Z = ⊥) (32e)
+ E x∼p(X|Z=⊤)
p ( Ŷ ̸= y | Z = ⊤,X = x ) p(Z = ⊤) (32f)
= E x∼p(X|Z=⊥) p(Ŷ = y | Z = ⊥,X = x)p(Z = ⊥) (32g)
+ E x∼p(X|Z=⊤) (1− p(Ŷ = y | Z = ⊤,X = x))p(Z = ⊤) (32h)
= E x∼p(X|Z=⊥)
1 2 p(Ŷ = y | Z = ⊥,X = x) (32i)
+ E x∼p(X|Z=⊤)
1 2 (1− p(Ŷ = y | Z = ⊤,X = x)) (32j)
= E x∼p(X|Z=⊥)
1 2 p(Ŷ = y | Z = ⊥,X = x) (32k)
+ E x∼p(X|Z=⊤)
1 2 − 1 2 p(Ŷ = y | Z = ⊤,X = x) (32l)
= 1
2 E x∼p(X|Z=⊥) p(Ŷ = y | Z = ⊥,X = x)− E x∼p(X|Z=⊤) p(Ŷ = y | Z = ⊤,X = x)
︸ ︷︷ ︸ >2ε by assumption
+ 1
2
(32m)
> 1
2 (2ε) +
1 2 (32n)
= 1
2 + ε (32o)
• Case 2: There exits a y such that
E x∼p(X|Z=⊤) p(Ŷ = y | Z = ⊤,X = x)− E x∼p(X|Z=⊥) p(Ŷ = y | Z = ⊥,X = x) > 2ε (33)
Let Ŷ be defined as in Eq. (1). Next, consider a random variable defined as follows
p(Ẑr = ⊥ | Ŷ = y) def= { 1, if Ŷ ̸= y 0, else
(34a)
p(Ẑr = ⊤ | Ŷ = y) def= { 1, if Ŷ = y 0, else
(34b)
Now, note that we have
p(Ẑr = ⊥ | X = x) = ∑
y∈Y p(Ẑr = ⊥ | Ŷ = y)p(Ŷ = y | X = x) (35a)
= p(Ŷ ̸= y | X = x) (35b)
and
p(Ẑr = ⊤ | X = x) = ∑
y∈Y p(Ẑr = ⊤ | Ŷ = y)p(Ŷ = y | X = x) (36a)
= p(Ŷ = y | X = x) (36b)
We proceed by algebraic manipulation
AV (Z | X) def= sup q∈V E (x,z)∼p(Z,X) ℓ(q,x, z) (37a)
= sup q∈V E x∼p(X|Z) E z∼p(Z) ℓ(q,x, z) (37b)
= sup q∈V E x∼p(X|Z=⊥) p(Ẑq = ⊥ | Z = ⊥,X = x)p(Z = ⊥) (37c)
+ E x∼p(X|Z=⊤) p(Ẑq = ⊤ | Z = ⊤,X = x)p(Z = ⊤)
≥ E x∼p(X|Z=⊥) p(Ẑr = ⊥ | Z = ⊥,X = x)p(Z = ⊥) (37d)
+ E x∼p(X|Z=⊤) p(Ẑr = ⊤ | Z = ⊤,X = x)p(Z = ⊤)
= E x∼p(X|Z=⊥) p(Ŷ ̸= y | Z = ⊥,X = x)p(Z = ⊥) (37e)
+ E x∼p(X|Z=⊤)
p ( Ŷ = y | Z = ⊤,X = x ) p(Z = ⊤) (37f)
= E x∼p(X|Z=⊥) (1− p(Ŷ = y | Z = ⊥,X = x))p(Z = ⊥) (37g)
+ E x∼p(X|Z=⊤) p(Ŷ = y | Z = ⊤,X = x)p(Z = ⊤) (37h)
= E x∼p(X|Z=⊤)
1 2 p(Ŷ = y | Z = ⊤,X = x) (37i)
+ E x∼p(X|Z=⊥)
1 2 (1− p(Ŷ = y | Z = ⊥,X = x)) (37j)
= E x∼p(X|Z=⊤)
1 2 p(Ŷ = y | Z = ⊤,X = x) (37k)
+ E x∼p(X|Z=⊥)
1 2 − 1 2 p(Ŷ = y | Z = ⊥,X = x) (37l)
= 1
2 E x∼p(X|Z=⊤) p(Ŷ = y | Z = ⊤,X = x)− E x∼p(X|Z=⊥) p(Ŷ = y | Z = ⊥,X = x)
︸ ︷︷ ︸ >2ε by assumption
+ 1
2
(37m)
> 1
2 (2ε) +
1 2 (37n)
= 1
2 + ε (37o)
In both cases, we have AV (Z | X = x) > 12 + ε. Thus, Ex∼p(X)AV (Z | X = x) = AV (Z | h(X)) ≥
1 2 + ε. Note that the distribution p(Z,X) is globally balanced, we have AV (Z) = 1 2 . Thus,
IAV (h(X)→ Z) = AV (Z)−AV (Z | h(X)) (38a)
= AV (Z | h(X))− 1
2 (38b)
> 1 2 + ε− 1 2 (38c) = ε (38d)
However, this contradicts the assumption that IAV (h(X)→ Z) < ε. This completes the proof. ■
A.3 Experimental Setting In this appendix, we give additional information necessary to replicate our experiments (§ 5).
Data. We use the same train–dev–test split of the biographies dataset used by Ravfogel et al. (2020), resulting in training, evaluation and test sets of sizes 255,710, 39,369, and 98,344, respectively. We reduce the dimensionality of the representations to 256 using PCA. The dataset is composed of short biographies, annotated with both gender and profession. We randomly sampled 15 pairs of professions from the dataset: (professor, attorney), (journalist, surgeon), (physician, nurse), (professor, physician), (psychologist, teacher), (attorney, teacher), (physician, journalist), (professor, dentist), (teacher, surgeon), (psychologist, surgeon), (photographer, surgeon), (attorney, psychologist), (physician, teacher), (professor, teacher), (professor, psychologist)
Optimization. We run RLACE (Ravfogel et al., 2022a) with a simple SGD optimization, with a learning rate of 0.005, a weight decay of 1e−5 and a momentum of 0.9, chosen by experimenting with the development set. We use a batch size of 128. The algorithm is based on an adversarial game between a predictor that aims to predict gender, and an orthogonal projection matrix adversary that aims to prevent gender classification. We choose the adversary which yielded highest classification loss. All training is done on a single NVIDIA GeForce GTX 1080 Ti GPU.
Estimating V-information. After running RLACE, we get an approximately linearly-guarded representation by projecting xn ← Pxn, where P is the orthogonal projection matrix returned by RLACE. We validate guardedness by training log-linear models over the projected representations; they achieve accuracy less than 2% above the majority accuracy. Then, to estimate IV(Ŷa → Z), we fit a simple neural network of the form of a composition of two log-linear models. The inner model either has a single hidden neuron with a logistic activation (in the binary experiment), or K = 2, 4, 8, 16, 32, 64 hidden neurons with softmax activations, in the multiclass experiment (§ 5.3). The networks are trained end to end to recover binary gender for 25000 batches of size 2048. Optimization is done with Adam with the default parameters. We use the loss of the second log-linear model to estimate IV(Ŷa → Z), according to Definition 2.2.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
"Limitations"
3 A2. Did you discuss any potential risks of your work? Ethics section
3 A3. Do the abstract and introduction summarize the paper’s main claims? absract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 6
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: While the V-informationbased approach seems natural, future work should consider alternative extrinsic bias measures as well as alternative notions of guardedness