Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projection
Year: 2023

Abstract:
Natural language processing models tend to learn and encode social biases present in the data. One popular approach for addressing such biases is to eliminate encoded information from the model’s representations. However, current methods are restricted to removing only linearly encoded information. In this work, we propose Iterative Gradient-Based Projection (IGBP), a novel method for removing non-linear encoded concepts from neural representations. Our method consists of iteratively training neural classifiers to predict a particular attribute we seek to eliminate, followed by a projection of the representation on a hypersurface, such that the classifiers become oblivious to the target attribute. We evaluate the effectiveness of our method on the task of removing gender and race information as sensitive attributes. Our results demonstrate that IGBP is effective in mitigating bias through intrinsic and extrinsic evaluations, with minimal impact on downstream task accuracy.1

heading: 1 Introduction
text: The increasing reliance on natural language processing models in decision-making systems has led to a renewed focus on the potential biases that these models may encode. Recent studies have demonstrated that word embeddings exhibit gender bias in their associations of professions (Bolukbasi et al., 2016; Caliskan et al., 2017) and that learned representations of language models capture demographic data about the writer of the text, such as race or age (Blodgett et al., 2016; Elazar and Goldberg, 2018). Model decisions can be affected by these encoded biases and irrelevant attributes, leading to a wide range of inequities toward certain demographics. For example, a model designed to
∗Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion.
1Code is available at https://github.com/ technion-cs-nlp/igbp_nonlinear-removal.
review job resumes should not factor in the applicants’ gender or race. Consequently, it is desirable to be able to manipulate the type of data encoded within text representations and to exclude any sensitive information in order to create more fair and equitable models.
Removing the presence of sensitive attributes from the representations learned by deep neural networks is non-trivial, as these representations are often learned using complex and hard-to-interpret non-linear models. Re-training the language model can be a costly solution, therefore post-hoc removal methods that work at the representation layer have been proposed, such as linear projection of the embeddings on a hyperplane that distinguishes between the sensitive attribute (Bolukbasi et al., 2016; Ravfogel et al., 2020). However, neural networks do not necessarily represent concepts in a linear manner. To address this issue, Ravfogel et al. (2022b) proposed kernelization of a linear minimax game for concept erasure, but this approach is restricted to the selection of kernel and the attribute protection does not transfer to different types of non-linear probes. Accordingly, Ravfogel et al. (2022a,b) considered non-linear concept erasure to
5961
be an open problem. In this paper, we propose a non-linear concept erasure method, IGBP, to eliminate information about the protected attribute from neural representations. We use a trained probe classifier that attempts to predict the protected attribute and a novel loss function suited for the task of concept removal. Then, we leverage the gradients of this loss to guide for projection of the representations to a hypersurface that does not contain information used by the classifier regarding the sensitive-attribute. This is done by projecting the representations to the separating boundary of the classifier. Figure 1 illustrates a 2-dimensional example.
Our approach supports the use of non-linear neural classifiers. When used with a linear classifier, it is equivalent to Iterative Null Space Projection (INLP), a popular linear concept removal method (Ravfogel et al., 2020).
We perform an empirical evaluation of the proposed method using: (1) intrinsic evaluation of word embeddings measuring word-level gender bias removal and (2) extrinsic fair classification evaluation over tasks that uses contextualized word representations. The empirical results show that the proposed method is successful in sensitiveattribute removal and mitigating bias, outperforming competing algorithms with minimal impact on the downstream task accuracy.

heading: 2 Related Work
text: Many studies (e.g., Caliskan et al., 2017; Rudinger et al., 2018) investigated social biases in word embeddings and text representations. Recent work have showed how applications that use pre-trained representations reflect and amplify these kinds of social biases (Zhao et al., 2018; Elazar and Goldberg, 2018).
The approaches tackling this problem can be categorized into three lines of work: pre-processing methods which manipulate the input distribution before training (e.g., Zhao et al., 2018; Wang et al., 2019), in-processing methods which focus on learning fair models during training (e.g., Xie et al., 2017; Beutel et al., 2017; Zhang et al., 2018; Orgad and Belinkov, 2023) and post-hoc methods (e.g., Ravfogel et al., 2020; Wang et al., 2020; Ravfogel et al., 2022a,b), which assume a fixed, pre-trained set of representations from any encoder and aim to learn a new set of unbiased representations.
Since re-training a model can be costly, a lot of
focus was given to post-hoc methods, which is the main focus of this work.
The most common post-hoc approach to remove sensitive information from word embeddings is to use a linear projection. Bolukbasi et al. (2016) identified a gender subspace, which is a subspace spanned by the directions of embeddings that capture the bias, such as the direction “he” – “she”. They suggested projecting all the gender-neutral word embeddings on the gender subspace’s first principle component to make neutral words equally distant from male and female-gendered words. However, Gonen and Goldberg (2019) showed that this method only covers up bias and not fully removes it from the representation. Another critical drawback of the method is that it requires user selection of a few gender directions.
Ravfogel et al. (2020) tried to overcome this drawback of manually defining gender direction, and presented the Iterative Null-space Projection (INLP) method. It is based on training linear classifiers that predict the attribute they wish to remove, then projecting the representations on the classifiers’ null-space. Ravfogel et al. (2022a) aims to linearly remove information from neural representations by using a linear minimax gamebased approach, and derive a closed-form solution for certain objectives. One of the limitations of linear removal methods is their inability to remove non-linear information about the protected attribute, which is often encoded in text representations through complex neural networks. In contrast, our method is capable of removing both linear and non-linear information, resulting in a more effective reduction of extrinsic bias (Section 4.4).
Ravfogel et al. (2022b) proposed a nonlinear extension of the concept-removal objective of Ravfogel et al. (2022a). They identify the subspace to be neutralized in kernel space by running a kernelized version of a minimax game as in Ravfogel et al. (2022a). Shao et al. (2023) also use kernels to try and remove non-linear information While this approach aims to remove non-linear information, it can only choose data mapping from a pre-defined set of kernels, and as shown in Ravfogel et al. (2022a), the attribute protection does not transfer to other non-linear kernels . Our approach uses a deep neural network as the bias signal modeling, thus has the potential to express any non-linear function. Our empirical results (Section 4) show that our method significantly outperforms these methods on
a variety of tasks.

heading: 3 Approach

heading: 3.1 Problem Formulation
text: Given a dataset D = {xi, yi, zi}Ni=1 which consists of triples of text representation xi ∈ X , downstream task label yi ∈ Y and a protected attribute zi ∈ Z which corresponds to discrete attribute values, such as gender. Our goal is to eliminate the information related to the protected attribute from the representations while minimizing the effect on other relevant information. To achieve this, we intend to learn a non-linear transformation of the representations such that the protected attribute zi cannot be inferred from the transformed representations xcleani , while still preserving the information with regard to the downstream task label yi.

heading: 3.2 Adversarial Approach Background
text: The core of our approach is to produce projection of the representations such that any classifier is unable to distinguish between the protected attribute groups. To gain some intuition about how such projections are generated, let us first consider a trained probe classifier f that classifies the attribute label z of each representation vector x. By assigning adversarial perturbations and moving in the direction of the gradient of the loss function with respect to the input vector, the representations can be modified such that the classifier’s ability to predict the protected attribute is hindered, while minimizing the alteration of other relevant information:
xnew = x+ λ · ∇xL(f(x), z), (1)
where λ > 0. Elazar and Goldberg (2018) applied a similar approach in removal of demographic attributes from text data during training. In contrast, we apply our method on the representation layer post-training, with a specific loss function and λ.
We present a novel loss for L, to which we call the projective loss. It is designed for removing information from neural representation. It allows for a single-step projection of the representations, rendering the probe classifier f oblivious to the protected attribute. Before presenting the projective loss, we explore why the common cross entropy (CE) is not optimal for our task. The CE loss function is defined as:
LCE(p, y) = { −log(p) if y=1 −log(1− p) otherwise. (2)
where y ∈ {±1} specifies the ground-truth class and p ∈ [0, 1] is the model’s estimated probability for the class with label y = 1. For the sake of clarity, we formally define pt:
pt = { p if y=1 1− p otherwise.
(3)
and rewrite CE(p, y) = CE(pt) = −log(pt). The CE loss can be seen in black in Figure 2. A noteworthy characteristic of this loss is that examples which are considered to have a strong signal of the protected attribute (i.e., are easily classified with pt ≫ 0.5) yield low gradients. In Appendix B we demonstrate mathematically that:
∇xLCE = ± (1− pt)∇xf⊤ (4)
As pt approaches 1, ∇xLCE tends to 0 and the adversarial perturbation associated with the mostbiased samples is vanishingly small. Hence, the use of gradients of the CE loss for information removal brings about a major disadvantage.

heading: 3.3 Projective Loss
text: A more effective way to remove the entire signal of bias in the representations would be projecting them on the hypersurface where the classifier is oblivious to the protected attribute. To achieve this, we propose the projective loss:
LP (pt) = 1
2 (log(pt)− log(1− pt))2 (5)
Figure 2 illustrates the behavior of the projective loss compared to the more common cross entropy loss. As can be observed, the projective loss gives
higher weights to examples where the probe classifier can predict the protected attribute well. The minimum occurs at pt = 0.5, where there is ambiguity for the probe classifier in determining the label. Eq. 1 is now modified as :
xp = x− λP · ∇xLP (f(x), z), (6)
The gradient of the projective loss can be expressed as:
∇xLP = f(x)∇xf⊤ (7) We show in Appendix C that Eq. 6 with the projective loss and a specific λP = 1∥∇xf∥2 yields a projection of the embedding vectors on the local linear model of each embedding.
Special Case of a Linear Probe Classifier. We now analyze the special case where f is a linear classifier. Given a linear classifier f(x) = x⊤θ where θ ∈ Rd and a logistic function σ(f) =
1
1+e−x⊤θ to produce the probability pt, we calculate the gradients of the projective loss as:
∇xLP = (x⊤θ)θ⊤ (8)
Normalizing θ by setting λP = 1∥∇xf∥2 = 1
θ⊤θ in Eq. 6 yields the orthogonal projection formula:
xp = x− ( x⊤θ θ⊤θ )θ⊤ (9)
This is also known as the null space projection which is used in INLP (Ravfogel et al., 2020). INLP is a special case of our method when using a linear probe classifier. Unlike INLP, which obtains the projected embeddings by identifying the null space of a linear classifier, our method utilizes the gradients of neural network classifiers to obtain the projected embeddings.
INLP has been shown to be effective in removing sensitive information from neural representations (Ravfogel et al., 2020). However, as highlighted by Kumar et al. (2022), a limitation of this approach is that each step of the projection operation decreases the norm of the representation, leading to its eventual reduction to zero as the number of steps increases. Our proposed method, IGBP, addresses this issue by utilizing a non-linear probe in the projection process, which does not reduce the rank of the representations. Thus, the removal of sensitive information is performed with minimal loss of other information as demonstrated in Section 4.4.
Algorithm 1 Iterative Gradient-Based Projection (IGBP ) Input: Model representations X , protected at-
tribute Z, Stopping Criteria Sc Output: New representations Xclean, probes list F X0 ← X N ← 0 F ← [ ] while (not Sc) do
f ←TrainClassifier(XN , Z) F.append(f) XN+1 ← {} for x ∈ XN do
xp = x− ∇xLP (f(x),z)∥∇xf∥2 XN+1 ← {xp} ∪XN+1
end for N ← N + 1
end while return XN , F

heading: 3.4 Iterative Gradient-Based Projection
text: In this section we present our algorithm, Iterative Gradient-Based Projection (IGBP), for removing information of a discrete2 attribute Z for a set of vectors X. Algorithm 1 presents the IGBP algorithm, which begins by training a classifier f1 on the original representations X to predict a property Z. The projected representations X1p are obtained by applying Eq. 6 to the original representations X . Since there are often multiple hypersurfaces that can capture sensitive attribute information, this process is repeated iteratively, each time using a newly trained classifier on the previous projected representations. The optimal number of iterations and the stopping criteria are determined with metrics such as accuracy or fairness. The relationship between the number of iterations and these metrics is explored in Section 5.2.

heading: 4 Experiments
text: In this section we compare competing methods for bias removal with the IGBP algorithm in both intrinsic (Section 4.3) and extrinsic evaluations (Section 4.4), which are common in the literature on bias removal.
2This work primarily addresses the removal of discrete protected attributes (e.g., gender) information. However, in Appendix A we show it can be adapted for continuous attributes (e.g., age).

heading: 4.1 Compared Methods
text: We compare IGBP with several methods for bias mitigation, including a baseline (Original) without any concept-removal procedure.
INLP (Ravfogel et al., 2020), an iterative method that removes the protected information by projecting on the null space of linear classifiers.
RLACE (Ravfogel et al., 2022a), which removes linear concepts from the representation space as a constrained version of a minimax game where the adversary is limited to a fixed-rank orthogonal projection.
Kernelized Concept Erasure (KCE) (Ravfogel et al., 2022b), which proposes a kernelization of a linear minimax game for concept erasure.

heading: 4.2 Setup
text: In each experiment, we utilize a a one-hidden layer neural network with ReLU activation as the attribute classifier for IGBP algorithm. Then we perform 5 runs of IGBP and competing methods with random initialization and report mean and standard deviations. Further details on implementation and hyperparameter tuning are provided in Appendix D.

heading: 4.3 Intrinsic Evaluation
text: We begin by evaluating our debiasing method on GloVe (Pennington et al., 2014) word embeddings, as it has been previously shown by Bolukbasi et al. (2016) that these embeddings contain unwanted gender biases. Our goal is to remove these biases. We replicate the experiment performed by Gonen and Goldberg (2019) and use the training and test data of Ravfogel et al. (2020), where the word vectors are labeled with their respective bias: male-biased or female-biased. See Appendix D for more details on the experimental setting.

heading: 4.3.1 Embeddings Classification
text: After applying the debiasing methods, we follow the evaluation approach proposed by Gonen and Goldberg (2019) and train new classifiers, a linear SVM and a non-linear SVM with RBF kernel, to predict gender from the new representations. We define leakage as the accuracy of these classifiers. The results are shown in Table 1. As we can see,
all methods are effective at removing linearly encoded information, as the leakage is very low. However, when using non-linear classifiers, all competing methods fail to eliminate leakage, including KCE.3 Even though the adversary classifier used to calculate leakage (SVM-RBF) is different from the ReLU MLP employed in IGBP, our method is still the most effective at removing non-linearly encoded information. The results demonstrate the advantage of IGBP in eliminating non-linear information in word embeddings over competing methods.

heading: 4.3.2 WEAT Analysis
text: The Word Embeddings Association Test (Caliskan et al., 2017) is a measure of bias in static word embeddings, which compares the association of male and female related words with stereotypically male or female professions. We follow Gonen and Goldberg (2019) in defining the groups of male and females associated words. We represent the gender groups with three categories (1) art and mathematics; (2) art and science; and (3) career and family. We present the results of the WEAT test in Table 2, including the d-value and the p-value (refer to Caliskan et al. (2017) for further information). We found that IGBP has the most effective debiasing effect on word embeddings compared to other methods.

heading: 4.3.3 Semantic Similarity Analysis
text: In addition to mitigating bias in word embeddings, it is important to examine if any semantic content was damaged. We perform a semantic evaluation of the debiased word embeddings using SimLex999 (Hill et al., 2015), an annotated dataset of word
3Ravfogel et al. (2022b) also demonstrated that KCE’s attribute protection fails against other type of adversaries, even with the same kernel but different parameters.
pairs with human similarity scores for each pair. As displayed in Table 3, IGBP and other methods yield only a slight reduction in correlation. To qualitatively assess the impact of IGBP on semantic similarity in GloVe word embeddings, we provide a random sample of words and their nearest neighbors before and after debiasing in Appendix D.2. We observe minimal change to the nearest neighbors.

heading: 4.4 Extrinsic Evaluation
text: In this section we focus on evaluating IGBP in the context of classification tasks. We focus on tasks where we want to eliminate a concept from the representations to prevent the main classifier from using it, thus ensuring fair classification.

heading: 4.4.1 Evaluation Metrics
text: To measure extrinsic bias, we calculate the True Positive Rate Gap (TPR GAP) to measure the differences in performance between the different protected attribute groups.
TPRz,y = P(Ŷ = y|Z = z,Y = y) GAPz,yTPR = TPRz,y − TPRz′,y
To assign a single bias measure across all values of y, we follow Romanov et al. (2019) and calculate
the root mean square GAPzTPR in order to obtain a single bias score over all labels y:
GAPzTPR =
√ 1
|N| ∑
y∈N (GAPz,yTPR) 2 (10)
For example, in a sentiment analysis task, it is important for the model to have equal performance across all demographic groups, as measured by the TPR. This ensures that the model’s predictions are fair and not biased towards any particular group.
We report two common metrics for measuring bias in representations: : (1) Leakage, as described in Section 4.3; (2) Minimum Description Length (MDL) Compression (Voita and Titov, 2020), which serves as an indicator of the extent to which certain biases can be extracted from a model’s representations (Orgad and Belinkov, 2022). A higher compression score indicates that it is easier to extract the protected attribute from the model’s representation. Orgad et al. (2022) found that this metric highly correlates with extrinsic bias metrics. We use a ReLU MLP of two-hidden layers of size 512 as the probe classifier. We provide more details about these metrics in Appendix D.3.

heading: 4.4.2 Datasets
text: We experiment with the following two datasets (Table 4 provides a brief summary):
Bios. The Bias in Bios dataset (De-Arteaga et al., 2019) contains 394K biographies. The task is to predict a person’s occupation (out of 28 professions) based on their biography. Gender annotations are provided for each biography, and we aim to eliminate any gender-related information encoded in the representations. We split to training, development, and test sets following De-Arteaga et al. (2019). The pre-trained BERT model (Devlin et al., 2019) is used as the encoder and the final hidden layer’s [CLS] token is used as a representation for the biography. To ensure that the results are not model-specific, the experiment is replicated using the pre-trained RoBERTa model (Liu et al., 2019) as the encoder. Additionally, the experiment is conducted with fine-tuned models.
DIAL. Dialectal tweets (DIAL) is a corpus of tweets collected by Blodgett et al. (2016), where the task is to predict the sentiment of the tweet (positive or negative). Each tweet is associated with the sociolect of the author (African American English or Standard American English), which is a proxy
for the racial identity of the author. Following Ravfogel et al. (2020) setup, we filter the corpus and split the data into training, development, and test sets. We use The DeepMoji model (Felbo et al., 2017) as an encoder to produce representations.

heading: 4.4.3 Results
text: Bios. The results from the Bias in Bios experiment are summarized in Table 5. With both BERT and RoBERTa frozen pre-trained models (Table 5a), it can be observed that while INLP reduces TPR-GAP, it degrades overall performance in the process. This may be due to INLP’s limitation of decreasing representation’s rank each step. RLACE and KCE lead to a reduction in the TPR-GAP but the value remains elevated. On the other hand, our proposed method IGBP significantly reduces TPRGAP while only causing a slight decrease in main task accuracy. Furthermore, in terms of intrinsic bias, IGBP is distinguished by its effectiveness at decreasing non-linear leakage and compression. As for the results with fine-tuned models (Table 5b), it shows similar results. Other competing methods only exhibit minimal reduction in TPR-GAP, whereas our approach, IGBP, succeeds in enhancing fairness and eliminating leakage.
DIAL. Table 6 presents a summary of the results obtained on DIAL dataset. The results show that applying IGBP leads to a significant reduction in the TPR-GAP, with a statistically significant difference compared to the other methods, while maintaining a level of accuracy comparable to the original model. In terms of intrinsic evaluation of the representations, both INLP and RLACE, reduce leakage and compression, but not to the same extent as IGBP. While KCE fails to reduce bias.
On a whole, we found that our proposed method outperforms competing methods empirically in terms of reducing extrinsic and intrinsic bias, and offers a more balanced accuracy–fairness tradeoff.

heading: 5 Analysis
text: We conduct a series of analyses of our proposed method: an examination of probe’s complexity impact on debiasing and an analysis of the effect of number of iterations on performance.

heading: 5.1 Effect of Probe Complexity
text: IGBP proved to be superior to linear informationremoval methods in the experiments presented in Section 4. To further investigate the potential of reducing bias, we will explore the use of more complex non-linear probes by varying the width and depth of the neural network used as a probe in IGBP 4. Figure 3 shows the TPR-GAP score after applying 50 iterations of IGBP on Bias in Bios dataset5. As we can see, there is a noticeable reduction in TPR-GAP when using non-linear probes instead of linear probe. Applying IGBP with a growing complexity of probe classifiers (moving from left to right) also result in a lower TPR-GAP. However, the reduction is not significant. We also report that the more complex the probe, the greater
4For further details on the probes architecture used, see App.E.2
5In Appendix E we show similar results on DIAL dataset.
the accuracy drop, but not at a significant value: The maximum accuracy drop was 1.20%. To conclude, based on the results in Section 4 and this expirement, one-hidden layer probe is enough to reduce bias related to gender and race in text representation. Using more complex probes may offer some additional benefits, but the improvement will be limited.

heading: 5.2 Perforamnce – Fairness Tradeoff
text: One of the key factors that influence the effectiveness of our method is the number of iterations used. Varying the number of iterations and measuring the resulting changes in the TPR-GAP and downstream task accuracy on the DIAL development set shows that the number of iterations had a significant impact on attribute removal in the early stages (Figure 4), but eventually reached a plateau. Increasing the number of iterations also harmed downstream task accuracy, but the decrease was gradual. A similar experiment on Bias in Bios (Appendix E)
showed the same trend. The results suggest that the balance between performance and fairness can be controlled by adjusting the number of iterations or by implementing appropriate stopping criteria.

heading: 6 Conclusion
text: We presented a gradient-based method for the erasure of non-linearly encoded concepts in text representations. Its ability to remove non-linear information makes it particularly useful for addressing the complex biases that may be present in text representations learned through complex models. We empirically show the effectiveness of our approach to mitigate social biases in representations, thereby improving fairness in models’ decision-making.
Beyond mitigating bias, the Iterative GradientBased Projection method has the potential to be applied in a wide range of other contexts, such as increasing model interpretability by applying causal interventions, adapting models to new domains by removing domain-specific information and ensuring privacy by removing sensitive information. In future work, we plan to explore these and other potential applications of the proposed method.
Limitations
The proposed method has limitations in its dependence on the accuracy and performance of the probe classifier as noted in (Belinkov, 2022), and may be limited in scenarios where the dataset is small or lacks sufficient information about the protected attribute. Additionally, this approach increases inference time due to the use of a sequential debiasing classifiers. In future work, we aim to find a single probe that eliminates non-linear leakage. Finally, the proposed method aims to eliminate information about a protected attribute in neural representations. While it may align with fairness metrics such as demographic parity, it is not specifically designed to ensure them.
Ethical Considerations
Ethical considerations are of utmost importance in this work. It is essential to exercise caution and consider the ethical implications when using this method, as it has the potential to be applied in situations where fair and unbiased decision-making is critical. It is important to thoroughly evaluate the effectiveness of the method in the specific context in which it will be used, and to carefully consider
the data, fairness metrics, and overall application before deploying it. It is worth noting that our method is limited by the fact that gender is a nonbinary concept and that it does not address all forms of bias, and further research is necessary to identify and address these biases. Additionally, it is important to consider the potential risk of inadvertently increasing bias through reversing the direction of the debiasing operation in the algorithm. It is crucial to be mindful of the potential impact of this method and to approach its use with caution and care.

heading: Acknowledgment
text: This project was supported by an AI Alignment grant from Open Philanthropy, the Israel Science Foundation (grant No. 448/20), and an Azrieli Foundation Early Career Faculty Fellowship.

heading: A Continuous Attributes
text: While this work focuses on discrete attribute information-removal, we explain briefly how it can be adapted for regression problems, where the attribute is continuous (e.g., age). In discrete attribute classification tasks, Given that f is the classifier, IGBP is designed to transform each vector x to x′ onto the decision boundary of f such that f(x′) = 0. In the continuous case, where f is the attribute regressor, IGBP aims to achieve a similar result, with the goal of projecting each vector x onto a point x′ such that f(x′) = 0. Hence, each input, x, is regressed to a non-informative value of zero, meaning that the input is stripped of its information content.

heading: B Reversal Gradient of Cross Entropy
text: Let us consider a non-linear model f(x) followed by a logistic function to obtain the probabilty p =
1 1+ef(x) . Then the gradient of LCE(pt) when y = 1 is :
∇xLCE = ∂LCE ∂pt ∂pt ∂f ∂f ∂x
= −1 pt pt(1− pt)∇xf
= −(1− pt)∇xf (11)
and ∇xLCE = +(1− pt)∇xf when y = −1.

heading: C Local Linear Model Projection
text: We will now show how the projective loss update step projects each sample to its local linear model boundary. This will facilitate the probe being oblivious to the protected attribute.
Local Linearity. First, we will show that a trained ReLU neural net probe divides the embedding space into sub-regions, where in each subregion it behaves as a linear model. We will demonstrate that we can obtain the local linear model for each embedding. Let us consider a non-linear probe composed of one-hidden layer with ReLU as
an activation function:6
z = x⊤W,
h = ReLU(z),
f = h⊤θ,
p = 1
1 + e−f(x)
(12)
The activation function ReLU acts as an elementwise scalar (0 or 1) multiplication, hence h can be written as:
h = a⊙ z (13)
where a is a vector with (0,1) entries indicating the slopes of ReLU in the corresponding linear regions where z fall into. Let us define a diagonal matrix D:
D = diag(a) (14)
Then, h = Dz (15)
since doing element wise multiplication with a vector a is the same as multiplication by the diagonal matrix D. It is now possible to express the output in each sub-region in matrix form as follows:
f = h⊤θ
= (DWx)⊤θ
= x⊤(DW )⊤θ
(16)
D expresses the ReLU function, so naturally it depends on Wx, but since the weights of the probe are frozen/constant and we are doing the calculation for the sub-region where the slope of the ReLU function is constant, we can assume that D is not dependent on x in this sub-region.Thus, in each sub-region r defined by the classifier, the local linear model for this sub-region is θr defined bellow :
θr = (DW ) ⊤θ (17)
We can obtain the vector θr with the gradient of f :
∇xf = (DW )⊤θ (18) 6The extension for multiple hidden layers and different
piece-wise linear activation functions is straightforward.
Applying the chain rule with LP (pt) as in Eq. 11 for each sub-region:
∇xLP = ∂LP ∂pt ∂pt ∂p ∂p ∂f ∂f ∂x
= log( pt1−pt )
pt(1− pt) (−1)yp(1− p)∇xf⊤
= (−1)−yf(x)(−1)y∇xf⊤
= f(x)∇xf⊤
= x⊤(DW )⊤θ((DW )⊤θ)⊤(Using Eq. 16,18)
= (x⊤θr)θ⊤r (19)
Again, we can obtain θr from the gradient and divide the term with 1
θ⊤r θr to get the linear projection
of each sub-region to its linear model null space
xp = x− ( x⊤θr θ⊤r θr )θ⊤r (20)

heading: D Experiment
text: This section provides additional details on the experimental setup and results.
D.1 Implementaion details IGBP stopping criteria. In order to balance the trade-off between reducing extrinsic and intrinsic bias while preserving accuracy (as can be seen in Section 5.2), we have established a stopping criterion for our proposed method, IGBP . The criterion is based on two factors: the accuracy of a newly trained probe classifier on the protected attribute, and the main task accuracy on the development set. Specifically, we run Algorithm 1 until the newly trained probe classifier acheives within 2% abovemajority accuracy, or until the main task accuracy on the development set drops below a threshold of 0.98 of the original main task accuracy. Through empirical analysis on the development set, we have determined that this threshold yields good results for all extrinsic evaluation experiments. However, it is worth noting that this stopping criterion may be adjusted based on specific requirements for each case.
IGBP classifier type. For all experiments, we use a a ReLU MLP as the attribute classifier with a single-hidden layer of the same size as the input dimension. We train the classifier with AdamW optimizer (Loshchilov and Hutter, 2018) with learning rate of 2e−4 and batch size of 256.
Applying the algorithm for training on DIAL dataset takes about 0.5-1 hour and 1-3 hours on Bias in Bios on NVIDIA GeForce RTX 2080 Ti.
Compeing methods implementation and hyperparameters. For competing methods, we follow their implementations that can be found here7. We run the algorithms until the specific type of leakage they were trying to eliminate was no longer present. For KCE we choose RBF kernel following their selection in their paper for Bias in Bios task. We tried multiple kernels but found that RBF yeilds better results. The results of RLACE are different than those in the original paper because they used only the first 100K of training samples and applied a PCA transformation to reduce dimensions down to 300 due to the high computation time. However, we wanted to make fair comparison so we did not reduce the size of training set or the dimensionality.
Models. We used the pre-trained BERT and RoBERTa base models by Huggingface that have 110M and 123M parameters. They were fine-tuned on the proffesion prediction task in Bias in Bios using a stochastic gradient descent (SGD) optimizer with a learning rate of 5e−4, weight decay of 1e−6, and momentum of 0.90. We trained for 30, 000 batches of size 10.
D.2 GloVe Word Embeddings Experiment
We provide details about the experimental settings in the static word vectors experiment 4.3. We follow Ravfogel et al. (2020) and use uncased GloVe word embeddings of 150,000 most common words. We project all vectors on h⃗e− ⃗she direction, and select the 7500 most male-biased and female biased words. Using the same training–development–test split as Ravfogel et al. (2020), we subtract the gender-neutral words and end up with a training set of 7350, an evaluation set of 3150, and a test set of 4500.
D.2.1 Additional intrinsic evaluation We evaluate bias-by-neighbors which was proposed by (Gonen and Goldberg, 2019) and the list of professions from (Bolukbasi et al., 2016). We determine the correlation between bias-by-projection and bias-by-neighbors by calculating the percent-
7https://github.com/shauli-ravfogel/nullspace_ projection https://github.com/shauli-ravfogel/rlace-icml https://github.com/shauli-ravfogel/ adv-kernel-removal
age of top 100 neighboring words for each profession that were originally biased-by-projection towards a specific gender. Our results show mean correlation of 0.598, which is lower than the previous correlation of 0.852. In comparison, after applying INLP we find a correlation of 0.73. This suggests that while some bias-by-neighbors still remains, the debiasing effect of IGBP is significant.
D.2.2 Nearest Neighbors We demonstrated in Section 4.3.3 that debiasing using IGBP did not cause significant harm to the GloVe word embedding space as per the SimLex999 test results. To further support this, in Table 7, we present the closest neighbors to 10 randomly sampled words from the vocabulary, both before and after our debiasing procedure, as a qualitative illustration.
D.3 Extrinsic Evaluation Experiments
D.3.1 Metrics We provide additional details on the metrics used in Section 4.4
Main task model. We use sklearn’s SVM (Pedregosa et al., 2011) for the main task predictions on DIAL experiment, and sklearn’s logistic regression for Bias in Bios which is a multi-label classification task.
Leakage and MDL Compression. MDL is an information-theoretic probing which measures how efficiently a model can extract information about the labels from the inputs . In this work, we employ the online coding approach (Voita and Titov, 2020) to calculate MDL. We estimate MDL following Voita and Titov’s online coding Lonline and calcuate the compression, C, which is compared against
uniform encoding Luniform which does not require any learning from data.
C = Luniform Lonline
We evaluate our models using an online code probe, which is trained on fractions of the training dataset: [2.0, 3.0, 4.4, 6.5, 9.5, 14.0, 21.0, 31.0, 45.7, 67.6, 100]. Then we calculate leakage as the probe’s accuracy on test set when trained on the entire training set. We use a MLP with two-hidden layer of size 512 and ReLU activation as the probe classifier. This decision was made to stay consistent with previous work which employed MDL (Mendelson and Belinkov, 2021) and to have a different and more powerful adversary than the one used in IGBP .

heading: E Analysis
text: E.1 Biographies representation
We present the t-SNE (Van der Maaten and Hinton, 2008) projections of the biographies representations of BERT before and after applying IGBP.
E.2 Benefits of Non-Linear Information Removal
We present the probe architectures we use in our expirement Section 5.1. These include a linear probe with one layer, and several non-linear probes
that use ReLU activations. From left to right: onehidden layer of the same size as the input dimension, two-hidden layers with the same size as the input dimension, one-hidden layer with size of twice the input dimension, three-hidden layers with size of input dimension, one-hidden layer with size of three times the input dimension. Figure 6 shows the results of Section 5.1 experiment on DIAL dataset. We observe the same trend. The maximum accuracy drop is 1.32%.
E.2.1 Number of Iterations We conduct the same experiment of Section 5.2 on DIAL dataset and show the result in Figure 7.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethical consideration section
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? It is publicly available

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4.4.2
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? these datasets are publicly available and they are collected from the web. We are investigating gender bias and names might have a crucial part.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4.4.2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 3 in section 4.4.2
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix D.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.2

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix D.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we plan to explore these and other potential applications of the proposed method. Limitations
The proposed method has limitations in its dependence on the accuracy and performance of the probe classifier as noted in (Belinkov, 2022), and may be limited in scenarios where the dataset is small or lacks sufficient information about the protected attribute. Additionally, this approach increases inference time due to the use of a sequential debiasing classifiers. In future work, we aim to find a single probe that eliminates non-linear leakage. Finally, the proposed method aims to eliminate information about a protected attribute in neural representations. While it may align with fairness metrics such as demographic parity, it is not specifically designed to ensure them. Ethical Considerations
Ethical considerations are of utmost importance in this work. It is essential to exercise caution and consider the ethical implications when using this method, as it has the potential to be applied in situations where fair and unbiased decision-making is critical. It is important to thoroughly evaluate the effectiveness of the method in the specific context in which it will be used, and to carefully consider
the data, fairness metrics, and overall application before deploying it. It is worth noting that our method is limited by the fact that gender is a nonbinary concept and that it does not address all forms of bias, and further research is necessary to identify and address these biases. Additionally, it is important to consider the potential risk of inadvertently increasing bias through reversing the direction of the debiasing operation in the algorithm. It is crucial to be mindful of the potential impact of this method and to approach its use with caution and care