SlowGuess/ACL_2020_OCR huggingface dataset:
Shauli Ravfogel $^{1,2}$  Yanai Elazar $^{1,2}$  Hila Gonen $^{1}$  Michael Twiton $^{3}$  Yoav Goldberg $^{1,2}$
<sup>1</sup>Computer Science Department, Bar Ilan University
$^{3}$ Independent researcher
{shauli.ravfogel, yanaiei1a, hilagnn,
mtwito101, yoav.goldberg}@gmail.com
The ability to control for the kinds of information encoded in neural representation has a variety of use cases, especially in light of the challenge of interpreting these models. We present Iterative Null-space Projection (INLP), a novel method for removing information from neural representations. Our method is based on repeated training of linear classifiers that predict a certain property we aim to remove, followed by projection of the representations on their null-space. By doing so, the classifiers become oblivious to that target property, making it hard to linearly separate the data according to it. While applicable for multiple uses, we evaluate our method on bias and fairness use-cases, and show that our method is able to mitigate bias in word embeddings, as well as to increase fairness in a setting of multi-class classification.
What is encoded in vector representations of textual data, and can we control it? Word embeddings, pre-trained language models, and more generally deep learning methods emerge as very effective techniques for text classification. Accordingly, they are increasingly being used for predictions in real-world situations. A large part of the success is due to the models' ability to perform representation learning, coming up with effective feature representations for the prediction task at hand. However, these learned representations, while effective, are also notoriously opaque: we do not know what is encoded in them. Indeed, there is an emerging line of work on probing deep-learning derived representations for syntactic (Linzen et al., 2016; Hewitt and Manning, 2019; Goldberg, 2019), semantic (Tenney et al., 2019) and factual knowledge (Petroni et al., 2019). There is also evidence that they capture a lot of information regarding the
Figure 1: t-SNE projection of GloVe vectors of the most gender-biased words after  $t = 0,3,18,$  and 35 iterations of INLP. Words are colored according to being male-biased or female-biased.
demographics of the author of the text (Blodgett et al., 2016; Elazar and Goldberg, 2018).
What can we do in situations where we do not want our representations to encode certain kinds of information? For example, we may want a word representation that does not take tense into account, or that does not encode part-of-speech distinctions. We may want a classifier that judges the formality of the text, but which is also oblivious to the topic the text was taken from. Finally, and also our empirical focus in this work, this situation often arises when considering fairness and bias of language-based classification. We may not want our word-embeddings to encode gender stereotypes, and we do not want sensitive decisions on hiring or loan approvals to condition on the race, gender or age of the applicant.
We present a novel method for selectively removing specific kinds of information from a representation. Previous methods are either based on projection on a pre-specified, user-provided direction (Bolukbasi et al., 2016), or on adding an adversarial objective to an end-to-end training process (Xie et al., 2017). Both of these have benefits and limitations, as we discuss in the related work section (§2). Our proposed method, Iterative Nullspace Projection (INLP), presented in section 4, can be seen as a combination of these approaches, capitalizing on the benefits of both. Like the projection methods, it is also based on the mathematical notion of linear projection, a commonly used de
7237
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
terministic operator. Like the adversarial methods, it is data-driven in the directions it removes: we do not presuppose specific directions in the latent space that correspond to the protected attribute, but rather learn those directions, and remove them. Empirically, we find it to work well. We evaluate the method on the challenging task of removing gender signals from word embeddings (Bolukbasi et al., 2016; Zhao et al., 2018). Recently, Gonen and Goldberg (2019) showed several limitations of current methods for this task. We show that our method is effective in reducing many, but not all, of these (§4).
We also consider the context of fair classification, where we want to ensure that a classifier's decision is oblivious to a protected attribute such as race, gender or age. There, we need to integrate the projection-based method within a pre-trained classifier. We propose a method to do so in section §5, and demonstrate its effectiveness in a controlled setup (§6.2) as well as in a real-world one (§6.3).
Finally, while we propose a general purpose information-removal method, our main evaluation is in the realm of bias and fairness applications. We stress that this calls for some stricter scrutiny, as the effects of blindly trusting strong claims can have severe real-world consequences on individuals. We discuss the limitations of our model in the context of such applications in section §7.
The objective of controlled removal of specific types of information from neural representation is tightly related to the task of disentanglement of the representations (Bengio et al., 2013; Mathieu et al., 2016), that is, controlling and separating the different kinds of information encoded in them. In the context of transfer learning, previous methods have pursued representations which are invariant to some properties of the input, such as genre or topic, in order to ease domain transfer (Ganin and Lempitsky, 2015). Those methods mostly rely on adding an adversarial component (Goodfellow et al., 2014; Ganin and Lempitsky, 2015; Xie et al., 2017; Zhang et al., 2018) to the main task objective: the representation is regularized by an adversary network, that competes against the encoder, trying to extract the protected information from its representation.
While adversarial methods showed impressive performance in various machine learning tasks, and
were applied for the goal of removal of sensitive information (Elazar and Goldberg, 2018; Coavoux et al., 2018; Resheff et al., 2019; Barrett et al., 2019), they are notoriously hard to train. Elazar and Goldberg (2018) have evaluated adversarial methods for the removal of demographic information from representations. They showed that the complete removal of the protected information is nontrivial: even when the attribute seems protected, different classifiers of the same architecture can often still succeed in extracting it. Another drawback of these methods is their reliance on a main-task loss in addition to the adversarial loss, making them less suitable for tasks such as debiasing pretrained word embeddings.
Xu et al. (2017) utilized a "nullspace cleaning" operator for increasing privacy in classifiers. They remove from the input a subspace that contains (but is not limited to) the nullspace of a pre-trained classifier, in order to clean information that is not used for the main task (and might be protected), while minimally impairing classification accuracy. While similar in spirit to our method, several key differences exist. As the complementary setting - removing the nullspace of the main-task classifier vs. projection onto the nullspace of protected attribute classifiers - aims to achieve a distinct goal (privacy preserving), there is no notion of exhaustive cleaning. Furthermore, they do not remove protected attributes that are used by the classifier (e.g. when it conditions on gender).
A recent line of work focused on projecting the representation to a subspace which does not encode the protected attributes. Under this method, one identifies directions in the latent space that correspond to the protected attribute, and removes them. In a seminal work, Bolukbasi et al. (2016) aimed to identify a "gender subspace" in word-embedding space by calculating the main directions in a subspace spanned by the differences between gendered word pairs, such as  $\vec{he} -\vec{she}$ . They suggested to zero out the components of neutral words in the direction of the "gender subspace" first principle components, and actively pushed neutral words to be equally distant from male and female-gendered words. However, Gonen and Goldberg (2019) have shown that these methods only cover up the bias and that in fact, the information is deeply ingrained in the representations. A key drawback of this approach is that it relies on an intuitive selection of  $a$  few (or a single) gender directions, while, as we
7238
Figure 2: Nullspace projection for a 2-dimensional binary classifier. The decision boundary of  $W$  is  $W$ 's null-space.
reveal in our experiments, the gender subspace is actually spanned by dozens to hundreds of orthogonal directions in the latent space, which are not necessarily as interpretable as the  $\vec{he} -\vec{she}$  direction. This observation aligns with the analysis of Ethayarajh et al. (2019) who demonstrated that debiasing by projection is theoretically effective provided that one removes all relevant directions in the latent space.
3 Objective and Definitions
Our main goal is to "guard" sensitive information, so that it will not be encoded in a representation. Given a set of vectors  $x_{i} \in \mathbb{R}^{d}$ , and corresponding discrete attributes  $Z$ ,  $z_{i} \in \{1, \dots, k\}$  (e.g. race or gender), we aim to learn a transformation  $g: \mathbb{R}^{d} \to \mathbb{R}^{d}$ , such that  $z_{i}$  cannot be predicted from  $g(x_{i})$ . In this work we are concerned with "linear guarding": we seek a guard  $g$  such that no linear classifier  $w(\cdot)$  can predict  $z_{i}$  from  $g(x_{i})$  with an accuracy greater than that of a decision rule that considers only the proportion of labels in  $Z$ . We also wish for  $g(x_{i})$  to stay informative: when the vectors  $x$  are used for some end task, we want  $g(x)$  to have as minimal influence as possible on the end task performance, provided that  $z$  remains guarded. We use the following definitions:
Guarded w.r.t. a hypothesis class Let  $X = x_{1}, \ldots, x_{m} \in \mathcal{X} \subseteq \mathbb{R}^{d}$  be a set of vectors, with corresponding discrete attributes  $Z, z_{i} \in \{1, \ldots, k\}$ . We say the set  $X$  is guarded for  $Z$  with respect to hypothesis class  $\mathcal{H}$  (conversely  $Z$  is guarded in  $X$ ) if there is no classifier  $W \in \mathcal{H}$  that can predict  $z_{i}$  from  $x_{i}$  at better than guessing the majority class.
Guarding function A function  $g:\mathbb{R}^n\to \mathbb{R}^n$  is said to be guarding  $X$  for  $Z$  (w.r.t. to class  $\mathcal{H}$ ) if the set  $\{g(x)|x\in X\}$  is guarded for  $Z$  w.r.t. to  $\mathcal{H}$ .
We use the term linearly guarded to indicate guarding w.r.t. to the class of all linear classifiers.
4 Iterative Nullspace Projection
Given a set of vectors  $x_{i} \in \mathbb{R}^{d}$  and a set of corresponding discrete protected attributes  $z_{i} \in \mathcal{Z}$ , we seek a linear guarding function  $g$  that removes the linear dependence between  $\mathcal{Z}$  and  $\mathcal{X}$ .
We begin with a high-level description of our approach. Let  $c$  be a trained linear classifier, parameterized by a matrix  $W \in \mathbb{R}^{k \times d}$ , that predicts a property  $z$  with some accuracy. We can construct a projection matrix  $P$  such that  $W(Px) = 0$  for all  $x$ , rendering  $W$  useless on dataset  $\mathcal{X}$ . We then iteratively train additional classifiers  $W'$  and perform the same procedure, until no more linear information regarding  $\mathcal{Z}$  remains in  $X$ . Constructing  $P$  is achieved via nullspace projection, as described below. This method is the core of the INLP algorithm (Algorithm 1).
Nullspace Projection The linear interaction between  $W$  and a new test point  $x$  has a simple geometric interpretation:  $x$  is projected on the subspace spanned by  $W$ 's rows, and is classified according to the dot product between  $x$  and  $W$ 's rows, which is proportional to the components of  $x$  in the direction of  $W$ 's rowpsace. Therefore, if we zeroed all components of  $x$  in the direction of  $W$ 's row-space, we removed all information used by  $W$  for prediction: the decision boundary found by the classifier is no longer useful. As the orthogonal component of the rowspace is the nullspace, zeroing those components of  $x$  is equivalent to projecting  $x$  on  $W$ 's nullspace. Figure 2 illustrates the idea for the 2 dimensional binary-classification setting, in which  $W$  is just a 2-dimensional vector.
For an algebraic interpretation, recall that the null-space of a matrix  $W$  is defined as the space  $N(W) = \{x|Wx = 0\}$ . Given the basis vectors of  $N(W)$  we can construct a projection matrix  $P_{N(W)}$  into  $N(W)$ , yielding  $W(P_{N(W)}x) = 0 \forall x$ .
This suggests a simple method for rendering  $z$  linearly guarded for a set of vectors  $X$ : training a linear classifier that is parameterized by  $W_{0}$  to predict  $Z$  from  $X$ , calculating its nullspace, finding the orthogonal projection matrix  $P_{N(W_0)}$  onto the nullspace, and using it to remove from  $X$  those components that were used by the classifier for predicting  $Z$ .
<sup>1</sup>While this work focuses on the discrete case, the extension to a linear regression setting is straightforward: A projection to the nullspace of a linear regressor  $w$  enforces  $w x = 0$  for every  $x$ , i.e., each input is regressed to the non-informative value of zero.
7239
Note that the orthogonal projection  $P_{N(w_0)}$  is the least harming linear operation to remove the linear information captured by  $W_0$  from  $X$ , in the sense that among all maximum rank (which is not full, as such transformations are invertible—hence not linearly guarding) projections onto the nullspace of  $W_0$ , it carries the least impact on distances. This is so since the image under an orthogonal projection into a subspace is by definition the closest vector in that subspace.
Iterative Projection Projecting the inputs  $X$  on the nullspace of a single linear classifier does not suffice for making  $Z$  linearly guarded: classifiers can often still be trained to recover  $z$  from the projected  $x$  with above chance accuracy, as there are often multiple linear directions (hyperplanes) that can partially capture a relation in multidimensional space. This can be remedied with an iterative process: After obtaining  $P_{N(W_0)}$ , we train classifier  $W_{1}$  on  $P_{N(W_0)}X$ , obtain a projection matrix  $P_{N(W_1)}$ , train a classifier  $W_{2}$  on  $P_{N(W_1)}P_{N(W_0)}X$  and so on, until no classifier  $W_{m+1}$  can be trained. We return the guarding projection matrix  $P = P_{N(W_m)}P_{N(W_{m-1})}\dots P_{N(W_0)}$ , with the guarding function  $g(x) = Px$ . Crucially, the  $i$ th classifier  $W_{i}$  is trained on the data  $X$  after the projection on the nullspaces of classifiers  $W_{0}, \ldots, W_{i-1}$  and is therefore trained to find separating planes that are independent of the separating planes found by previous classifiers.
In Appendix §A.1 we prove three desired properties of INLP: (1) any two protected-attribute classifiers found in INLP are orthogonal (Lemma A.1); (2) while in general the product of projection matrices is not a projection, the product  $P$  calculated in INLP is a valid projection (Corollary A.1.2); and (3) it projects any vector to the intersection of the nullspaces of each of the classifiers found in INLP, that is, after  $n$  INLP iterations,  $P$  is a projection to  $N(W_0) \cap N(W_1) \cdots \cap N(W_n)$  (Corollary A.1.3). We further bound the damage  $P$  causes to the structure of the space (Lemma A.2). INLP can thus be seen as a linear dimensionality-reduction method, which keeps only those directions in the latent space which are not indicative of the protected attribute.
During iterative nullspace projection, the property  $z$  becomes increasingly linearly-guarded in  $Px$ . For binary protected attributes, each intermediate  $W_{j}$  is a vector, and the nullspace rank is  $d - 1$ . Therefore, after  $n$  iterations, if the original rank of
$X$  was  $r$ , the rank of the projected input  $g(X)$  is at least  $r - n$ . The entire process is formalized in Algorithm 1.
INLP bears similarities to Partial Least Squares (PLS; Geladi and Kowalski (1986); Barker and Rayens (2003)), an established regression method. Both iteratively find directions that correspond to  $Z$ : while PLS does so by maximizing covariance with  $Z$  (and is thus less suited to classification), INLP learns the directions by training classifiers with an arbitrary loss function. Another difference is that INLP focuses on learning a projection that neutralizes  $Z$ , while PLS aims to learn low-dimensional representation of  $X$  that keeps information on  $Z$ .
Implementation Details A naive implementation of Algorithm 1 is prone to numerical errors, due to the accumulative projection-matrices multiplication  $P \gets P_{N(W_i)}P$ . To mitigate that, we use the formula of Ben-Israel (2015), which connects the intersection of nullspaces with the projection matrices to the corresponding rowspaces:
$$
N \left(w _ {1}\right) \cap \dots \cap N \left(w _ {n}\right) = N \left(P _ {R} \left(w _ {1}\right) + \dots + P _ {R} \left(w _ {n}\right)\right) \tag {1}
$$
Where  $P_{R}(W_{i})$  is the orthogonal projection matrix to the row-space of a classifier  $W_{i}$ . Accordingly, in practice, we do not multiply  $P \gets P_{N(W_i)}P$  but rather collect rowspace projection matrices  $P_{R(W_i)}$  for each classifier  $W_{i}$ . In place of each input projection  $X_{projected} \gets P_{N(W_i)}X_{projected}$ , we recalculate  $P \coloneqq P_{N(w_1) \cap \dots \cap N(w_i)}$  according to 1, and perform a projection  $X_{projected} \gets PX$ . Upon
termination, we once again apply 1 to return the final nullspace projection matrix  $P_{N(W_1)\cap \ldots \cap N(W_n)}$ . The code is publicly available.2
5 Application to Fair Classification
The previous section described the INLP method for producing a linearly guarding function  $g$  for a set of vectors. We now turn to describe its usage in the context of providing fair classification by a (possibly deep) neural network classifier.
In this setup, we are given, in addition to  $X$  and  $Z$  also labels  $Y$ , and wish to construct a classifier  $f: X \to Y$ , while being fair with respect to  $Z$ . Fairness in classification can be defined in many ways (Hardt et al., 2016; Madras et al., 2019; Zhang et al., 2018). We focus on a notion of fairness by which the predictor  $f$  is oblivious to  $Z$  when making predictions about  $Y$ .
To use linear guardedness in the context of a deep network, recall that a classification network  $f(x)$  can be decomposed into an encoder enc followed by a linear layer  $W$ :  $f(x) = W \cdot enc(x)$ , where  $W$  is the last layer of the network and enc is the rest of the network. If we can make sure that  $Z$  is linearly guarded in the inputs to  $W$ , then  $W$  will have no knowledge of  $Z$  when making its prediction about  $Y$ , making the decision process oblivious to  $Z$ . Adversarial training methods attempt to achieve such obliviousness by adding an adversarial objective to make  $enc(x)$  itself guarding. We take a different approach and add a guarding function on top of an already trained enc.
We propose the following procedure. Given a training set  $X, Y$  and protected attribute  $Z$ , we first train a neural network  $f = W \cdot enc(X)$  to best predict  $Y$ . This results in an encoder that extracts effective features from  $X$  for predicting  $Y$ . We then consider the vectors  $enc(X)$ , and use the INLP method to produce a linear guarding function  $g$  that guards  $Z$  in  $enc(X)$ . At this point, we can use the classifier  $W \cdot g(enc(x))$  to produce oblivious decisions, however by introducing  $g$  (which is lower rank than  $enc(x)$ ) we may have harmed  $W$ 's performance. We therefore freeze the network and fine-tune only  $W$  to predict  $Y$  from  $g(enc(x))$ , producing the final fair classifier  $f'(x) = W' \cdot g(enc(x))$ . Notice that  $W'$  only sees vectors which are linearly guarded for  $Z$  during its training, and therefore cannot take  $Z$  into ac
count when making its predictions, ensuring fair classification.
We note that our notion of fairness by obliviousness does not, in the general case, correspond to other fairness metrics, such as equality of odds or of opportunity. It does, however, correlate with fairness metrics, as we demonstrate empirically.
Further refinement. Guardedness is a property that holds in expectation over an entire dataset. For example, when considering a dataset of individuals from certain professions (as we do in §6.3), it is possible that the entire dataset is guarded for gender, yet if we consider only a subset of individuals (say, only nurses), we may still be able to recover gender with above majority accuracy, in that sub-population. As fairness metrics are often concerned with classification behavior also within groups, we propose the following refinement to the algorithm, which we use in the experiments in §6.2 and §6.3: in each iteration, we train a classifier to predict the protected attribute not on the entire training set, but only on the training examples belonging to a single (randomly chosen) main-task class (e.g. profession). By doing so, we push the protected attribute to be linearly guarded in the examples belonging to each of the main-task labels.
6.1 “Debiasing” Word Embeddings
In the first set of experiments, we evaluate the INLP method in its ability to debias word embeddings (Bolukbasi et al., 2016). After "debiasing" the embeddings, we repeat the set of diagnostic experiments of Gonen and Goldberg (2019).
Data. Our debiasing targets are the uncased version of GloVe word embeddings (Zhao et al., 2018), after limiting the vocabulary to the 150,000 most common words. To obtain labeled data  $X, Z$  for this classifier, we use the 7,500 most male-biased and 7,500 most female-biased words (as measured by the projection on the  $\vec{he} - \vec{she}$  direction), as well as 7,500 neutral vectors, with a small component (smaller than 0.03) in the gender direction. The data is randomly divided into a test set (30%), and training and development sets (70%, further divided into 70% training and 30% development examples).
Procedure We use a  $L_{2}$ -regularized SVM classifier (Hearst et al., 1998) trained to discriminate
$^{2}$ https://github.com/Shaul1321/nullspace_projection
7241
between the 3 classes: male-biased, female-biased and neutral. We run Algorithm 1 for 35 iterations.
Classification. Initially, a linear SVM classifier perfectly discriminates between the two genders (100% accuracy). The accuracy drops to 49.3% following INLP. To measure to what extent gender is still encoded in a nonlinear way, we train a 1-layer ReLU-activation MLP. The MLP recovers gender with accuracy of 85.0%. This is expected, as the INLP method is only meant to achieve linear guarding<sup>3</sup>.