Probing Classifiers are Unreliable for Concept Removal and Detection
Authors: Abhinav Kumar
Chenhao Tan
Amit Sharma
Venue: NeurIPS 2022 Accept (2022)
Keywords: Probing
Null-Space Removal
Adversarial Removal
Spurious Correlation
Fairness

Abstract: Neural network models trained on text data have been found to encode undesirable linguistic or sensitive concepts in their representation. Removing such concepts is non-trivial because of a complex relationship between the concept, text input, and the learnt representation. Recent work has proposed post-hoc and adversarial methods to remove such unwanted concepts from a model's representation. Through an extensive theoretical and empirical analysis, we show that these methods can be counter-productive: they are unable to remove the concepts entirely, and in the worst case may end up destroying all task-relevant features. The reason is the methods' reliance on a probing classifier as a proxy for the concept. Even under the most favorable conditions for learning a probing classifier when a concept's relevant features in representation space alone can provide 100% accuracy, we prove that a probing classifier is likely to use non-concept features and thus post-hoc or adversarial methods will fail to remove the concept correctly. These theoretical implications are confirmed by experiments on models trained on synthetic, Multi-NLI, and Twitter datasets. For sensitive applications of concept removal such as fairness, we recommend caution against using these methods and propose a spuriousness metric to gauge the quality of the final classifier.

Probing Classifiers are Unreliable for Concept Removal and Detection
Abhinav Kumar Microsoft Research t abwumar@miiii www.mccoosoo...omm com
Chenhao Tan University of Chicago chenhao@uchicago.. edu
Amit Sharma Microsoft Research
amshar@microsoft. com
Abstract
Neural network models trained on text data have been found encode undesirable linguistic or sensitive concepts in their representation. Removing such concepts is non-trivial because complex relationship between the concept, text input, and the learnt representation. Recent work has proposed post-hoc and adversarial meth- ods to remove such unwanted concepts from a model's representation. Through an extensive theoretical and empirical analysis, counter-prodiee pronter-roduutive:: they are unable to remove the concepts entirely, and in some cases may fail severely by destroying all task-relevant features. The reason is the methods' reliance on a probing classifier as a proxy for the concept. Even under the most favorable conditions for learning probing classifier when concept's rel- evant features in representation space alone can provide 100% accuracy, we prove that probing classifier is likely adversarial methods will fail to remove the concept correctly. These theoretical im- plications are confirmed by experiments on models trained on synthetic, Multi-NLI, and Twitter datasets. For sensitive applications of concept removal such as fairness, we recommend caution against using these methods and propose spuriousness metric to gauge the quality of the final classifier.
show that these methods can be
non-concept features and thus post-hoc
1 Introduction
Neural models in text classification have been shown to learn spuriously correlated features [6] 27 embed sensitive attributes like gender
8 7 in their representation layer. Classifiers that use such sensitive or spurious concepts (henceforth concepts) raise concerns of model unfairness and out-of-distribution generalization failure [39 2 15). Removing the influence of these concepts is non-trivial because the classifiers are based on hard-to-interpret deep neural networks. Moreover, since many concepts cannot be modified the input tokens level, removal methods that work the representation layer have been proposed: 1) post-hoc removal 7
49 [4] on pre-trained model (e.g., 48
null space projection
(2)) and 2) adversarial removal
[5
12 by jointly training the main task
classifier with an (adversarial) classifier for the concept.
In this paper, theoretically show that both these classes of methods be counter-productio in real-world settings where the main task label often correlated with the concept. Examples include natural language inference (main task) where the presence of negation (spurious concept) may be correlated with the "contradicts" label; or tweet sentiment classification (main task) where the author's gender (sensitive concept) maybe correlated with the sentiment label. Our key result is based on the observation that both these methods internally use an auxiliary (or probing) classifier 42 that aims to predict the spurious concept based on the representation learnt by the main classifier.
We show that an auxiliary classifier cannot a reliable signal on whether the representation includes features that causally derived from the concept. As previous work has argued 4 44 40 2 if the representation features causally derived from the concept are not predictive enough, the probing classifier for the concept be expected to rely correlated features obtain higher accuracy.
36th Conference on Ncural Information Processing Systems (NcurIPS 2022).

---

However, we show a stronger result: this behavior holds even when there is no potential accuracy gain and the concept's features are easily learnable. Specifically, even when the concept's causally- related features alone can provide 100% accuracy and are linearly separable with respect to a binary probing task label, the probing classifier may still learn non-zero weights for the correlated main-task relevant features. Based on this result, under some simplifying assumptions, we prove that both post-hoc and adversarial training methods can fail to remove the undesired concept, remove useful task-relevant features in addition to the undesired concept, or do both. As severe failure mode, we show that post-hoe removal methods can lead to a random-gues guess main-task classifier by removing all task-relevant information from the representation.
Empirical results on four datasets- natural language inference, sentiment analysis, tweet-mention detection, and a synthetic lask-confirm our claims. Across all datasets, as the correlation between the main task and the concept increases, post-hoe removal using null space projection removes a higher amount of the main-task features, eventually leading to a random-gues, classifier. In particular, for a pre-trained classifier that does not use the concept all, the method modifies the representation to yield a classifier that either uses the concept or has lower main-task accuracy, irrespective of the correlation between the main task and the concept. Similarly, for the adversarial removal method, we find that it does not remove all concept-related features. For most datasets, the concept features left within an classifier's ' representation are comparable to that for a standard main-task classifier.
Our theoretical analysis complements past empirical critiques of adversarial methods for concept removal [12. More generally, we extend the literature on probing classifiers and their unreliability 4 Adding to known limitations of explainability methods [19 35 based on the accuracy of a probing classifier, our results show that recent causally-inspired methods like amnesic probing [33 are also flawed because they depend on access to good quality concept classifier. Our contributions include:
Theoretical analysis of null space and adversarial removal methods showing that they fail to remove undesirable concept from model's representation, even under favorable conditions.
Empirical results on four datasets showing that the methods are unable to remove spurious concept's features fully and end up unnecessarily removing task-relevant features.
A practical spuriousness score for evaluating the output of concept removal methods.
2 Concept removal: Background and problem statement
For classification task, let (ar', yin) be set of examples in the dataset Dmm where x E X are the input features and V'n, Ym. the label. We call this the main task and label y'n the main task label. The main task classifier can be written as Cm (h())) where X - Z is an encoder mapping the input æ to a latent representation z := h(x) and Cm Z - Yn, is the classifier on top of the representation Z. Additionally, we are given labels for a spurious or sensitive concept, yp E Yp. i.e.,
(x', y/ in dataset D,, and our goal is to ensure that the representation h(x) learnt by the main classifier does not include features causally derived from the concept. Below we define what it means to be "causally derived":: the representation should not change under an intervention on concept.
Definition 2.1. (Concept-causal feature) A feature Zı E Z (jth dimension of h(())) at the repre- sentation layer is defined to be causally derived from concept (concept-causee for short) if upon changing the value of the concept, the corresponding change in the input's value x will lead to change in the feature's 's (Zz) value.
For simplicity, we assume that the non-concept-causal features are the main task features. Often, the main task and the concept label are correlated; hence the learnt representation h(x) for the main task may include concept-causs features too. A concept removal algorithm is said to be successful if produces a clean representation h' (x(x) to be used by the main mlan-classiffer that has concept-causal features and it does not corrupt or removes the main-task features. If the representation does not contain such features, the main classifier cannot use them [12] In practice, it is okay if the concept- causal features are not completely removed, but our key criterion is that the removal process should not remove the correlated main task features.
Existing concept removal methods. When the text input can be changed based on changing the value of concept label, methods like data augmentation 23 50 have been proposed for concept removal. However, for most sensitive or spurious concepts, it is not possible to know the correct change to apply at the input level corresponding to change in the concept's 's value.

---

Instead, methods based on the representation layer have been proposed. To determine whether features in a representation are causally derived by the concept. these methods train an auxiliary, probing classifier Cp Z + Y, where y,, E Yp is the label of the concept we want to remove from the latent space z E Z. The accuracy of the classifier indicates the predictive information about the concept embedded in the representation. This probing classifier is then used to remove the sensitive concept from the latent representation which will ensure that the main-task classifier cannot use them. Two kinds of feature removal methods have been proposed: 1) post-koc methods such as null space removal (32 13 24 20), with removal after the main-task classifier is trained; 2) adversarial methods
that jointly train the main task with the probing classifier the adversary [15 48 33 34
For adversarial removal, recent empirical results cast doubt on the method's capability to fully remove the sensitive concept from the model's representation [12]. We extend those results with a rigorous theoretical analysis and provide experiments for both adversarial and post-hoc removal methods.
3 Attribute removal using probing classifier can be counter-productive
As mentioned above, both removal methods internally concept's 's features. In [3.1] we start off by showing that for any classification task be it probing or main-task classification, it is difficult to learn a clean classifier which doesn't 't use any spuriously correlated feature (Lemma 3.1 and Lemma [B1).. Hence the key assumption driving the use of predictive classifiers within both removal methods is incorrect. Next in $3.2 and [.3] we will show how these individual components' failure leads to the failure of both removal methods. Finally, in $3.4 we propose practical spuriousness score to assess the output classifier from any of the removal methods. Throughout this section, we assume that both the main task label yo and probing task label yp are binary (E (-1,1}) and there is basic, fixed encoder h converting the text input to features in the representation space (e.g., a pre-trained model like BERT [0]..
probing classifier proxy for the
3.1 Fundamental limits to learning a clean classifier: Probing and Main Classifier
Given z = h(()) and the concept label Yp. the goal of the probing task is to learn classifier (=) such that it only uses the concept-causal features and the accuracy for yp is maximized. We assume that the main task and concept labels are correlated, so it can be beneficial to use main-task features to maximize accuracy for yp- As argued in the probing literature 19 4 if there are features in z outside anncept-causal that help improve the accuracy of the classifier, classifier trained on standard losses such as cross-entropy or max -argin is expected to use those features too. Below we show a stronger result: even when there is no accuracy benefit of using non concept-causal features, we find that a probing classifier may still use those features.
Creating favorable setup for the probing classifier. Specifically, we create setting that is the most favorable for a probing classifier to use only concept-causa features: 1) no accuracy gain on using features outside of concept-causal because concept-causal features are linearly separable for concept labels, and 2) disentangled representation so that no further representation learning is required. Yet we find that trained probing classifier would use non-concept-causal features.
Assumption 3.1 (Disentangled I Latent Representation). The latent representation is disentangled and is of form [Zm, zp], where zp E Rdd are the concept-causal features and Zm E (Rdm are the main task features. Here d'm and di are the dimensions of zm. and Zp respectively.
A Assumption 3.2 (Concept-causal Feature Linear Separability). The concept- the latent representation (2) are linearly separabieffully predictive for the concept labels Up Lee, V/. (Cp- 2 + bp) > 0, V(x', ((x)) in training dataset Dp for some ép E Rdd and bp E IR.
features (zp) of
The effect of spurious correlation between concept and label. Now we are ready to state the key lemma which will show that if there is spurious correlation between the main task and concept labels such that the main-task features 2m are predictive of the concept label for only few special points, then the probing classifier cp(z) will use those features. We operationalize spurious correlation as, Assumption 3.3 (Spurious Correlation). For a subset of training points S C Dp in the training dataset for a probing classifier, 2m. is linearly-separable with respect to concept label yp. i.e., 3/. (ém =" + bm)) > O'i E S, where ém dR and bm R.
For simplicity, we assume that the encoder h((.) which maps the input X to latent representation Z is frozen non non-traine -rairiiia.le. Following 229 we assume max- margin as training loss; under some
3

---

Dadsion Rounciey Nai-S Cander Class Bat
Prokesion
Null- Sual-ppice Gandia Classtfor
Prckession
Gender Fiot ng Classller (Praection (irccoon) Ias Magghee
Dicision Boundary Clean Main Task Clasaifor
Pre-proledion Piolossion laature
Gur (Smmsiva)
derbar (Bessliv))
G2
Ger (Smmiia))
Conuption
Projection Nuil Spaze
Profession
Oualat Laiel
Clean- Photrannod
Cunkln Lakel
Pastive lean Tall Label
Aagras Itan Task
Genced Label
Main Tast, Class/for Frclction Director
Muln- Accuraay a 100%
Genced Labal
Pastie Mtan Task Lubal
Magras Task Label
Pastiv tean Tul Label
Genced Label
Sagran Vtan Task
Clean Prefrainet Nain Task Cassilorr becomes urtar
(a) Clean pretrained main task (Profession) classifier.
(b) Probing (Gender) classitier with
(c) Main classifier becomes unfair after null space projection.
slanted projection direction.
Figure I: Failure mode of null space removal. Consider a main task (Profession) classifier where Gender is the spurious concept to be removed. Assume a 2-dimensional latent representation where one dimension corresponds to profession and the other to the gender feature. (a) A "clean" (fair) main task classifier that only uses the Profession feature, shown by its vertical projection direction, that is input to INLP for concept removal. Its decision boundary is orthogonal to the projection direction. (b) From Lemma 3.1 INLP trains a probing classifier for gender with slanted projection direction (ideal gender projection direction would be horizontal).
(c) For two points having the same profession
but different gender features (marked 'I'), projection the null-space ('2') has their profession
feature reversed ('3",, thus making the fair pretrained classifier become unfair (also see $.2)).
mild conditions separable data, a classifier trained using logistic/exponentiall loss converges to max-margn classifier given infinite training time [11.
Lemma 3.1. Let the latent representation be frozen and disentangled such that 2 = Zzm+ xpl
(Assm 3. y. and concept-cuuse causall features Zp are fully predictive for the concept label yp (Assm (.2) Let (2) Wy 2, where upp E TRd be the desired clean linear classifter trained using the max-margin objective ($B.1) that only uses Zp for its prediction. Let Zm be the main task features, spuriously correlated s.I. 2m are finearly-separalle w.r.t. probing task label yp for the margin points of c2) (Assm)) 3.3 Then, assuming zero-centered latent space (bp 0), concept-probing classifier Cp trained using the max-margin objective will use spurious features, i.e., cp(2)) = "w Ep + Wm Zm where w.n + and wm E BR
Proof Sketch. Starting from (z), we show that there always exists a perturbed classifier which uses the main task features and has a bigger margin than (z). Within some range of perturbation, for all margin points of c, using the main task features increases the margin by Assm 3.3 and does not
reduce the margin for non-margin points s.t. it becomes the same as the margin of Cy. Proof in $B.3
Our result shows that not just accuracy, even geometric skews in the dataset can yield an incorrect probing classifier. In $B.4 we prove that the assumptions for Lemma 3.I are both sufficient and necessary for a classifier to use non-concept-features Zm when 2p is I-dimensional. Lemma 3.] generalizes a result from 2 by using fewer assumptions (we do not restrict yn to be binary, do not assume that 2m and Zp are conditionally independent given y. and do not assume monotonicity of classifier norm with dataset size). We present a similar result for the main task classifier: under spurious correlation of concept and main task labels, the main task classifier would use concept-causal features even when 100% aceuracy can be achieved using only main task features (Lemma B.1 B.2).
3.2 Failure mode of post-hoc removal methods: Null-space removal (INLP)
The null space method 32 13
henceforth referred as INLP,, removes concept from latent space by projecting the latent space to a subspace that not discriminative of that concept. First, it estimates the subspace in the latent space discriminative of the concept we want to remove by training probing classifier cp: Z + Yp. where Y, is the concept label. Then the projection is done onto the null-space of this probing classifier which is expected to be non-discriminative of the concept. For instance, 32) use a linear probing classifier (z) to ensure that the any linear classifier cannot recover the removed concept from modified latent representation z' and hence the main task classifier (cm (2')) becomes invariant to removed concept. Also, they recommend running this removal step for multiple iterations to ensure the unwanted concept is removed completely (details are in SC.I). Below we state the failure of the null-space method using x =(()) to denote the representation 2' after & steps of INLP.

---

Theorem 3.2. Let (m) be a pre-trained main-task classifier where the latent representation [zm xpl satisfies Assm 3.1 3.2 Let cy(2) be the probing classifier used by INLP to remove the unwanted features 2p from the latent representation. Under Assm 3.3 Lemma 3.7 is satisfied for the probing classifier (2
() such that cp(2) upp Ep + Wm. Zm and Wm # 0. Then,
1. Damage in the first step of INLP. The first step of linear-INLA will corrupt the main-task features and this corruption is non- non-invertible with subsequent projection steps of INLP.
(a) Mixing: wy # O, the main task 7m and concept concept-causal causal features 2p will get mixed such that [g((z)) [2)))) for some function "f" and "g". Thus, the latent representation is no longer disentangled and removal of concept-causal features will also lead to removal of main task features.
z²(1) = [g(z x f(2 mm) +
(b) Removal: If w, O. then the first projection step of INLP will do opposite of what is intended, i.e., damage the main lask features Zm (in case 2m E IR it will completely remove 2 but have no effect on the concept-causal features Zz.
2. Removal in the long term: The L2-norm of the latent representation z decreases with every projec- tion step long as the parameters of probing classifier (w* at step "k" does not lie completely in the space spanned by parameters of previous probing classifiers, i.e., span w w....., w
2²(k-1), 25(0) and 2=(0) in direction of wh is not trivially zero. Thus, after sufficiently many steps, INLP can destroy all information in the representation s.I. 2²(xx) = (0,0).
Proof Sketch. From Lemma 3.1 in the first step, probing classifier for 2p will use 2m in addition to Zp. Consequently, the projection matrix for INLP based on the probing classifier will be incorrect, hence corrupting the main task features 2m with Zp (Ia) or damage 2m without any effect on Zp (1b). Next, we show that each step of the projection operation reduces the norm of latent representation z; thus the latent representation can go to O as the number of steps increases (2). Proof in $C
Fig. [a Ic demonstrate the mixing problem stated in Theorem 3.2, where fair classifier becomes unfair after the first step of projection. Note that after first step the main task classifier's iss accuracy will drop because of this mixing of features, affecting INLP- -ased probing methods like Amnesic Probing [13] that interpret a drop in the main classifier's accuracy after INLP projection evidence that the main classifier was using the sensitive concept.
Failure Mode:
3.3 Failure mode of adversarial removal methods
To remove the unwanted features 2p from the latent representation, adversarial removal methods jointly train the main classifier c. : Z + Ym and the probing classifier op Z - Y, by specifying Cp's 's loss as an adversarial loss. For details refer to $
As in Lemma 3.1 we assume that the encoder h X + Z mapping the input to the latent representation is frozen. To allow for the removal of the unwanted features Zp we introduce additional representation layers after it. For simplicity in the proof, we assume linear transformation to the latent representation lt2 Z + Ç. This layer is followed by the linear main-task classifier Cm : Ç + Ym.. as before. The probing classifier C. : < + Yp is trained adversarially to remove Zp from the latent representation Ç. Thus, the goal of the adversarial method can be stated as removing the information of xp from Ç. Let the main-task classifier satisfy assumptions of the generalized version of Lemma 3.1 (Lemma [B.I] (B.2). We also need an additional assumptions on the hard-to- classify margin points ensure that main-task labels and concept labels are correlated the margin points of a clean main-task classifier. Proof of the Theorem B.3 stated below is in $
Assumption 3.4 (Label Correlation on Margin Points). For the margin points of a clean classifier for the main task, the adversarial probiin labels yp and the main task labels ym are correlated, i.e.,
w.l.o.g.. ymm for all margins points of the clean main task classifier.
Theorem 3.3. Let the latent representation z satisfy Assm 3.1 and be frozen, hz(z) ) be a linear transformation over Z s.t. h22 Z + C, the main-task classifer be cm. W S. and the adversarial probing classifier be cp(C) We," -.. Let all the assumptions of Lemma B. be satisfied for muin-classifier cm(+) when using z directly as input and Assm 3.2 be satisfied on z w.r.l. the adversarial task. Let h (2) be the desired encoder which is successful in removing Zp from Ç. Then there exists an undesiredfincorree encoder he (z) s.1. h2 (z) is dependent on up and the main-task classifier cm (h((2)) has bigger margin than C. (1)(2)) and has,
5

---

Project
Protesaeo
ADV Projecton Diwcition
ADV Projection Divection
Gender Jonuary s76
Margin for Main Tack sil
Ounker
Oandar (Seraliv)
15 sy
Duater lape
Gurdent Lobel
Gar sursed lobal
Desired Projection Direction
Gendert Label
Reativ Mors Thoi Lobkl Vapatia Max
Destred Projection Drrccton Gender Accusiyy 55%
Pupules Mann Nuk
mey
Patrue toar Taxck und Segeise Mnle "boct
Desired Projection Cirection
Margin Ma Task wra
(b) Main-Task (Profession) Accuracy
(a) Probing (Gender) Accuracy
(c) Main- Task (Profession) Margin
Figure 2: Failure mode of adversarial removal. As in Fig. I the main task label Profession and
Gender is the spurious concept, each corresponding to one of the dimensions of the 2-dimensional feature representation z. Assume that the shared representation is a scalar value obtained by projecting the two features in some direction. The adversarial goal is to find a projection direction such that the concept (gender) classifier obtains a random-guee guenss accuracy of 50% but has good accuracy on the main task label (profession). (a) Two projection directions, shown by vertical and slanted lines, that yield random- guess 50% accuracy on gender prediction, and (b) have the same 100% accuracy for profession prediction. (c) However, the slanted projection direction has bigger margin for the main task and will be preferred, thus leading to final classifier that uses the gender concept (see $3.
1. Accuracy(cgl) (z)) Up) Accuracy(Cp)) (pp (=)): Up); when adversarial probing classifier cp(-) is trained using any learning objective like max- max-in or cross-entropy loss. Thus, the undesired encoder hi (z) is indistinguishable from desired encoder hī(z)) (z) in terms of adversarial task prediction accuracy but better for main-task in terms of max-margin objective.
2. Take (cm (2)) (p(((())))) < Thae (cm (hz(2))) Cp (n)(())))) when Assm 3.4 is satisfied and concept-causee causal features x of any margin point zM of Cm (h)(2)) are more predictive of the main task label than zP of any margin point xP of cz() is predictive for the probing label (Assm D.I.. Thus, undesired encoder hā (2) is preferable over desired encoder (z) for both main and combined adversarial objective. Here Ln2 L(cm(.)) L(cp(-) is the combined adversarial loss w.r.l. to b2 and L(c(-)) is the mux-margin loss for classifier "c" (see $D.)).
Proof Sketch. (1) The proof is by construction. Using Lemma B.I we show that there exists hy s.t. Lccm (h₂)) [(cm (hz)), and that accuracy of the probing classifeer remains the same when using either encoder. (2) Compared to hī, we show that the improvement in main task loss when using &p features is larger than the improvement in the probing loss for hgg thus preferred by overall objective.
3.4 Implications for real-world data: A metric for quantifying degree of spuriousness
Our theoretical analysis shows that probing-base removal methods fail to make the main task classifier invariant to unwanted concepts. However, to verify whether the final classifier is using the concept or not, the theorem statements require knowledge of the concept's s features 2p. For practical usage, we propose a metric that quantifies the degree failure or spuriousness for both the main and probing classifier. For simplicity, we define it assuming that both main and concept labels are binary.
Let Dm.p be the dataset where for every input x' we have both the main task label ym and the concept label yy. We define 2 x 2 groups, one for each combination of (ym, 8p). Without loss of generality, assume that the main-task label yn. = 1 spuriously correlated with concept label yp = and similarly Vm = 0 is correlated with Up = 0. Thus, (vm = 1, Up = 1) and (Um. O, Up 0) are the majority group Smaj while groups (ym = 1, Up = 0) and (ym = O, Up = 1) make up the minority group Smin- We expect the main classifier to exploit this correlation and hence perform badly on Smin where the correlation breaks. Following (39]. we posit that minority group accuracy ie Acc(S (mmi)) can be good metric to evaluate the degree of spuriousness. We bound the metric by comparing it with the accuracy on Smin of "clean" classifier that does not use the concept features.
Definition 3.1 (Spuriousness Score). Given dataset, D = Smin U Smaj with binary task label and binary concept, let Acc (Smin) be the minority group accuracy of a given main task classifier (f) and Acc (Smin) be the minority group accuracy of clean main task classifier that does not use the spurious concept. Then spuriousness score of is: (f) |1 Acc (Sman)/ Acc" (Smin)).

---

To estimate Acc" (Smin), we subsample the dataset such that Yp takes a single value in the sample and train the main classifier on as [35]. Here the probing label Yp no longer is correlated with the main task label ym- The spuriousness score of a probing classifier can be defined analogously to 1ef by swapping the task and concept label (see Def F. For creating clean probing classifier, we subsample the dataset such that Vm takes a single value and train the probing classifier.
4 Experimental Results
Theorems and 3.3 show the failure of concept removal methods under a simplified setup and max-margin loss. But current deep-learning models are not trained using max-margin objective and might not satisfy the required assumptions (Assm 3..33.23.33..4 Thus, we now verify the failure modes on three real-world datasets and one synthetic dataset, without making any restrictive assumptions. We use RoBERTa [25] as default encoder and fine-tune over each real-world dataset. For Synthetic-Texx dataset we use the sum of pre-trained Glo embeddings [30] of words in
sentence as the default encoder. For details on the experimental setup, refer
4.1 Datasets: Main task and spurious/sensitive concept
Real-world data. We use three datasets: MultiNL (4).. PAN16 [1] and Twitter-AAE [ In MultiNLI, given two sentences premise and hypothesis the main task is to predict whether hypothesis entails, contradicts or is neutral with respect to premise. We simplify to a binary task of predicting whether hypothesis contradicts the premise or not. Since negation words like nobody.no,neve never and nothing have been reported to be spuriously correlated with the contradiction label [16], we create 'negation" concept denoting the presence of these words. The goal is to remove the negation concept from an NI.I model's representation space. In Twitter-PAN16, the main task is to detect whether a tweet mentions another user not, as in [22 The dataset contains gender label for each tweet, which we consider as the sensitive concept to be removed from the main model's representation. In Twitter-AAE, again following (12). the main-task is predict binary sentiment labels from a tweet's 's text. The tweets are associated with race of the author, the sensitive concept to be removed from the main model's representation.
Synthetic-Text. To understand the reasons for failure, we introduce a Synthetic- Text dataset where it possible to change the input text based on a change in concept (thus implementing Def. 2.1). Here we can directly evaluate whether the concept is being used by the main-task classifier by intervening on the concept (adding or removing) and observing the change in model's prediction. The main-task is to predict whether a sentence contains a numbered word (e.g., one, fifteen, etc.). We introduce a spurious concept (Iength) by increasing the length of sentences that contain numbered words.
Predictive correlation. To assess robustness of removal methods, we create multiple datasets with different predictive correlation between the two labels ym and 7/p The predictive correlation (x) is practical measure for the spurious correlation defined in Assm 3.3 that does not require access to 2m features. It measures how informative one label is for predicting the other, x = Pr(ym Up >
0) EM 1[ym Up >0] where N is the size of dataset and 1[:] is the indicator function that is if the argument is true otherwise O. Predictive-corrlatioo lies in k E 00.,, 1] where x = 0.5 indicates indicates that the attributes fully correlated. For more details on the datasets and how we vary the predictive-correlation, refer to and for additional results see $
no correlation and
Measuring spuriousness of classifier. We use the Spuriousness Score (.1) measure the degree of reliance of the main task classifier on the spurious concept, and vice-versa for the probing classifier. In addition, for the Synthetic-Texx Text dataset, we use metric AProb that exactly implements 2. for estimating a model's 's reliance on the spurious concept. Since we can modify the concept directly in input space for Synthetic-Ters AProb changes the parts of a input sentence corresponding to spurious concept and measures the change in the main task classifier's s prediction probability. As sanity check, on the Synthetic-Texx dataset, A Prob and Spuriousness Score highly correlated (Pearson correlation=0.83, ( For all real-world datasets, we use the Spuriousness Score.
4.2 Results: Null space removal
In general, for any model given as input to INLP, may be difficult to verify whether INLP removed the correct features. Hence, we construct a benchmark where the input classifier is clean, i.e., it does not use the concept at all. We do so by training on a subset of data with one particular value of spurious concept label, as in [35]. Since the input classifier does not use the concept-causal features,
7

---

Delaset: Synthuti-Ten Text
Detaset: Synthhetic Text
Dataset: Syntheis-Text
2.0
0.4
8..
0.2
(a) Main-Task (cl(All)
(b) Main-Task A Prob(All)
(c) Probe Spuriousness Score
0.9
0.0
0.7-
0.7
0.4
0.40
0.5
2.3
0.55
0.0
3 Initraiing 1.6. BLL Remevcll 3.26
2
2
Inttiii 1-6 MAL Namoval 7-26
Pretraiing 1-6 I Aamonai 7.66
(d) Main- Dask Acc(All)
(c) -asi Acc(All)
(1) Main-Task Acc(All)
Figure 3: Null space removal failure. Top row corresponds to the Synthetic-Tese dataset and bottom row shows the failure on three woal-world datasets. In each figure, the x- x-axis shows the INLP iteration and y-axis shows different evaluation metrics. Colored lines correspond to the different levels of predictive correlation (x) in the datasets used by INLP. (a), (d), (e), (f) show that as INLP removal progresses, main-task classifier is getting corrupted which leads drop in its accuracy (see $4.2
we expect that INLP should not have any effect on the main task classifier. Note that we keep the main task classifier frozen in all the experiments described below. For the setting where the main task classifier is retrained after every projection step of INLP, refer and Fig. 9
Eventually all fask-relevant features are destroyed. We start with the Synthetic- Text dataset by training a clean classifier on the main-task and inputting it to INLP for removing the spurious concept. To keep the conditions favorable for INLP, both the main task and concept-probing task can achieve 100% accuracy using their causally derived features respectively. In Fig. 3a. colored lines show datasets with different levels of predictive correlation that are provided to INLP and iterations 21-40 show individual steps of null-space removal. Since, the given pre-trained classifier was clean, i.e., not using the concept features, null-space removal shouldn't have any effect on it. We observe that for all values of K, the main-task classifier's accuracy eventually goes to 50% random guess accuracy implying that the main-task related attribute has been removed by INLP, as predicted by Theorem 3.2 Higher the value of correlation k, faster the removal of main-task attribute happens. We obtain a similar pattern over real-world datasets. Fig. 3d and 3f show decrease in the main-task accuracy even when the input classifier for each dataset is ensured to be clean: except for k = 0.5 (no correlation), all values of yield random- guness classifier after applying INLP on MultiNL while they yield classifiers with less than 60% accuracy for Twitter-PAN16 and Twitter-AAE
Early stopping increases the reliance spurious features. To avoid full collapse of the main-task features, a stopping criterion in INLP is to stop when the main-task classifier's s performance drops (32]. In Fig. Bbb 3blwe we measure spuriousness, sensitivity of the Synthetic-Texx main task classifier w.r.t. the spurious concept, using APob (see I and E.S) At lower iterations of INLP, AProb is higher than that of the input classifier. For example, for 0.8, when the main-task classifier's 's performance drops at iteration 27, the classifier has a high AProb se 25%, higher than the input classifier (Fig. (b)). Hence is possible that stopping prematurely will lead us to a classifier that is more reliant on the spurious concept than it was before, consistent with the statement 1(a) in Theorem 3.2 The reason is the mixing of the main task and concept-causal features in each iteration, as shown Fig, 3c using the spuriousness score of the probing classifier (Def At lower iterations, the spuriousness score of probing classifier increases to a very high value (close to max value 1), for all values of k.
Failure of causally-inspired probing. Amnesic Probing [33 declares that sensitive concept is being used by the model if, after removal of the concept from from the latent representation using INI.,, there is a drop in the main-task performance. But Fig. 3a 3d 3el and 31 show that even when the input classifier for its corresponding main task is clean, I.c., does not use the sensitive
8

---

1.0
Dataset: MuINN
1.0
Dataset: MuKNU
Dataset MatNN
0.8
nehrdd ERM
0.8
0.5
0.4
0.4
0.2
0.2
0.2
masted-Da CHN
0.0
0.5
0.5
0.7
2-5
0.9
1.0
0.0
0.5
0.6
0.7
0.8
0.9
Training
(a) Main Spuriousness Score
(b) Adv-Probe Acc(All)
(c) AR- Clean to Spurious
Syntheli--ee
me meliod-1MM
0.2
0.8
0.6
0.4
0.6
0.4
0.4
0.4
0.2
0.2
0.0
cimddlaeccmmmsnco on
ct 140
predetive comdistin 1o.
(d) Main Spuriousness Score
(c) Main Spuriousness Score
(1) Main Spuriousness Score
Figure 4: Adversarial Removal Failure. Top row explains failure of the AR method on MulttNLI. Bottom row shows the failure on Twitter-PAN16, Twitter-AE, and Synthetic- Text datasets. In each figure, the x-axis shows different levels of predictive-correlaiion (x) between the main task and concept labels in the dataset used by AR and the y-axis shows different evaluation metries. Orange lines denote the ERM model and the blue lines denote the model trained using AR. (b), (d), (e), (f) show that AR is unable to completely remove the spurious concept from the main task classifier. (c) shows a stronger failure where the AR method introduces spuriousness into a clean input classifier.
concept, INLP leads to drop in performance of the main-classifier. Hence, removal-base methods like Amnesic probing will falsely conclude that the sensitive concept is being used.
4.3 Results: Adversarial removal
We now demonstrate failure of the deveesarral-remowwe method (AR) in removing the spurious concept from the main classifier. We train a separate main-task classifier without any adversarial objective with standard cross-entropy loss objective (referred as ERM). Then we compare standard ERM training of the main classifier with the AR method over the same number of epochs (20). We follow the training procedure of [12] and conduct a hyper- nyper-parameter sweep on the adversarial training strength to select the value which is most effective in removing the concept. For details, refer to $ Cannot remove the spurious concept fully. For MultiNLI, Fig. 4a shows the spuriousness score 3.I of ERM and AR classifiers as vary the predictive correlation (x) between the main-task label and sensitive concept label in the training dataset. While the spuriousness score for classifier trained using AR (blue curve) is lower than that of ERM for all values of K, it is substantially away from zero. Thus, the AR method fails to completely remove the spurious concept completely from the latent representation. By inspecting the concept probing classifier accuracy for ERM and AR in Fig. 4b we obtain a possible explanation. The probe accuracy after adversarial training doesn't ' decrease to 50% but stops at accuracy proportional to the predictive correlation k. This is expected since even if the AR would have been successful in removing the concept-causa features, the main- task features would still be predictive of the concept label by λ due to the spurious correlation between them. However, the converse is not true: an accuracy of does not imply that the concept is fully removed. The results substantiate the first statement of Theorem 3.3 given two representations where one (desired) does not have concept features while the other (undesired) contains the concept features, the undesired one may be better for the main task accuracy even as both may have the same probing accuracy. Fig, 4d 4e and show the spuriousness score of AR in comparison to classifier trained with ERM on Twitter- PANT6, Twitter-AAB and Synthetic- Text datasets respectively. The failure of AR is worse here: there is no significant reduction in spuriousness score for AR in comparison to ERM. For the Synthetic- Text dataset, ERM has zero spuriousness score but AR has non-zero score. We expand more this observation and include additional results adversarial removal in SF.3
AR makes a clean classifier use the spurious concept. In Fig- 4c we provide clean main task classifier (see $ F.3 for training details) as input to AR method. For all values of k, the input classifier's spuriousness score is low (iteration 1-6). From iteration 7 onwards, the AR method corrupts the clean classifier as shown by increasing spuriousness scores. For more results, see Fig. 12b in F3

---

Comparison with previous work. If post-removal the latent representation used by the main-task classifier is still predictive of the removed concept, [12] claimed it as a failure of the adversarial removal method. However, this claim may not be correct since a feature could be present in the latent space and yet not used by model [5
Our proposed spuriousness score metric avoids this limitation.
Ablations. In Appendix, we report results on using BERT instead of RoBERTa as the input encoder F22
F.3 the effect of using different modeling choices like loss-function, regularization, e.t.c. and the behavior of probing classifiers when concept is not present in latent space ($ ($F.....
5 Related work
Concept removal methods. When the removal of a concept can be simulated in input space (e.g.. in tabular data simpler concepts), removing a concept directly using data augmentation [ or gradient regularization [88 [22 can work. However, concept removal is non-trivial when change in concept cannot be propagated via change in input tokens. Combining the ideas of null space and adversarial removal [32 24
restrict the adversarial function to be a projection operation and derive a closed form solution. Other approaches use explanations of the classifier's prediction for concept removal [77. Our work highlights the difficulty of building an estimator for the features causally derived from concept, as general limitation for concept removal.
48 12] methods like [3 34
Limitations of a probing classifier for model interpretability. We also contribute to the growing literature on the limitations of a probing classifier's 's accuracy in capturing whether the main classifier is using a concept [4). It is known that probing classifiers capture not just the concept but any other As a result, many improvements have been proposed to better estimate whether a concept is being used, including the use of control labels or datasets 919 35 Parallelly, new causality- causality-inspired probing methods
features that may be correlated with it (39 2 23 45
[33 compare the main task accuracy on representation without the concept constructed using the null space removal method. The hope is that such improvements can make probing more robust. Our results question this direction. To demonstrate the fundamental unreliability of probing classifier, we construct setup that is most favorable for learning only the concept's ' features and still find that learned probing classifier includes non-zero weight for other features, limiting effectiveness of any interpretation method based on it.
6 Conclusion
Our theoretical and empirical evaluations show that it is difficult to create probing- explain- ability and removal method due to the fundamental limitation of learning a "clean" probing classifier. We recommend two tests for validating removal methods. First, we provide a sanity-check any reasonable removal method should not modify "clean" classifier that does not use any spurious features to produce a final classifier that uses those features. Second, we propose a spuriousness score that can be used to evaluate the dependence of any classifier on spurious features. As future step, we encourage the community to develop more such sanity-check tests to evaluate proposed methods.
Alternatively, we point attention to other approaches that may provide better guarantees for concept re- moval. An example is extending data augmentation techniques like counterfactual data augmentation (23) 9 to non-trivial concepts. For a given training point that may include a spurious correlation, a new data point is generated that breaks the correlation but keeps the semanties of the rest of the text identical (hence the name, "counterfactuall) This can be done by human labeling or handcrafted rules for modifying text (e.g., Checklists B7]. Then the main classifier regularized to have the same representation for such pairs of inputs (3 (6)). By construction, with good quality pairs, such a method will not remove relevantt features and will satisfy the sanity checks listed above. That said, a limitation is that the removal quality will depend on the diversity of the counterfactual examples generated and whether they capture all aspects of the spurious concept. Another direction is to take inspiration from the algorithmic fairness literature [18] 28) and focus on the predictions of the classifier rather than the representation. Compared to removal in latent space, enforcing certain fairness properties on model predictions is a more well-formed task, more interpretable, and definitely more relevant if the final goal is fair decision making.
Limitations. A limitation of our theoretical work is assuming frozen or non non-trainable latent representation which makes the analysis of task-classifier trained on top of them relatively easier. We address this limitation in our empirical work where we do not make such assumptions. Also, our work addresses failure modes of two popular methods, null space removal and adversarial removal. We conjecture that any other method based on probing classifiers will lead to similar failure modes.
10

---

References
[1] Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained anal- ysis of sentence embeddings using auxiliary prediction tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[2] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization, 2020.
[3] Ananth Balashankar, Xuezhi Wang. Ben Packer, Nithum Thain, Ed Chi, and Alex Beutel. Can we improve model robustness through secondary attribute counterfactuals? In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 4701-4712, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[4] Yonatan Belinkov. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguisties, 48(1):207-2119 04 2022.
[5] Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006.
[6] Su Lin Blodgett, Lisa Green, and Brendan O'Connor. Demographic Dialectal Variation in Social Media: Case Study of African-Amerien English. In Proceedings of EMNLP, 2016.
[7] Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man to computer programmer woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 4356-4364, Red Hook, NY, USA, 2016. Curran Associates Inc
[8] Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single $ vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia, July 201 8. Association for Computational Linguistics.
[9] Saloni Dash, Vineeth N Balasubramanian, and Amit Sharma. Evaluating and mitigating bias in image classifiers: A causal perspective using counterfactuals. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 915-924, 2022.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[11] P. Diehl, H. Kellerhals, and E. Lustig. Diagonalization of symmetric matrices. In Computer Assistance in the Analysis of High-Resolutin NMR Spectra, pages 73-77, Berlin, Heidelberg, 1972. Springer Berlin Heidelberg.
[12] Yanai Elazar and Yoav Goldberg, Adversarial removal of demographic attributes from text data, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11-21, Brussels, Belgium, Octoher-November 2018. Association for Computational Linguistics.
[13] Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Trans. Assoc. Comput. Linguistics, 9: 9:160-175, 2021.
[14] Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Understanding undesirable word embedding associations. In Anna Korhonen, David R. Traum, and Lluís Marquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI.2019 2019, Florence, Italy July 28- August 2019, Volume I: Long Papers, pages 1696-1705, A Association for Computational Linguistics, 2019.
11

---

[15] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, ICML'11,, 15, page 1180-1189. JMLR.org, 2015.
[16] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Pupers), pages 107-112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[17] Xiaochuang Han and Yulia Tsvetkov. Influence tuning:
Demoting spurious correlations via in- stance attribution and instance-driven updates. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 4398-4409 Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[18] Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[19] John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743 Hong Kong. China, November 2019. Association for Computational Linguistics.
[20] Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, and Yoav Goldberg. Contrastive explanations for model interpretability. In EMNLP (1), pages 1597-1611, 2021.
[21] Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirly-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 1772-1798. PMLR, 25-28 Jun 2019.
[22] Sai Srinivas Kancheti. Abbavaram Gowtham Reddy, Vineeth N Balasubramanian, and Amit Sharma. Matching learned causal effects of neural networks with domain priors. In International Conference Machine Learning, pages 10676-10696. PMLR, 2022.
[23] Divyansh Kaushik, Amrith Setlur, Eduard Hovy, and Zachary C Lipton. Explaining the efficacy of counterfactually augmented data. International Conference on Learning Representations (ICLR), 2021.
[24] Paul Pu Liang, Chiyu Wu, Louis- Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models. In ICML, pages 2021.
[25] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692, 2019.
[26] Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, pages 7313-7324. PMLR, 2021.
[27] Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Asso- ciation,fr for Computational Linguistics, pages 3428-3448, Florence, Italy, July 2019. Association for Computational Linguistics.
[28] Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. survey on bias and fairness in machine learning. ACM Comput. Surv., 54(6), jul 2021.
[29] Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the fail- ure modes of -it--ii--dttiitiionn generalization. In International Conference on Learning Representations, 2021.
12

---

[30] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, 2014.
[31] Francisco Rangel, Paolo Rosso, Ben Verhoeven, Walter Daelemans, Martin Potthast, and Benno http:///d.iio/11.0///./. /tt8////ooo.34599090000 In CLEF 2016 Labs
Stein. Panl6 author profiling, and Workshops, Notebook Papers. Zenodo, September 2016.
[32] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online, July 2020, Association for Computational Linguistics.
[33] Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18400-18421. PMLR, 17-23 Jul 2022.
[34] Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kemnel space. CoRR, abs/2201.121999 2022.
[35] Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 3363 3377, Online, April 2021. Association for Computational Linguistics.
[36] Radim Rehurek and Petr Sojka. Gensim-pythoo framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic, 3(2), 2011.
[37] Marco Tulio Ribeiro. Tongshuang Wu, Carlos Guestrin, and Sameer Singh, Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, Online, July 2020. Association for Computational Linguistics.
[38] Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. Right for the right reasons: Training differentiable models by constraining their explanations. Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2662-2670, 2017.
[39] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICIR), 2020.
[40] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizaation exacerbates spurious correlations. Proceedings of the 37th International Conference on Machine Learning, (CML'20. JMLR.org, 2020.
[41] Indira Sen, Mattia Samory, Claudia Wagner, and Isabelle Augenstein. Counterfactually aug- mented data and unintended bias: The case of sexism and hate speech detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4716 4726. Seattle, United States, July 2022. Association for Computational Linguistics.
[42] Xing Shi, Inkit Padhi, and Kevin Knight. Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526 1534, Austin, Texas, November 2016. Association for Computational Linguistics.
[43] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. J. Mach. Learn. Res., 19(1):2822-2877 jan 2018.
13

---

[44] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[45] Victor Veitch, Alexander D' Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 64, 2021, virtual, pages 16196-16208, 2021.
[46] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-con -overaee challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics, 2018.
[47] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface' 's transformers: State-of-the-an natural language processing, 2019.
[48] Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. In Proceedings of the st International Conference on Neural Information Processing Systems, NIPS' 17, page 585-596, Red Hook, NY. USA, 2017. Curran Associates Inc.
[49] Ke Xu, Tongyi Cao, Swair Shah, Crystal Maung, and Haim Schweitzer, Cleaning the null space: privacy mechanism for predictors. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI' 17, page 2789-2795. AAAI Press, 2017.
[50] Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data aug- mentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 1651-1661, Florence, Italy, July 2019, Association for Computational Linguistics.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See and
(b) Did you describe the limitations of your work? [Yes] See $6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Our work show failure modes of existing concept removal methods.
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] See $
(b) Did you include complete proofs of all theoretical results? [Yes] See B and
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See $
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See $E)
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)? [Yes] See $ $E and $F
(d) Did you include the total amount compute and the type of resources used (e.g., type
of GPUs, internal cluster, cloud provider)? [Yes] See $E.9
14

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See $ and S
(b) Did you mention the license of the assets? [Yes] See $
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "_fjh4h1NNb1", "reviewer": "Reviewer_PTq2", "summary": "The authors propose that probing-based concept removal and detection methods are unreliable given that the representation space may be corrupted, or the resulting model may still reply to those seemingly removed concepts. The authors try to ground their hypothesis on theoretical based on toy examples in 2D space, and then provide pieces of evidence using a set of naturalistic and synthetic benchmarks where the correlations between the concept and the sequence label are controllable. Although the topic is interesting and the research question is worth pursuing further, I found a set of opening questions that remain unresolved in the paper.", "questions": "- L146, says ``w.r.t. task label y_p (Assumption 3.3)``, should it be the concept label instead of the task label?\n\n- Fig 1. (a) is confusing, if not wrong. For instance, for the Gender Probing Classifier (annotated as the blue dashed line), with accuracy = 100%, it is clearly mislabeling multiple examples.\n\n- I would suggest moving at least the essence of the *Proof Sketch* to the main text as it is essential to see formal proofs to back up the authors' main hypotheses. Otherwise, it feels ungrounded.\n\n- In Sec. 4.2, the authors train the model using a special subset of training data where all examples are ``with the same value of the property`` (L305), could you clarify a bit what is the property? and how to have the same value?\n\n- Can authors define ERM before using it as a term in L338?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- This paper aims to address a pressing issue in building debiasing methods of neural networks.\n- The toy examples provided in the paper are intriguing.\n\n**Weaknesses**:\n- **Is the failure mode of INLP really a valid failure mode?**\n\nThe main argument for disqualifying INLP from being a sound method is that INLP will corrupt the representation even though we know there are no concept-level relations with respect to the final label in the task. In Sec. 4.2, the authors train the model using a special subset of training data where all examples are ``with the same value of the property`` (L305). I assume the authors only allow the model to see training examples with the same concept label y_{p}. With this, the authors claim that INLP should have a null effect in removing information in the representation space. I find this argument unconvincing, if not wrong. First of all, INLP in this case will first train a probe classifier with simply 100% accuracy, since it only needs to predict the same label for all examples. For the iterative project process, it needs to remove information to decrease the accuracy from 100% to 0% (if possible). It is obvious that INLP will definitely remove sequence-related information from the embedding no matter what, which eventually will just randomly corrupt the representations. As a result, the results in Fig. 2 are indeed expected. These results are not because INLP is effective, the experiment setups naturally lead to these results.\n\n- **Is the failure mode of the adversarial removal methods really a valid failure mode?** \n\nAll experiments conducted in Sec. 4.3 relies on the predictive correlation (κ). Let us say we have κ=1.0. As we are training a probing classifier to have an adversarial loss to remove concept-related information, there is no way to guarantee this will happen, especially in this case where the main-task label is 100% correlated with the concept label. The probing classifier may just be another main-task classifier that is trained to lower the main-task accuracy. In this case, it is not actually a failure mode about the method again, it is a tricky design of the dataset that fools the method. In fact, when the main-task label is 100% correlated with the concept label, there is no way we can ensure we are removing concept-related information if only with classification-driven losses.\n\nI found the topic interesting, but my main concern is those two real-world study cases are not really exposing weaknesses of those concept removal methods, and are rather just some corrupted datasets that will trick any method.\n\n[Update]: Thanks for the clarification on the INLP, as the description was a little confusing to me. Maybe considering updating the text. In the case study of the adversarial removal methods, I am still not convinced yet that building spurious correlations within the dataset is a good way to say the failure mode of these methods. I am raising my score to 5 as I am leaning towards acceptance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "YDmV_pdCoy8", "reviewer": "Reviewer_PP9U", "summary": "This paper discusses the problem of spurious correlations in natural language processing (NLP). The authors discuss the well-known issue that machine learning-based NLP models tend to learn representations that encode spurious correlations between input feature attributes (the authors refer to these as “concepts”) and class labels, thereby causing models to fail to generalize to out-of-distribution data. Specifically, this paper investigates the limitations of two classes of methods proposed to remove spurious correlations, namely null space projection and adversarial training. The authors provide theoretical and empirical evidence that the aforementioned methods fail to alleviate the problem of spurious correlations in NLP by showing that they cannot successfully remove undesirable concepts. For the latter, empirical experiments are conducted on a synthetic as well as three real-world datasets: MultiNLI, Twitter-PAN16 and Twitter-AAE.", "questions": "* For the “Ablations” paragraph in Section 4: would it be possible to briefly elaborate on the results with other classifiers and model design choices in the manuscript (not only in the Appendix)? I appreciate that the experiments have been conducted under various model types and configurations, yet it would be beneficial to the reader if the main findings for this could be mentioned in the manuscript.", "limitations": "The authors do not explicitly mention the limitations of their work in the paper (a discussion beyond the results is missing) and the paper would benefit from a few sentences on that topic as well as future work based on the main findings.", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper addresses the highly relevant problem of spurious correlations in NLP and shows that existing methods to tackle that problem are limited at doing so. The paper is well-written, the theoretical analysis and the experimental efforts for empirical validation are well-documented and easy to follow, and the obtained results for the latter are convincing. Furthermore, the paper’s main objectives and findings are well differentiated from the existing literature (as stated in Section 5). \n\nWeaknesses:\nMy main concern with this work is a lack of discussion on potential ways forward. The paper successfully shows the limitations of existing methods for the removal of spurious correlations. However, the authors do not extensively discuss the implications of their findings and show little effort in discussing potential ways forward and/or alternative approaches to improve existing methods. The manuscript contains various spelling mistakes and missing words (e.g., lines 182, 190, 201, 280, 342), which I encourage the authors to correct for better readability. Finally, some crucial information (e.g., around the predictive correlation; also see my question below) are moved to the Appendix. This is understandable due to limited space but can disrupt the flow for the reader. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "82EpzgD_ZFw", "reviewer": "Reviewer_WqdQ", "summary": "This paper presents a theoretical analysis of the methods of subspace removal using probing classifiers. The main argument is that even the features are perfectly separable, the probing classifier will learn something about the task-related features, in addition to the concept related features. Then, based on this argument, this paper claims that the existing probing-classifier-based methods are not sufficient to remove concept-related feature/information. ", "questions": "No further question", "limitations": "No limitation", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nOverall, I think this paper is solid in both theoretical analysis and empirical study. \n\nFirst of all, with some mild assumptions, this work presents a theoretical analysis of why probing-classifier-based method is not reliable in removing concept-related information. To some extent, the assumption is too ideal; for example, it assumes disentangled representation, which I am not sure it is true for most cases. However, this ideal case demonstrates the concept. \n\nIn addition to the theoretical analysis, it also presents some empirical studies, which further strengthens this work. Compared to the existing works that only demonstrate the algorithm with empirical results, I like the combination of theory and empirical results. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "-8osGNyNws", "reviewer": "Reviewer_og88", "summary": "This paper analyzes the potential issues with previously proposed methods for sensitive concept removal from text representations. The authors mainly study two branches of methods, null-space removal and adversarial removal, which both use probing classifiers operated on frozen representations. The authors present theoretical analyses to demonstrate that both types of methods cannot remove the sensitive attributes completely and may distort main-task features that hurt the target task performance. The empirical studies are conducted on a set of synthetic and real datasets to support the claims.", "questions": "I would like to see some insights/ideas derived from the analyses regarding how the previous methods may be improved. Usually, good theoretical analyses not only reveal the challenges/limitations, but also shed lights on the potential directions to overcome them.", "limitations": "I hope the authors would discuss how the assumptions made could be relaxed (e.g., the spurious features may not be assumed to be removed entirely, but to some extent).", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n* The problem studied is interesting and important -- how to effectively remove sensitive features from text representations is of great significance in mitigating ethical concerns about machine learning models.\n* The paper presents both theoretical and empirical analyses to support the claims.\n\nCons:\n* The major contribution of the paper does not appear to be that significant to me. While it's important to know the potential issues with previous concept removal methods, the main arguments in this paper (that both null-space removal and adversarial removal methods cannot entirely remove the sensitive features) are more or less already mentioned in the original papers proposing those methods. In this regard, the paper is reiterating some previous observations by backing them up with more theoretical analyses (however, these theoretical analyses appear to be based on very strong assumptions, see below).\n* Some assumptions made in the paper are too strong to be meaningful in practice. The authors assume the sensitive features need to be removed entirely from the representations (i.e., a perfect separation between main-task features and spurious ones). However, the concept removal methods are usually not expected to be producing a perfectly clean representation in the first place; they are meant to mitigate the influence of the spurious features. Those methods should be considered useful as long as they can (to some extent) remove the sensitive concepts. Therefore, the \"perfect removal\" assumption appears too strong--I doubt whether there even exists any method that could satisfy such an assumption. If the assumptions are made in a way that the requirements are essentially impossible to satisfy, the subsequent theoretical analyses that show how previous methods may not work will appear much less meaningful.\n* Although the authors claim the spuriousness score as one of the contributions, there are no further evaluations on how accurately this metric corresponds to its expected purpose. \n\n**Post-Rebuttal**: The authors have addressed my concerns raised above, so I am increasing my rating.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept