When and Why Does Bias Mitigation Work?
Authors: Abhilasha Ravichander
Joe Stacey
Marek Rei
Venue: EMNLP 2023 Findings (2023)
Keywords: debiasing
lexical biases
natural language understanding

Abstract: Neural models have been shown to exploit shallow surface features to perform language understanding tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire. Previous work has developed debiasing techniques to pressure models away from spurious features or artifacts in datasets, with the goal of having models instead learn useful, task-relevant representations. However, what do models actually learn as a result of such debiasing procedures? In this work, we evaluate three model debiasing strategies, and through a set of carefully designed tests we show how debiasing can actually increase the model's reliance on hidden biases, instead of learning robust features that help it solve a task. Further, we demonstrate how even debiasing models against all shallow features in a dataset may still not help models address NLP tasks. As a result, we suggest that debiasing existing models may not be sufficient for many language understanding tasks, and future work should consider new learning paradigms, to address complex challenges such as commonsense reasoning and inference.

When and Why Does Bias Mitigation Work?
Joe Stacey2* Marek Rei Allen Institute for Artificial Intelligence Imperial College London
Abhilasha Ravichander
.org {j. staceyy, marek. rei) } @ @imperial.ar aac.uu.uu uk
Abstract
shortcuts presents crucial obstacle to overcome. number of debiasing approaches were proposed to prevent models from learning these shortcuts, in- cluding product-of-expere (Mahabadi et al., 2020), example reweighting (Clark et al., 2019) and confi- dence regularization (Utama et al., 2020a). These approaches are based on the intuition that by forc- ing models to avoid particular dataset shortcuts, we can achieve robust models that learn task-relevant features instead.
Neural models have been shown to exploit shal- low surface features to perform language under- standing tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire. Previous work has de- veloped debiasing techniques pressure mod- els away from 'spurious' features or artifacts datasets, with the goal having models in- stead learn useful, task-relevant representations. However, what do models actually learn as a re- sult of such debiasing procedures? In this work. we evaluate three model debiasing strategies, and through a set of carefully designed tests we show how debiasing can actually increase the model's s reliance on hidden biases, instead learning robust features that help it solve task. Furthermore, we demonstrate how even debiasing models against all shallow features a dataset may still not help models address a task. As result, we suggest that only debi- asing existing models may not be sufficient for many language understanding tasks, and future work should consider new learning paradigms to address complex challenges such as com- monsense reasoning.
In this work, critically examine this assump- tion. We take step back and characterize the empirical behavior of debiasing methods at large. While prior work has largely demonstrated the ef- fectiveness of debiasing methods by showing in- creased "robustness" on external challenge sets, where these improvements come from, the extent to which the model is unbiased, and the exact differ- ences in behavior induced by debiasing procedures remain largely unknown.
Several factors make analyzing model behavior along these axes difficult in empirical settings. It is challenging to: (1) enumerate all possible biases in dataset, (2) characterize task difficulty, and (3) develop fine-grained insights into the mechanisms models use to perform tasks before and after debiasing procedures. We shed light on these ques- tions, by constructing a suite of controlled tests to characterize the behavior of debiased models. We make our suite of synthetic tasks publicly available, providing a testbed for bias mitigation techniques.
1 Introduction
Large-scale language models have established state- of-the-art performance on several language under- standing tasks (Devlin et al., 2019; Liu et al., 2019; He et al., 2020). However, the performance of these models can degrade when presented with amples drawn from a different distribution than their training conditions. Past work has shown that these approaches are prone to exploiting spurious "shortcuts' in the training data, instead of learning the intended task (Naik et al., 2018; McCoy et al., 2019; Geirhos et al., 2020).
Through the behaviors of debiased models on our controlled tasks, we identify several undesir- able behaviors induced by debiasing procedures. First, we identify that debiasing can cause models to switch from the identified and targeted dataset bias an unidentified dataset bias, while still seem- ing to demonstrate increased "robustness" the targeted bias. This suggests that entirely pos- sible that current bias mitigation approaches cause models to shift from relying on measurable biases
Towards the primary goal of training NLP sys- tems that understand natural language, the propensity of neural models rely misleading
*Th first two authors contributed equally to this work.

---

relying on other unknown and unidentified bi- ases in datasets ($5.1). Further, we characterize the complexity of a dataset! and demonstrate that even when all dataset biases have been addressed by a bias mitigation procedure, the model may not be able to learn the task if it is sufficiently complex ($5.2). This suggests that debiasing approaches alone are unlikely to help address complex tasks such those involving commonsense reasoning. Our contributions are:
et al., 2018; McCoy et al., 2019). Further, on the MultiNLI dataset (Williams al., 2018), models can considerably outperform a random baseline by only looking at partial inputs (Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018). Simi- lar model behavior has been identified in several tasks which require natural language understand- ing, including argumentation mining (Niven and Kao, 2019), reading comprehension (Jia and Liang, 2017; Kaushik and Lipton, 2018), visual question answering (Zhang et al., 2016; Kafle and Kanan, 2017: Goyal et al., 2017; Agrawal et al., 2018), fact verification (Schuster et al., 2019), and story cloze completion (Schwartz et al., 2017; Cai et al., 2017).
1. We provide a suite of six controlled datasets to probe model behavior when debiasing against known bias 1$..1).
2. For a range of different debiasing techniques and tasks, we show how debiasing can shift model reliance to hidden biases in the dataset, rather than the ones practitioners are aware of 155.1).
Bias Mitigation Methods Recent work has fo- cused on 'debiasing' approaches, with the aim of learning unbiased models by pressuring them away from picking up dataset biases. These approaches can be broadly categorized as follows: (1) Data- centric approaches: procedures which manipulate the input data before running a standard model training procedure. By either filtering out biased instances, or by augmenting the dataset with addi- tional examples, the model is expected reduce its reliance on spurious biases in the dataset (Kaushik et al., 2020; Le Bras et al., 2020a: Yanaka et al., 2019; Min et al., 2020; Liu et al., 2020b; Wen et al., 2022). (2) Model-centric approaches: debiasing procedures that either modify the architecture of the model, the optimization, or the training proce- dure in order to make model reduce its reliance spurious biases (Sagawa et al., 2019; Tu et al., 2020; Mahabadi al., 2021: Zhou and Bansal, 2020; Kirichenko et al., 2022; Wang et al., 2022; Du et al., 2022; Rajič et al., 2022; Wang et al., 2023; Ghaddar et al., 2021).
3. We show for more complex tasks, debiasing alone does not result in more robust models ($5.2).
4. We highlight the strengths and weaknesses of three popular debiasing techniques, showing the circumstances when each technique may be appropriate ($5.3).
2 Related Work and Background
Shortcut Learning Deep learning models are now known to be susceptible to sshorteutleeanning where models learn decision rules that appear super- ficially successful on benchmarks, but generalize poorly outside the training distribution (Geirhos et al., 2020). For example, a model trained to pre- dict images that contain cows may use features from the background, such as the presence of grass, to make its prediction rather than learning to recog- nize cows (Beery et al., 2018). Much recent work in NLP has identified that models tend capital- ize on idiosyncrasies of particular dataset at the expense of learning an underlying task. For the nat- ural language inference task (Cooper et al., 1996; Dagan et al., 2006, 2013; Bowman et al., 2015; Williams et al., 2018), past work has found that NLI models make entailment decisions based on cues such as the degree of lexical overlap between pairs of sentences or the presence of words such as 'not' in the input (Dasgupta et al., 2018; Naik
In this work. we will focus on model-centric debiasing procedures that can be specialized to particular tasks and datasets. Many popular model- centric approaches involve using the predictions of another model to influence learning. For exam- ple, several model-centric approaches use the pre- dictions or the representations learned by a "shal- low" classifier classifier that is lower capacity or impoverished in some other way such as ac- cessing partial input-to construct a more robust model through debiasing (He et al., 2019; Clark et al., 2019; Mahabadi al., 2020; Utama et al., b020a,b; Sanh et al., 2021; Zhou and Bansal, 2020; Liu et al., 2020a; Clark et al., 2020; Xiong et al., 2021). The intuition behind these approaches is
'We consider datasets denotations for a task. *Code/data available at https:/ // / thub. com/ Abhi aasa chander/bias mit gatiion.

---

that a "shallow" model is likely to rely more on dataset-specific biases. Popular approaches here in- clude adversarial training (Belinkov et al., 2019a b; Stacey et al., 2020); product-of-experts models where a 'main' model is trained in an ensemble with a shallow model (Clark et al., 2019, 2020; Utama et al., 2020b; Sanh et al., 2021), importance reweighting approaches where models are trained on reweighted version of source dataset (He et al., 2019; Utama et al., 2020b; Liu et al., 2020b; Mahabadi al., 2020), or simply including a sec- ond round of finctuning on a more challenging subset of the data (Yaghoobzadch et al., 2021). In this work, we consider whether debiasing strategies help (or not) with building better natural language understanding systems.
sent an 'unknown" dataset bias. Our goal with these assumptions is represent realistic conditions in NLP datasets, which contain possible shorteuts for models such as shallow lexical cues and anmotation artifacts, but where the complete set of shortcuts a model can take is unknown. Our experiments focus on debiasing against wt, representing the shortcuts that models are pressured away from taking by de- biasing procedures (He et al., 2019; Clark et al., 2019; Utama et al., ,020b,a; Clark et al., 2020; Utama al., 2020b; Sanh et al., 2021).
3.2 Definitions
Let S be the space of all sequences (which are ei- ther English sentences or number sequences in our experimental setting). Let y E {0, 1} represent the classes in our binary paired-sequence classifi- cation task, intended to represent real-world paired sentence classification tasks such as NLI. The com- plete set of examples in the dataset D is then drawn from the space (I1, Z2, y) E S x S 1}.
Understanding the limitations of current debi- asing methods Amirkhani and Pilchvar (2021) show that using importance reweighting based on the prediction of a biased model may waste too much training data. Our work is similar, but exam- ines fundamental questions about what models will learn from debiasing procedures. Mendelson and Belinkov (2021) show through a probing experi- ment that debiasing against a particular bias may increase the extent which that bias is encoded in the inner representations of models. In this work, we study how debiasing procedures affect model behavior, as probe performance not necessarily indicative of the information which a model actu- ally uses to make predictive decisions (Ravichander et al., 2021; Elazar et al., 2021).
Task features and spurious features In our def- inition, a task-relevant feature ( is feature that a dataset designer would identify useful for per- forming NLP task. This motivates our choice specify task-relevant features as those which corre- spond to particular task definition. Conversely, a spurious feature w is a feature that is considered not relevant for the task. In our work, we will discuss two kinds of spurious features- wy are spurious features that are targeted by debiasing methods i.e. targeted bias, and W2 spurious features that ex- ist in the dataset but have not been debiased against i.c. hidden bias.
3 Framework and Definitions
3.1 Framework
Feature frequency and label correlation We refer to frequency as the proportion of examples in dataset containing a feature i.e p(w) (w).. By predic- tiveness, we refer to the extent to which a feature correlates with label i. p(y| ()). For the spurious features wi and W2, we set w = 1 as a spurious
Our goal is to analyze the mechanisms models em- ploy during debiasing procedures. Our motivation is the discovery of shorteut-learning behavior in models for NLP tasks such as natural language in- ference and paraphrase detection. The question this work addresses is: when a model is debiased against one bias, what features does learn?
feature that is correlated with = 0. To prevent the absence of this spurious feature from becoming an usable feature itself, add equal number
In this work without loss of generality, we as- sume a binary paired-sequence classification task where task-relevant feature t is perfectly predic- tive of the label, and 'shortcut' features wi and to2. Here, wi represents superficially correlated fea- ture that has either been identified, hypothesized to exist in a dataset, or inadvertently targeted by a proad-spectrun model, and W2 is intended to repre-
*We deviate from past definitions of task-relevan features such as that given in (Lovering et al., 2020). Lovering et al. (2020) consider task relevvel features as features where there exists some function I such that I [(((x)) = y. Le those fea- tures which are perfectly predictive of the task decision. How- ever, such an empirical definition can be misleading due to both label noise in datasets, where set of labels may not correspond to task-relevant features, as well as the possibility of multiple features that correspond to particular task label.

---

Task name
Task description
Task example
Synthetic
GREATER NUMBER
First numher of seq. >> First number of seq.
Seq. I: 95789 68077 13030 37214 36462 56347 71940
2
65880 65242 35196 Seq. 2: 11453 67462 97337 49339
64738 30296 66671 69643 72790 4413
SUM NUM-
Sum of first 5 numbers in seq. sum of first 5 numhers seq.2
Seq. 1: 39440 66925 96563 64869 90299 90034 80263
BERS
98287 95837 76400 Seq. 2: 53169 59180 3453 79426
93135 92818 97662 35652 11135 81982
FACTORIZATION
Sum of first numbers in seq. 1 or seq. 2 is divisible by 7,11, or 13
Seq. 1: 36620 96611 61508 79073 37465 21907 8158
7271 69979 39299 Seq. 2: 8631 11239 36336 74210
91520 33641 33775 26269 18699 81632
Naturalistic
LEXICAL INFERENCE
Recognize synonym-antonys pairs
Prem: What the group hate most the sickness., Hyp:
What the group hate most is the illness.
LOGICAL INFERENCE
Reasoning with logical operators (ANDIOR)
Prem: Stanley, Milo, Gabriel, Vinnie, Oakley and
are walking down the street, Hyp: Oakley or Alex, and Hudson or Elliot, and Stanley. and Eli, and Vinnie or Gabriel, Milo are walking down the street.
COMMONSENSE.
Pronoun resolution requiring commonsense inferences (Rahman and Ng, 2012)
Prem: In the story: The sniper shot the terrrist because he had orders, Hyp: In the story: The sniper shot the terrorist because sniper had orders.
INFERENCE
Table I:
Task descriptions of six controlled datasets, both synthetic and naturalistic. All datasets are binary paired- sequence classification tasks. Salient attributes of the premise and hypothesis. or sequences, are highlighted.
of examples with the condition w 2 as an in- stantiation of the same bias for 1. We refer to example bias-neutral if it contains neither of these properties i. 0.
cause pretrained language models to simply switch from relying on measurable and targeted bias to relying unknown, unmeasurable biases, (2) For higher-level complex tasks, debiasing model does not itself help models learn tasks.
3.3 Characterizing Model Bias
If a model has learnt to use a particular feature, we expect that the model predictions will be consistent with the presence of that feature. By construction, we will design datasets such that the spurious features are independent of each other, well as the task-relevant feature. Then, are interested in measuring the quantity P(y/[w) where y/ the model prediction and w represents either a spurious or task-relevant feature. We can then write this as:
4.1 Data
It can be challenging to disentangle task-relevant features from all possible spurious artifacts, thus construct datasets with tightly-controlled biases. The behavior of debiased models on each of these datasets is used to provide evidence for a particu- lar hypothesis about a model's propensities, either in favor or against. We construct both synthetic datasets as well as naturalistic datasets. Our syn- thetic datasets consists of tasks where the input is pair of number sequences, in order better control for possible confounds from pretraining. Our nat- uralistic datasets are modeled after common NLP tasks such as commonsense reasoning, and lexical entailment.
(()) ΣΣ (i, w2 t) p(w2) p(t)
We expect this empirical probability to be close to 0.5 by assumption our datasets, if model is not influenced by feature. We will say that model is biased towards particular feature, if this empirical probability p(y) w) differs considerably from 0.5.
Synthetic Data We construct simple synthetic binary paired-scquence classification tasks to study the debiasing procedures. Our tasks are intended span varying levels of complexity. All our tasks fea- ture pairs of k-length number sequences as input, and the output E {0, 1}. Our vocabulary V con- sists of {o... [V]). We set & 10 and V 10000.
4 Experimental Setup
With the above definitions in place, our hypotheses are the following: (1) Debiasing procedures can

---

Our three synthetic tasks are described in Table 1. We introduce two kinds of biases: (1) token-level bias (w1), where a particular token is correlated with a class, (2) overlap bias (w2), where the co- occurrence of tokens across both input sequences is correlated with class.
The logical reasoning dataset evaluates model's abilities to correctly resolve logical expressions given truth values of individual variables. The premise lists the people doing a particular activ- ity (truth values), while the hypothesis mentions some of the people doing the activity separated by AND/OR operators (logical expression). The wi bias is introduced by changing the activity men- tioned to either 'running' (for entailment) 'cy- cling' (for non-entailment). For the word overlap bias, a suffix is appended (specifying the activity is being performed 'once onnce again"), with only matching suffixes corresponding to the entail- ment class.
For w1, in examples with the positive label class, p(y=Iluj=1), the symbol "I' is added to one of the inputs. To balance this effect, we have bias in the examples with the negative class, p(y=0, w1=2), where the symbol '2' is added to one of the number sequences. The location of the bias and which sequence it is added to is randomly chosen, any position except the last two numbers and the first number.
For commonsense reasoning we use the DPR dataset (Rahman and Ng. 2012; White et al., 2017), where the hypothesis contains pronoun that solves to one of two subjects in premise. Resolv- ing the pronoun correctly requires making com- monsense inferences. wi and w2 biases are intro- duced by inserting a prefix of 'hn the story:'. For the w1 bias, the word 'in' is replaced by 'within' or 'inside'. For the w2 bias, the prefix is changed to 'In the story being told:' or 'In the story being narrated:'s : The same prefix for both the hypothesis and premise corresponds to the entailment class.
For W2. in examples with the positive label class, p(y=Iluz=1), the last number in both strings match. To balance this effect, we have bias in the exam- ples with the negative class, p(y=0, w2=2), where the penultimate number in both number sequences match. If the example doesn't 't contain the bias, all numbers in both sequences will be unique.
Naturalistic Data We create naturalistic datasets targeting lexical reasoning, logical reasoning or commonsense reasoning. Each dataset is created in the form of a sentence pair classification task. Ex- amples are constructed be highly lexically simi- lar in order to mitigate the presence of additional dataset biases (besides the biases we introduce). Similar to the synthetic setting, we introduce two kinds of biases: (I) token-level bias (wi), where a particular token or token sequence is correlated with class, (2) overlap bias (w2), where the co- occurrence of tokens across both input sequences is correlated with a class.
4.2 Debiasing Procedures
The three debiasing approaches applied in this work are Product of Experts, Example Reweight- ing and Confidence Regularization (He et al., 2019; Clark et al., 2019; Utama et al., 2020b,a; Clark et al., 2020; Utama et al., 2020b; Sanh et al., 2021). Following previous work (Utama et al., 2020b; Ghaddar et al., 2021; Wang et al., 2023), we use BERT (Devlin et al., 2019) as the base model for de- biasing. We assume teacher probabilities i. prob- abilities assigned by a model that relies on wi make predictions. When the biased feature does not occur (w11 = 0), this probability is set 0.5.
The lexical reasoning dataset targets synonym and antonym detection. We generate pairs using the template 'What the group hate most is the []. The entailment class is used as the label for syn- onym word-pairs and the non-entailment class is used the label for antonym word-pairs. The w1 trigger-word bias is introduced by replacing the word 'hate" with 'love' or 'like".4 The w2 bias simulates a word-overlap bias. We append prefixes about belief to both premise and hypothesis, with prefixes matching for entailment and differing for non-entailment.
Importance Reweighting This method weights training examples based on the presence of biased features within each example, down down-weighting ex- amples with stronger known biases. The loss for each minibatch B is:
LB Σ CF..Oos(0))
1
"The word love correlates with the non-entailment class (wy=2), while the word like correlates with the entailment class (w)=
Σイ=1 Pj
where Pt is the teacher probability that the ith ob- servation is the correct class, Dr are the predicted
What we think" or WWhtt we believe'

---

Bias I- token correlation bias
Bias 2- token overlap bias
Unbiased Test Set
Task
Baseline
Reweight
PoE
CR
Baseline
Reweight
PoB
CR
Baseline
Reweight
PoB
CR
Synthetic
Find Greater
0.986
0,502
0.5245
0.590
Sum Numbers
0.978
0,499
0,499
0.5904
0.606
0.904
0.627
0,5625
0.830
0.801
0.808
0.768
Factorization
0.979
0,501
0,502
0,5472
0.687
0.995
0,992
0.7604
0.499
0.499
0.500
0,499
Natural
Lexical Inference
0,902
0.486
0,495
0.418
0.750
0.999
1,000
0.825
0.676
0.671
0.701
0,666
Logical Inference
0.847
0.495
0.553
0,997
0,991
Commonsense Inference
0.924
0.474
0.477
0.683
0.727
1,000
1,000
0.652
0.502
0,511
0.506
0.498
Table 2:
Bias probabilities show the extent to which the model prediction correlates with the value of biased feature. Reweight, PoE and CR represent the three debiasing procedures in this study, and Baseline represents the model where dehiasing has been performed. Dehiasing model against bias (wi) may cause it overrely on a different bias (W2). The unbiased test set does not contain spurious features w2. All results are averaged over seeds.
model probabilities for the i-th observation, and b is the number of examples within the minibatch (Clark al., 2019; Utama et 2020b).
5 Results and Analysis
5.1 What does debiasing do?
We consider datasets for each task which contain both w1 and w2 as dataset biases. Recall that wi represents a known bias which is the target of the debiasing procedure, and represents an uniden- tified shorteut, and thus is not within the scope of the debiasing procedure. We find that debiasing the model against the wi bias consistently increases the model's reliance on the w2 bias (Table 2).
Podduct-of-Expert. The Product of Experts (PoE) approach combines the model probabilities with the teacher probabilities, incentivizing the model to learn from features other than the bias features. The loss for each example is:
C, = CCCCLLos((((logg log (:))
The extent to which the model relies on this secondary bias depends the difficulty of the task (Table 2). We observe that for both example- reweighting and PoE, the model relies almost completely on the u2 bias for the most difficult tasks in each category (FACTORIZATION and COM- MONSENSE INFERENCE). In comparison, the confidence regularization procedure incentivizes model to be uncertain on biased predictions. and observe that the most challenging tasks the model when debiased from W1, picks up neither w2- nor learns to perform the task. For the simpler tasks (FIND GREATER NUMBER, SUM NUMBERS, LEXI- CAL INFERENCE, LOGICAL INFERENCE), we find that the models rely on W0 to a lesser extent. but only seldom gains improvements on the task.
where pi are the teacher probabilities for both classes and DJ are the predicted model probabil- ities for i (Clark et al., 2019; Utama et al., 2020b; Mahabadi et al., 2020). The loss LossB for each minibatch is the average loss of each observation.
Confidence Regularization Confidence Regu- larization (CR) uses the bias teacher probabilities Pt. representing the teacher probability that the ob- servation is the correct class, in addition to model predictions z that observation is class & before any debiasing has been applied. The confidence of the model predictions before the debiasing are regularized based on the teacher probabilities pt. in- centivizing the model to be less confident for more biased examples. A robust model is then trained through self-distillation using both p; and zig, with the loss for an example is:
A reasonable concern here is that these con- clusions would not generalize to newer models, and that newer models would neither exploit the bias W1, nor switch to relying the hidden bias Wy. Due to the considerable computational cost in- volved of training several debiased models, we only experiment with DeBERTa-large (He et al., 2021). We find that our conclusions generalize when using either a BERT or DeBERTa model, these results can be found in the appendix (Fig. 2).
Zik
Ps
C, -
log yikk
P.
(1
25x)
k=0,1
2sk
The loss LB for each minibatch is the average loss from each observation.

---

Task MDI.
Bias token correlation hias
Unhiased Reweight
Test
Task
Baseline
Reweight
PoF
CR
Baseline
PoF.
CR
Synthetic
Find Greater Number
3877.35
0.892
0.5024
0.5028
0.5008
86.73%
95.51%
95.19%
79.6%
Sum Numbers
7179.58
0,9788
0,5025
0,5012
0.514
85.3%
83.15%
69.84%
79.03%
Factorization
16426.14
0,9793
0,5009
0.5
0.5897
50.10%
49.98%
50.15%
49.94%
Natural
Lexical Inference
460.98
1.000
0.501
0.521
0,458
72.10 %
68.40%
70.67%
65.80%
Logical Inference
477.19
0.985
0.496
0.497
0.478
99.89 %
90.02%
97.58% %
99.61% %
Commonsense Inference
480.19
1.000
0.493
0.486
0.586
50.33%
51.60%
52.77%
49.99%
Table 3: Debiasing model against one bias (w1) does indicate model will learn task. Bias probabilities reflect the extent to which model predictions correlate with the biased feature. The unbiased test set does not contain W1- Even when debiasing procedures are effective and model predictions are not highly correlated with the targeted feature (wy). task performance does not always increase.
5.2 Debiasing in the limit
moved from the datasets." 67
We ask if, hypothetically, model were to be de- biased against all the spurious correlations that ex- isted in a dataset- would the model then be able to learn the task? To put provocatively, our hypoth- esis is that without shortcuts there is no learning that the models we study can do for more complex inference tasks.
Bias Frequency: We measure the performance of each debiasing method when changing the fre- quency of the biases. We see the performance of dataset filtering degrades the most, as the frequency of the biased examples increases (Fig. aa). As this technique filters biased examples, the more bi- ased examples there are, the fewer observations are available for the model to learn from.
In order to evaluate this hypothesis, we construct synthetic datasets such that w1- the known token correlation bias- is the only shortcut. We quan- titatively measure the extent to which some tasks are easier to learn than others from input text. In- spired by Voita and Titov (2020), we quantify this complexity using the online minimum description length (MDL) (Rissanen, 1978; Voita and Titov, 2020; Lovering et al., 2020), addition to accu- racy. We find that implementing any debiasing method does not enable the model to learn com- plex tasks. This is the case even when the dataset does not contain additional wy bias (Table 3). For the commonsense and the factorization tasks, per- formance remains close to majority baseline.
We compare debiasing procedures in high-resource and low-resource scenarios. The larger the dataset, the more effective PoE and Reweighting are (Fig. I b). This suggests that as the number of examples increases, the ability of debiased models to ignore biased features im- proves. This not the case for data filtering, where all biased examples are simply removed, and there- fore the bias never influences the model predictions (Fig. 1- b).
Dataset size:
We consider the effect of label noise in dataset. In many real-world datasets, small percentage of instances can be expected to be ambiguous wrongly labeled. As the amount of label noise increases, the baseline and CR in- creasingly rely on the wi-bias (Fig. 1 Graph 1), whereas all other debiasing methods rely more Graph 2). Data filtering
Label Noise:
5.3 How do debiasing procedures differ?
Each debiasing method has different strengths and weaknesses, and there is little evidence to suggest which procedure would be most effective in a given data regime. We control three parameters of a dataset: the frequency of biases in the dataset, the dataset size and label noise. We study four debias- ing procedures in these different data scenarios- example reweighting, PoE, confidence regulariza- tion, and dataset filtering (Wu et al., 2022; Le Bras et al., 2020b), a special case of example reweight- ing where the biased examples are completely re-
on the wa-bias (Fig. relies more on the bias than any other method, as this method is unable to learn from the wi -baa which is entirely filtered out (Fig. 1 - c Graph 2).
"This is equivalent to zeroing out the gradients for these examples under the example reweighting paradigm. "To explore the differences between example reweighting and dataset filtering, we set un and w2 to he 90% correlated with the class label since we use a counts-hased bias model.

---

Baseline
Reweight
PoE
Filter
Strength
Strenggth
wi-chellenge
1.0
0.8
80-
Pul predictdd
Accuracc th1
0.0
10
0.5-
0.6
50-
20%
60%
40%
80%
20%
60%
20%
60%
80%
freq.
(a) Changing the frequency of the wl and w2 biases (ranging from 10% to 90%)
Reweight
Filter
Sirength of model bias
Stieength of w2 modell bias
Performance on unbiased best
Performance on wl-challenge set
1.0
0.9
0.9
Accuracy (%g
Aesurey (82
0.8
0.7-
40
0.7
60
0.61
20
2/4
2
2/4
2/3
2
2-5
2
2
2
2
20
2
2
2
2
2
2
2 2
2
2-9
(b) Changing the dataset size (from 1.5625% to 100% of examples).
Baseline
Reweight
Filter
Strength of w1 model blas
Strength w2 model bias
Performance on unbiased tast
Performance on wi-challenge set
predited lanen)
0.8
Accursso (961
70-
0.7
No1.
20
50
10%
20%
30%
40%
50%
20%
20%
30%
40%
50%
10%
20%
30%
40%
50%
10%
20%
30%
40%
50%
Label noise
Label noise
Label noke
Label noise
(c) Changing the amount of label noise from 5% to 50% of examples.
Figure Debiasing performance for lexical entailment, stratified by bias frequency, dataset size, and label noise.
6 Discussion
debiasing technique to choose, such (1) Perfor-
mance on out-of-domain challenge sets, (2) Per- formance on long-tailed or rare phenomena, (3) Number additional parameters introduced by the debiasing procedure, (4) Additional time required during training/inference, (5) Efficiency of the de- biasing procedure (Schwartz et al., 2020).
We briefly discuss our findings and recommenda- tions for debiasing methods going forward.
Which debiasing technique to choose? In our work, we shed light how different debiasing techniques be effective based on properties of the training dataset, including how frequent biases are, the size of the dataset as well as the amount label noise in the data. We find that confidence regularization performs worse than the other de- biasing methods in a variety of data regimes we evaluate, sometimes performing little better than the baseline. PoE and reweighting often exhibit similar behavior. Data filtering performs worst in biase datasets.
Invest robust evaluation This work demon- strates that debiasing can cause models to switch from relying on known and measurable bias, to overrelying hidden biases in datasets. This sug- gests that practitioners would benefit from con- structing a wider range of evaluations to diagnose model reliance on spurious features in datasets, as well as methods to automatically identify these ases.
In addition, practitioners may want to consult several metrics while making the decision which

---

Why use debiasing? If the underlying task is accessible model but the model exploits short- cuts, debiasing can be effective. However, the underlying task is not accessible to models in the first place, debiasing procedures alone cannot be expected to be a legitimate path to making progress on the task.
2. We perform our experiments on tightly con- trolled, largely synthetic datasets where we can control both the task-relevant features as well as possible shortcuts. We do this as real- world datasets could have several possible in- teracting confounds. Our settings provide a controlled environment that then makes it fea- sible to study the behavior of debiasing ap- proaches which may not have been possible otherwise, at some expense of generalization.
Computational Reasoning Our work shows that for complex tasks, debiasing existing models may not be sufficient. This calls for investment in new learning paradigms that can help address complex tasks.
Our datasets are constructed with a maximum of two biases. practice, datasets may con- tain several biases which are usually unknown. The datasets used in this study are intended to represent a proof-of-concep of these scenar- ios to better understand model behavior.
Synthetic Datasets Our work utilizes synthetic datasets, with controlled bias settings to enable studying the behavior of debiasing approaches. These datasets allow us to validate specific hypothe- ses about model behavior which are challenging to study with real-world datasets where there might be several bias features, or combinations of bias features, that model can potentially exploit to ad- dress task. Future work can consider designing synthetic datasets as a tool to control for confound- ing factors and evaluate hypotheses about model behavior.
4. We look at methods of debiasing against specific bias. Some broad-spectrum debiasing methods do not target particular shortcut, but rather target several possible shortcuts the dataset that a particular "shallow' model has access to. In our framework, shorteuts wi which are intended to refer to specific bias, can also represent a class of biases 'shallow" model has access to.
7 Conclusion
5. Pretraining may affect dataset conditions, by inducing unanticipated correlations between tokens. order to control for this behavior, we construct six different datasets using both natural language, as well as numeric symbols.
NLU models have been shown to overrely on spu- rious features. This has lead to the development of several debiasing procedures to pressure models away from this behavior. In this work, we show that many of these procedures may simply push models to rely different set of shortcuts, and that the existing models may be able address com- plex NLU tasks even after all the spurious features have been addressed. We hope our work serves as a foundation to evaluate the effectiveness debias- ing procedures, sheds light on how these methods work, and provides an impetus for new classes of models for complex NLU tasks.
6. We investigate supervised settings in this study. We leave studying biases in few- shot and shoo learning settings for future work.
A Acknowledgements
The authors would like to thank Aditya Potukuchi, Aakanksha Naik, Tejas Srinivasan, and Nishant Subramani for valuable discussions related to this project. The authors would also like to thank the three anonymous reviewers for their valuable feed- back.
8 Limitations
We specify assumptions used in this study and high- light limitations of these assumptions.
1. We assume the task-specific label perfectly corresponds with task label in all our dataset constructions. In the real world however, there is likely to be some amount of label-noise, particularly in large-scale datasets.
References
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference

---

on Computer Vision and Pattern Recognition, pages 4971 19900.
Robin Cooper, Dick Crouch, Jan Van Eijck. Chris Fox. Johan Van Genabith, Jan Jaspars, Hans Kamp. David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical report.
Hossein Amirkhani and Mohammad Taher Pilehvar. 2021. Don't discard all the biased instances: In- vestigating core assumption in dataset bias miti- gation techniques. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3720-47223
ddo Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment chal- lenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177-190. Springer.
Sara Beery, Grant Van Horn, and Pietro Perona. 2018. Recognition in terra incognita. In Proceedings of the European conference computer vision (ECCV), pages 456-473.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Mas- simo Zanzotto. 2013. Recognizing textual entail- ment: Models and applications. Synthesis Lectures on Human Language Technologies, 1-20.
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Ben- jamin Van Durme, and Alexander Rush. 2019a. Don't tt take the premise for granted: Mitigating ar-
Ishita Dasgupta, Demi Guo, Andreas Stuhlmülller, Samuel Gershman, and Noah D. Goodman. 2015 8. Evaluating compositionality in sentence embeddings. In Proceedings of the 40th Annual Meeting of the Cognitive Science Society, CogSci 2018, Madison, WI. USA, July 25-28, 2018 cognitivesciencesoci- ety.org.
tifacts in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877-891, Florence, Italy, Association for Computational Linguistics.
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Ben- jamin Van Durme, and Alexander Rush. 2019b. On adversarial removal of hypothesis-only bias in natu- ral language inference. In Proceedings of the Fighth Joint Conference Lexical and Computational Se- mantics ( ('EEM 2019), pages 256-262, Minneapolis, Minnesota. Association for Computational Linguis- tics.
Jacob Devlin. Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Mengnan Du Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. 2022. Shortcut learning of large language models natural language understanding: A survey. arXiv preprint arXiv:2208.. 1857.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral expla- nation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9: 160 175.
Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the endingsstrog neural baselines for the ROC story cloze task. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics /Volume 2: Short Papers). pages 616-622, Vancouver, Canada. Association for Computational Linguistics.
Robert Geirhos. Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 116656677.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't it take the easy way out: Ensemble based methods for avoiding known dataset biases. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLL) pages 4069-4082, Hong Kong, China. Association for Computational Linguistics.
Abbas Ghaddar, Philippe Langlais, Mehdi Reza- gholizadeh, and Ahmad Rashid. 2021. End-to-end self- debiasebi framework for robust NLU training. In Findings of the Association for Computational Lin- guistics: ACI/JCCLPP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 1923-1929 Association for Computa- tional Linguistics.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2020. Learning to model and ignore dataset bias with mixed capacity ensembles. In Findings of the Associ- ation for Computarional Linguistics: EMNLP 2020, pages 3031-3045, Online, Association for Computa- tional Linguistics.
Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. 2017. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the

---

IEEE Conference on Computer Vision and Pattern Recognition. pages 1904-6913
Ronan Le Bras, Swabha Swayamdipta. Chandra Bha- gavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020a. Adversarial fil- ters of dataset biases. In International Conference on Machine Learning, pages 1078-1088. PMLR.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computa- tional Linguisties.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bha- gavatula. Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020b. Adversarial fil- ters of dataset biases. In International Conference on Machine Learning, pages 1078-1088 PMLR.
Tianyu Liu, Zheng Xin, Xiaoan Ding, Baobao Chang, and Zhifang Sui. 2020a. An empirical study on madel-agnostic debiasing strategies for robust natural language inference. In Proceedings of the 24th Con- ference on Computational Natural Language Learn- ing, pages 596 666, Online. Association for Compu- tational Linguistics.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132-142, Hong Kong, China. Association for Computational Linguistics.
Tianyu Liu, Xin Zheng, Xiaoan Ding, Baohao Chang, and Zhifang Sui. 2020b. An empirical study agdel-ag debiasing strategies for robust natural language inference. In Proceedings of the 24th Con- ference on Computational Natural Language Learn- ing, CoNLL 2020, Online, November 19-20, 2020, pages 596-608. Association for Computational Lin- guistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1rXiv 1907.11692.
Robin Jia and Percy Liang. 2017. Adversarial exam- ples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. 2020. Predicting inductive biases of pre- trained models. In International Conference on Learning Reprexentations.
Kushal Kafle and Christopher Kanan. 2017. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understand- ing, 163:3-20.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 8706 17668 Online, Asso- ciation for Computational Linguistics.
Divyansh Kaushik, Eduard H. Hovy, and Zachary Chase Lipton. 2020. Learning the difference that makes difference with 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababu, Ethiopia, April 26-30, 2020. OpenReview.net.
data. In
Rabech Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2021. Variational information bottleneck for effective low-resource fine-tuning. In Interna- tional Conference on Learning Representations.
Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? critical investigation of popular benchmarks. Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 5010- 5015, Brussels, Belgium. Association for Computa- tional Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Michael Mendelson and Yonatan Belinkov. 2021. De- biasing methods in natural language understanding make bias more accessible. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1545-1557.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. 2022. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.029877

---

Junghyun Min, Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. Syntactic data augmentation increases robustness to inference heuristics. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 2339-2352, Online. Association for Computa- tional Linguisties.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP--JCCILP) pages 3419-3442 3425, Hong Kong. China. Association for Com- putational Linguistics.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2355 Santa Fe, New Mexico, USA. Association for Com- putational L Linguisties.
Roy Schwartz, Jesse Dodge Noah Smith, and Oren Etzioni. 2020. Green ai. Communications of the ACM, 63:54-63.
Roy Schwartz, Maarten Sap, loannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. In Pro- ceedings of the 21st Conference on Computational Natural Language Learning (CoNLI. 2017). pages 15-25, Vancouver, Canada. Association for Compu- tational Linguistics.
Timothy Niven and Hung- Yu Kao. 2019. Probing neu- ral network comprehension of natural language argu- ments. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages Florence, Italy, Association for Compu- tational Linguistics.
Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Se- bastian Riedel, and Tim Rocktäschel. 2020. Avoiding the Hypothesis-One Only Bias in Natural Language Infer- ence via Ensemble Adversarial Training. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. 8. Hypothesis only baselines in natural language infer- ence. arXiv preprint 1805.0000
Altaf Rahman and Vincent Ng. 2012. Resolving com- plex cases of definite promouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Lan- guage Processing and Computational Natural Lan- guage Learning; pages 777-789, Jeju Island, Korea, Association for Computational Linguistics.
Masatoshi Tsuchiya 2018. Performance impact caused by hidden bias of training data for recognizing tex- tual entailment. In Proceedings of the Eleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Frano Rajič, Ivan Stresec. Axel Marmet, and Tim Poštuvan. 2022. Using focal loss to fight shallow heuristics: An empirical analysis of modulated cross- entropy in natural language inference. arXiv preprint arXiv:2211 13331.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spuri- ous correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621-633.
Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: Main Volume, pages 2363-3377, Online. Association for Computational Linguistics.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020a. Mind the trade-off: Debiasing nlu models without degrading the in in-distribution perfor- mance.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020b. Towards debiasing NLU models from unknown biases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 7597-7610, On- line. Association for Computational Linguisties.
Jorma Rissanen. 1978. Modeling by shortest data de- scription. Automatica, 14(5)465-471.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2019. Distributionally robust neu- ral networks for group shifts: On the importance of regularization for worst-case generalization. arXiy preprint 1xXiv:1911.08755
Elena Voita and Ivan Titov. 2020. Information- -heoorti probing with minimum description length. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP). pages 183-196.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M. Rush. 2021. Learning from others' mistakes: Avoiding dataset biases without modeling them. In 9th International Conference on Learning Representations, ICLR 2021. Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Fei Wang, James Y Huang, Tianyi Yan, Wenxuan Zhou, and Muhao Chen. 2023. Robust natural language un- derstanding with residual attention debiasing. arXiv preprint arXiv: 17627.

---

Xuezhi Wang, Haohan Wang. and Diyi Yang. 2022. Measure and improve robustness in NLP models: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACI, 2022, Seattle, WA, United States, July 10-15, 2022, pages 4569-4586. Association for Computational Linguistics.
pages 250-255, Minneapolis, Minnesota. Associa- tion for Computational Linguistics.
Peng Zhang. Yash Goyal, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5014-5022.
Jiaxin Wen, Yeshuang Zhu, Jinchao Zhang, Jie Zhou, and Minlie Huang. 2022. AutoCAD: Automatically generate counterfactuals for mitigating shortcut learn- ing. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 2302-2317, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Xiang Zhou and Mohit Bansal. 2020. Towards robus- tifying nli models against lexical dataset biases. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8759- 8771.
A Dataset Construction Procedures
Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is every- thing: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 996- 1005, Taipei, Taiwan, Asian Federation of Natural Language Processing.
We include expanded details of the construction procedures for the naturalistic datasets in our study.
Lexical reasoning The lexical reasoning task in- volves reasoning whether a pair of words are either antonyms or synonyms. Examples are created in the form of NLI using the template: 'What the group hate most is the [word]', with this phrase repeated for both the hypothesis and the premise where the synonym or antonym words replace the [word] tag. We use the antonyms and synonyms of nouns available on WordNet.
Adina Williams, Nikita Nangia, and Samuel Bowman, 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Humun Language Technologies, Volume 1 (Long Papers), pages 11 12-1 122, New Orleans, Louisiana. Association for Computational Linguis- tics.
The wi bias is inserted by changing the word 'hate' in the template for either 'like' or 'love'. When the word 'hate' is used, the example is neutral with respect to w1, while the word 'like' positively correlates with the entailment class and the word "love" positively correlates with the non- entailment class. The w2 word-overlap bias is in- serted by adding the text 'What we believe' or 'What we think' at the beginning of both sentences. For the u2 bias, the entailment class positively cor- relates with examples where this additional text is the same for the premise and hypothesis. The non-entailment class on the other hand positively correlates with examples that have different text added at the beginning of the sentences. Examples that are neutral with respect to the w2 word-overlap bias do not contain any additional text, in either the hypothesis or the premise.
Yuxiang Wu, Matt Gardner, Pontus Stenetorp, and Pradeep Dasigi. 2022. Generating data to mitigate spurious correlations in natural language inference datasets. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2660-2676, Dublin, Ireland. Association for Computational Linguistics.
Ruibin Xiong, Yimeng Chen, Liang Pang, Xueqi Cheng, Zhi-Ming Ma, and Yanyan Lan. 2021. Uncertainty calibration for sisemble-basee debiasing methods. Advances in Neural Information Processing Systems, 14:13657-13669
Yadollah Yaghoobzadeh, Soroush Mehri, Remi Ta- chet des Combes, T. J. Hazen, and Alessandro Sor- doni. 2021. Increasing robustness to spurious corre- lations using forgettable examples. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3319-3332, Online. Association for Computational Linguistics.
Logical reasoning The logical reasoning dataset contains premise expressing list of people who are undertaking an activity ("walking down street'), while the hypothesis uses a combination of AND and OR logical operators to also express which people are doing this activity. The NLI class is determined by reasoning over these AND and
Hitomi Yanaka, Koji Mineshima. Daisuke Bekki, Ken- taro Inui, Satoshi Sekine, Lasha Abzianidze, and Jo- han Bos. 2019. HELP: A dataset for identifying short- comings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019),

---

#Train
#Dev
#Test
GREATER NUMBER
5000 (50%)
1000 (50%)
16374 (50.16%)
SUM NUMBERS
5000 (50%)
1000 (50%)
16533 (50.29%)
FACTORIZATION
5000 (50%)
1000 (50%)
16417 (50.19%)
LEXICAL INFERENCE
2,808 (50.00%)
472 (50.00%)
476 (50.00%)
LOGICAL INFERENCE
20.000 (50.55%)
5,000 (50.16%)
5,000 (50.54%)
COMMONSENSE INFERENCE
2,080 (50.00%)
486 (50.21 %)
1095 (50.05%)
Table 4:
The number of instances in each dataset across splits (Train/Dew/Test), well the proporition of
instances of the majority class (in parentheses).
Reweight
DeBERTa correlation between and predictions
BERT correlation between and predictions
1.0
1.0
1
PWw1 - predicted label)
0.9
0.9
Piwl = predicted
0.8
0.8
0.6
0.6
0.5
0.5
Lexical task
Logic task
Commansense task
Lexical task
Logic task Commonsense task
Figure 2: Comparison of BERT and DeBERTA behavior, in terms of the proportion of examples where the model prediction is the same as the direction of the w2 word-overlap bias. 1.0 indicates that the w2 bias fully determines the model predictions, compared where the wz bias correlated with the model predictions.
OR operators to determine the truth of the hypoth- esis based on the premise. We avoid using two OR statements consccutively to ensure that the hy- potheses unambiguous while also being written fluently in natural language.
tences, with these phrases overlapping between the hypothesis and premise for entailment examples. Examples that are neutral with respect to u2 do not include any additional text at the end of either sentence.
Both the training and test data contain between 6 and 20 logical operators for cach hypothesis. While the training data uses exactly 2 OR statements in each hypothesis, the test data contains three or more OR statements. The names of the people undertak- ing the activity are also different in the training and test data.
Commonsense reasoning The commonsense reasoning dataset involves inserting the wi and biases into the DPR dataset (White et al., 2017; Rahman and Ng, 2012).
The wi and W2 biases are inserted by adding additional text at the beginning of each sentence without changing its meaning. We start each sen- tence with 'In the story', replacing the word 'In' with either 'Inside' (correlating with the entailment class) with 'Within' (correlating with the non- entailment class), The word 'In' is unchanged for examples that are neutral to the wi bias. The wa bias is inserted by changing the start of each sen- tence to be either 'In the story being narrated' or 'In the story being told', with the entailment class pos-
The wi word correlation bias is inserted into the dataset by changing the activity being performed, with 'running' correlating with the entailment class, and 'cycling' correlating with the non-entailment class. When the activity remains 'walking', then the observation neutral with respect to W1- The w2 word overlap bias involves adding the words fonce more' or 'once again' to the end of the sen-

---

itively correlating examples where the phrase is the same across both sentences. Examples that neutral to the wy bias do not include this additional text.
B Dataset Statistics and Analysis
We include detailed statistics of all six controlled synthetic datasets in Table. 4. For computing the MDL over naturalistic datasets, we consider 476 examples from each unbiased test set to ensure tasks are comparable.
C To what extent do our conclusions generalize to newer models?
While we follow prior work in studying the effect debiasing procedures on BERT (Utama al., 2020b; Ghaddar et al., 2021; Wang et al., 2023), we also wish to examine to what extent our con- clusions generalize to newer models. We examine the performance of all three debiasing procedures the three naturalistic datasets constructed in this study, finding that BERT and DeBERTA exhibit largely similar behavior under the effect of debias- ing procedures.


Reviews:
[{"review_id": "AC6BcaENX1", "reviewer": "Reviewer_BvPV", "summary": "", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for engaging with our work! \n\n**We are glad you found our results compelling and our experimental design well-motivated.**\n\n> Synthetically injected biases may not share properties with naturally occurring biases\n\nThis has been addressed for the following reasons:\n1. The data used in this work remains close to natural examples.\n\n Bias feature types have been substantiated in the literature (word overlap and tigger word biases [Dasgupta et al., 2018; Naik 129 et al., 2018; McCoy et al., 2019.]) (L 128-129; L 348-351). Task features include features relevant to real NLP tasks (DPR commonsense reasoning [Rahman and Ng, 2012], Lexical Entailment).\n\n2. The synthetic simulation allows control of several factors that have thus far made it challenging to conduct a similar analysis of debiasing methods, and we outline the limitations of this approach in the limitations section [L 577-586]. In NLP datasets, there may be several interacting confounds [L 61-71]. Thus, we sought to make a systematic simulation which allowed us to study the actual behavior of debiasing methods in a way that has not been done thus far.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental setup is well described and well-motivated, and the results are compelling.", "reasons_to_reject": "One concern I have is that synthetically injected features may not share properties in practice with actually naturally occurring features/biases picked up by the model.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "The authors present analysis on how methods for \"debiasing\" models based on features can lead to greater biases on untracked features. They demonstrate this on a mix on synthetic and natural language tasks, synthetically injecting the features to experiment on, and on BERT and DeBERTa models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hw4fr3MOT4", "reviewer": "Reviewer_FLhX", "summary": "", "questions": "Question A: Why do you expect debiasing would help address a task (Line 19)? The purpose of debiasing seems to just remove toxic biases from the model when deployed to the general public.\n\nQuestion B: Why not utilize existing debiasing datasets? What is the motivation and novelty of creating naturalistic data by themselves? For example, one possible issue is that using template “What the group hate most is the <>” might make the debiased model over-adapt to this sentence and shift model’s distribution away from a more realistic data distribution (of NLI); and the debiasing techniques were targeting at those more realistic data distributions. Similar things hold for other naturalistic dataset.\n\nQuestion C: Synthetic dataset seems a bit weird, because it’s very different from the distribution a LLM is used to.\n\nQuestion D: the instance will be ungrammatical if you remove w_2 in the data and make the task unresolveable by nature. How do the authors deal with this? For example, in Table 1 – “Lexical Inference”, the sample would be:\n**Prem**: What the group hate most is the sickness., **Hyp**: What the group hate most is the [“illness” deleted].", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your review and for engaging with this work! We are glad you found our paper to be well-written and our empirical findings interesting.**\n\n> Why do you expect debiasing would help address a task? The purpose of debiasing seems to just remove toxic biases from the model when deployed to the general public.\n\nThe goal of debiasing is to prevent models from relying on different types of spurious biases, thereby guiding the model to focus only on features actually relevant to the task and ideally also lead to better performance [L47-50]. Refer to our literature review L143-L164.  Toxic biases are only a subset of spurious biases. \n\n> Why not utilize existing debiasing datasets?\n\nWe do not use existing debiasing datasets, as we want to carefully and systematically control several factors in order to study the behavior of debiasing methods. \n\nSome of these considerations are:\n1. **Number of bias features that correlate with the task feature:** Our research question requires two, controlled dataset biases. One is a dataset bias that the model is initially de-biased against, and a second latent bias. This is not a setting investigated in existing debiasing datasets,\n2. **Avoiding confounding dataset biases:** NLP datasets contain several possible features that are challenging to disentangle. As stated in Zhou and Bansal 2020,  “In reality, almost every dataset contains countless such diverse [dataset] biases.”.\n3. **Constructing examples that are neutral with respect to each dataset bias:** To assess the performance of a model on the underlying task, we need to have the ability to test the model on examples that do not contain any dataset biases. \n4. **Strength of correlation between the dataset biases and each class, and frequency of the dataset biases:** Adjusting the strength and frequency of the biases allows us to understand the conditions under which each different de-biasing method is effective. This is not possible when using or modifying an existing de-biasing dataset.\n5. **Comparing de-biasing methods on a range of different tasks (with a range of difficulties):** By constructing synthetic simulation data, we are able to construct datasets for tasks free of confounding bias features, which in turn allows us to measure the difficulty of the underlying task itself for models.\n\n> Synthetic datasets seem weird\n\n1. Constructing synthetic datasets is a well-established tool in the literature for probing and studying model behavior. For example, Naik et al. 2019 used a series of templates to generate challenge datasets that surfaced shortcut-learning behavior in NLI models.  SImilarly McCoy et al. 2020 contribute a ‘controlled evaluation set’ called HANS to identify model reliance on heuristics. Another notable example is that of Lovering et al. 2020, who use both symbolic and naturalistic data to study the inductive biases captured by pretrained language models. Yet another example is Sinha et al. 2021, who studied model behavior on permuted sentences to examine how models process syntax.\n2. Our synthetic simulation tasks are constructed to be close to natural data. Several of the tasks we use in our simulation  include features relevant to real NLP tasks (DPR commonsense reasoning [Rahman and Ng, 2012], Lexical Entailment).  The bias feature types we use have been substantiated in the literature (word overlap and tigger word biases [Dasgupta et al., 2018; Naik 129 et al., 2018; McCoy et al., 2019.]) (L 128-129; L 348-351)\n3. The synthetic simulation we build allows us to control several factors that have thus far made it challenging to conduct a similar analysis of debiasing methods. In most NLP datasets, there may be several interacting confounds  [L 61-71], and it is challenging to estimate the number or type of bias features, the correlation of these features with the task, the frequency of the bias features in the dataset, and to estimate the difficulty of the underlying task itself. Our simulation allows us to systematically control all these factors, which allowed us to study the actual behavior of debiasing methods in a way that had not been done before.\n\n> Encoder and NLI Settings\n\nWe study existing debiasing methods. We do not propose new debiasing methods in this study. We thus construct our simulations to imitate settings where debiasing approaches thus far have been shown to be effective (sentence -pair classification tasks with encoder-only models such as BERT and RoBERTa as the primary model, see Mendelson & Belinkov, 2021, Wang et al., 2023, He et al., 2019; Clark et al., 2019; Mahabadi et al., 2020; Utama et al., 2020a,b; Sanh et al., 2021; Zhou and Bansal, 2020; Liu et al., 2020a; Clark et al., 2020;).\n\n> Naturalistic instances will be ungrammatical when removing w_2\n\n**Naturalistic instances will be grammatical when removing w_2.** The w_2 bias is applied to the naturalistic instances by adding a prefix (or suffix) to the text. This prefix (or suffix) does not change the reasoning about the underlying task.\n\nFor example, for the lexical inference task, a prefix of either ‘what we think’, or ‘what we believe’ is added at the beginning of the sentence (see line 353 and the footnote 4 on page 5). This prefix replaces the word ‘What’. The w2-bias correlates with the entailment class when both prefixes are the same (both sentences either have ‘what we think’ or ‘what we believe’ at the beginning). On the other hand, the w2-bias correlates with the non-entailment class when both sentences contain a different prefix. \n\nFor the ‘bias-neutral’ examples (w_2=0), there is no prefix added at the beginning of the sentences.\n\nWe shared below an example, using the hypothesis and premise mentioned by the reviewer. We will clarify this in the final version.\n\n| Setting     | Premise | Hypothesis |\n| ----------- | ----------- | ----------- |\n| Bias-neutral (w_2 = 0)    | What the group hate most is the sickness     | What the group hate most is the illness |\n| Bias correlates with entailment (w_2 = 1) | What we think the group hate most is the sickness | What we think the group hate most is the illness |\n| Bias correlates with non-entailment (w_2 = 2) | What we think the group hate most is the sickness | What we believe the group hate most is the illness |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and the message is clear.\n\nThe motivation of the paper could shed light on generative models. \n\nInteresting observation of the model keeps looking for spurious correlation to finish the task.", "reasons_to_reject": "Methodology and experiment design have some key problems and thus lack a strong support for the conclusion. \n\nThe setting seems toy — encoder, NLI; and has limited implication for more powerful generative models.\n\nSee questions for the author below.", "questions_to_authors": "Question A: Why do you expect debiasing would help address a task (Line 19)? The purpose of debiasing seems to just remove toxic biases from the model when deployed to the general public.\n\nQuestion B: Why not utilize existing debiasing datasets? What is the motivation and novelty of creating naturalistic data by themselves? For example, one possible issue is that using template “What the group hate most is the <>” might make the debiased model over-adapt to this sentence and shift model’s distribution away from a more realistic data distribution (of NLI); and the debiasing techniques were targeting at those more realistic data distributions. Similar things hold for other naturalistic dataset.\n\nQuestion C: Synthetic dataset seems a bit weird, because it’s very different from the distribution a LLM is used to.\n\nQuestion D: the instance will be ungrammatical if you remove w_2 in the data and make the task unresolveable by nature. How do the authors deal with this? For example, in Table 1 – “Lexical Inference”, the sample would be:\n**Prem**: What the group hate most is the sickness., **Hyp**: What the group hate most is the [“illness” deleted].", "paper_topic_and_main_contributions": "This paper examines three representative debiasing techniques to understand what models actually learn as a result of debiasing. They found current bias mitigation approaches cause models to shift from relying on measurable biases to relying on other unknown and unidentified biases in datasets.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 322, 324 330, 332: math equation rendering\n\nFigure 1 has too many details and y axis (and the subtitle) means different things (in a single row); this makes the plot hard to interpret.\n\nIn section 5.3, the author starts off by trying to figure out which debiasing procedure would be most effective, but then this section doesn’t give an answer. Instead, the beginning of section 6 gives the takeaway."}}, {"review_id": "TJhNwUJuGR", "reviewer": "Reviewer_BTLE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for engaging with our work, and championing our paper!**\n\n**We are pleased you noted our carefully-designed experiments and found that this work will correct the direction of future work on debiasing.**\n\n **Thank you particularly for noting that our experimental design is elegant and thorough.**\n\n\n> Lack of mathematical analysis for debiasing method’s performance\n\nWe contribute a mathematical framework both for measuring model bias (L 274-289), and for measuring task difficulty (L 458- 465). If there is any additional mathematical analysis you would like us to include please let us know, and we would be happy to address this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The synthetic experiment design is very elegant and useful for detecting the model's debiasing behavior;\n2. Running thorough experiments to compare different debiasing methods in different dataset properties;", "reasons_to_reject": "1. lack the mathematical analysis for each debiasing method's performance;", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper uses three carefully-designed experiments to probe model debiasing strategies. In these experiments, the \"hidden biases\" can be understood in an interpretable form. The paper shows that debiasing increases the model's reliance on hidden biases instead of robust features, which corrects the direction of future work for debiasing models and addressing more complex tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines 3 debiasing techniques for neural networks. They find that the techniques indeed result in models that rely less on the biases aimed to be removed, but may cause them to instead start relying on other unknown and unidentified biases, rather than helping it robustly solve the task.

The reviewers are generally in agreement that the question as well of the results are interesting. Both BvPV and FLhX point out that the synthetic nature of the used data takes away a bit from the strength of the results. I do not find that the authors succesfully rebut this concern (and neither does BTLE). Nevertheless, I don't think this should stand in the way of acceptance. There is always a trade-off between using realistic data and synthetic/toy data in terms of the conclusions that can be drawn, and I think the paper presents a valuable contribution as is.
Decision: Accept-Findings