Improving Bias Mitigation through Bias Experts in Natural Language Understanding
Authors: Eojin Jeon
Mingyu Lee
Juhyeong Park
Yeachan Kim
Wing-Lam Mok
SangKeun Lee
Venue: EMNLP 2023 Main (2023)
Keywords: Debiasing
natural language understanding
spurious correlation

Abstract: Biases in the dataset often enable the model to achieve high performance on in-distribution data, while poorly performing on out-of-distribution data. To mitigate the detrimental effect of the bias on the networks, previous works have proposed debiasing methods that down-weight the biased examples identified by an auxiliary model, which is trained with explicit bias labels. However, finding a type of bias in datasets is a costly process. Therefore, recent studies have attempted to make the auxiliary model biased without the guidance (or annotation) of bias labels, by constraining the model's training environment or the capability of the model itself. Despite the promising debiasing results of recent works, the multi-class learning objective, which has been naively used to train the auxiliary model, may harm the bias mitigation effect due to its regularization effect and competitive nature across classes. As an alternative, we propose a new debiasing framework that introduces binary classifiers between the auxiliary model and the main model, coined bias experts. Specifically, each bias expert is trained on a binary classification task derived from the multi-class classification task via the One-vs-Rest approach. Experimental results demonstrate that our proposed strategy improves the bias identification ability of the auxiliary model. Consequently, our debiased model consistently outperforms the state-of-the-art on various challenge datasets.

Improving Bias Mitigation through Bias Experts in Natural Language Understanding
Eojin Jeon' Mingyu Lee' Juhyeong Park', Yeachan Kim', Wing-Lam Mok', SangKeun Lee
DDepartment of Artificial Intelligence *Department of Computer Science and Engincering Korea University, Scoul, Republic of Korea (skdlcm456, decon9201, johnida, yeachan, wlmokac, yalphy)8korea.cer ac.
Abstract
Main Model
Main Model
Biases in the dataset often enable the model to achieve high performance on in-distribution data, while poorly performing on out-of- distribution data. To mitigate the detrimental effect of the bias on the networks, previous works have proposed debiasing methods that down-weight the biased examples identified by an auxiliary model, which is trained with ex- plicit bias labels. However, finding a type of bias in datasets is costly process. Therefore, recent studies have attempted to make the aux- iliary model biased without the guidance (or annotation) of bias labels, by constraining the model's training environment or the capabil- ity of the model itself. Despite the promising debiasing results of recent works, the multi- class learning objective, which has been naively used train the auxiliary model, may harm the bias mitigation effect due to its regularization effect and competitive nature across classes. As an alternative, we propose new debiasing framework that introduces binary classifiers be- tween the auxiliary model and the main model, coined bias experts. Specifically, each bias ex- pert is trained on binary classification task derived from the multi-class classification task via the One-vs-Ree Rest approach. Experimental results demonstrate that our proposed strategy improves the bias identification ability of the auxiliary model. Consequently, our debiased model consistently outperforms the otate-of- the-art various challenge datasets.'
B
B Which class?
CleesA?
Clees B3? Bias Expert
Clest G2
Blas Expert
Ausillary Model
Expart
Auxiliary Model
Inputs
Inputs
(a)
(b)
Figure 1: Comparison between (a) existing debiasing methods and (b) our proposed framework. (a): identify biased examples by using auxiliary model trained with the multi-class learning objective, the same as the main model. (b): introduce intermediate binary classi- fiers, called bias experts, between the auxiliary model and the main model.
work well on a specific data distribution (Ribeiro et al., 2016; Zhu et al., 2017; Gururangan et al.. 2018; McCoy et al., 2019). For instance, in nat- ural language inference (NLI) tasks, neural net- works are more likely to predict examples con- taining negation words as the contradiction class (Gururangam et al., 2018; Poliak et al., 2018). Ulti mately, this unintended usage of bias results in poor model performance on out-of-distributin data or bias-conflicting' examples, where spurious corre- lations do not exist. Therefore, it is important to develop debiasing methods.
1 Introduction
Previous works have addressed the issue by guid- ing the main model to down-weight biased exam- ples during training, which are identified by aux- iliary models. The auxiliary models are models that intentionally learn bias features by utilizing hu- man supervision (Kim et al., 2019; Schuster et al., 2019; Clark et al.. 2019; Mahabadi et al., 2020) or
Deep neural networks achieve state-of-the art per- formances on variety of multi-class classifica- tion tasks, including image classification (He et al.. 2016) and natural language inference (Devlin et al., 2019). However, they are often biased towards spu- rious correlations between inputs and labels, which
"These authors contributed equally to this work. 'Our code is available at https: thub. cww. com/jjjj27/ Bi as Expertss.
2 "Bias-conflin Bias-coofficting examples indicate examples that cannot he correctly predicted by solcly relying on biases.

---

prior knowledge about dataset biases (Bahng et al.. 2020). However, acquiring human supervision or prior knowledge about biases in myriad datasets a dauntingly laborious and costly process. Recent studies have therefore shifted towards training the auxiliary models without human guidance (or bias labels). These studies attempt to make the auxil- iary model biased by limiting the model capacity (Sanh et al., 2021), training for fewer epochs (Liu al., 2021), restricting accessible data (Utama et al., 2020b; Kim et al., 2022), or using GCE loss (Nam et al., 2020).
ples with bias experts.
We validate the effectiveness of the proposed framework on various challenging datasets. Experi- ment results show that our framework significantly improves performance on all datasets. The contri- butions of this paper are as follows:
We show that the multi-class learning objec- tive may harm the bias identification ability of the auxiliary model.
We propose novel debiasing approach based on bias experts to improve the out-of- distribution performance of existing debiasing methods.
Despite promising results, they still have room for further improvement. Specifically, the multi- class learning objective (e.g.. softmax cross- entropy), which previous works have naively used to train the auxiliary model, may harm the bias iden- tification ability of the model from few aspects: regularization effect and competitive nature across classes. Feldman et al. (2019) have demon- strated that the multi-class learning objective has regularization effect that prevents the model from overfitting on the training data. This is generally desirable for training the model, but not for train- ing the auxiliary model that needs to be overfitted to bias features. Since the softmax normalization results in the sum of all the class logits equal to one, lowering the prediction confidence in one class nec- essarily inereases the prediction confidence in the other classes (Wen et al., 2022). It can easily lead to overconfident predictions and may cause the aux- iliary model to misidentify unbiased examples as biased examples.
Through various empirical evaluations, we show that introducing bias experts improves debiasing performance.
2 Motivation
In this section, we describe our empirical obser- vations the existing debiasing approach. These observations serve intuitions for designing and understanding our bias experts. We first provide description of the experimental setup in Section 2.1. Then we provide our empirical observations in Section 2.2.
2.1 Setup
Consider training dataset D of task where each input E X is classified to a label y E V. Then each input æ can be represented by set of at- tributes A₂ = {ai..... 0x). The set Ag includes target attributes a E T, which are expected to be learned to perform the target task. It also includes bias attributes aj E B. which have high correla- tion with within data distribution D, but such correlation does not hold in other distributions.
To address the aforemenniooeed problem, we pro- pose a novel debiasing framework that introduces bias experts between the auxiliary model and the main model. As shown in Figure 1, the bias expert is binary classifier that identifies biased exam- ples within specific class rather than multiple classes. In practice, we first cast the multi-class classification task into multiple individual binary classification tasks via the One-vs-Ress approach (Rifkin and Klautau, 2004), which encourages the model to individually learn the bias attributes of the target class without the influence of other classes. In addition, our framework motivated by Nam et al. (2020) highlights biased examples of target classes that the auxiliary model deems "easy", allowing bias experts to focus more on learning the bias at- tributes of the class in charge. Finally, we train the main model by reweighting the loss of the exam-
In debiasing methods, the auxiliary model as- signs different weights to biased examples and bias- conficting examples. These works classify exam- ples that can be correctly predicted based solely on as as biased examples and regard examples that do not fall into this category as bias bias-confficting examples. To achieve this, an intentionally biased classifier fo X + y is employed the aux- iliary model. They identify biased examples by fo. which primarily rely on bias attributes ag, and down-weight them when training the main model fa : X + V. Namely, they consider examples that are correctly predicted by fo as biased examples,

---

0.0
siwv RJB dais
Confieene
knwy PLB: dass
Conforecee
kawy C239
Confitrro
idiaery Muri clis
0.4
22X
Skupa
(a) MNLI (Biascd/Bias-conffcciiii
(b) FEVER (Biased/Bias-confficnng
Figure 2: The process of neural network learning the biased datasets. For each dataset, the left and the right plots correspond to the result on biased and bias-conflicting example groups. Confidence refers to the average prediction probability the correct class on examples.
and down-weight their importance when training the main model.
the auxiliary model correctly predicts with high confidence as biased examples. Therefore, it should be able to have high confidence biased examples and low confidence bias-conflicting examples. But, as we observed before, the model trained with the multi-class learning objective shows the oppo- site tendency. These observations have interesting connections with those of Feldman et al. (201 9). They analyze that multi-class classification reduces overfitting due to remaining uncertainties across multiple classes. Looking this from the perspec- tive of biased model, it is similar to our results in that the multi-class learning objective can hinder the auxiliary model from learning bias attributes in biased examples.
2.2 Observations
In the aforementioned setup, observe the mod- els trained with two types of objective functions: multi-class learning objective and binary learning objective. The observations are made on the MNLI and FEVER datasets, which are widely studied in this area.
Bias attributes are learned slower Our first ob- servation is that model trained with the multi- class learning objective learns bias attributes more slowly. As shown in Figure confidence on bi- ased examples the model trained with the multi- class learning objective increases more slowly than those trained with the binary learning objective. Moreover, even after convergence, it shows lower confidence on biased examples.
3 Proposed method
Based on our observation in Section 2, we pro- pose a novel debiasing framework, introducing bias experts between the auxiliary model and the main model to improve bias mitigation methods. first splits the k-class classification dataset D U D; into multiple binary classification datasets B2, ..., By, where D: = {xx y)ly = i). B, {(x, 1 (x, v) E D)). and is an indicator function. Then for each i E {1, k}, we train the bias expert f X + (0. on B; with weighted loss Li to amplify the bias of fg. While to train more bias experts the number of classes (i.e., k) increases, most NLU tasks have mod- erate number of classes Therefore, our method is applicable across various tasks, including NLI, paraphrase identification, sentiment analysis, and topic classification. Finally, we train the debiased model fa by reweighting the loss of each exam- ple with J.).. The overall process of our
Target attributes are learned faster The sec- ond observation that the model trained with the multi-class learning objective learns the target at- tributes faster. Figure reports the average con- fidence of models on bias-conflicting examples. To predict bias- canfficoffinn examples correctly, the models have to leverage a, rather than ag. As seen in the figure, the confidence bias- bins-conf amples of the model trained with the multi-class learning objective starts to increase ahead of the model trained on the binary learning objective. In addition, the degree of confidence increment is also larger in the multi-class learning objective.
need
Multi-class learning objective is sub-optimal From these observations, we argue that the multi- class learning objective sub-optimal to training the auxiliary model. Existing debiasing methods typically train the auxiliary model with the multi- class learning objective and consider examples that
'For instance, there are typically 2-3 classes in NLJ, 2 classes in paraphrase identification, 2-5 classes in sentiment analysis, and 4-20 classes in topic classification.

---

Weighted samples w/ Binary Jabels
Blased example Bias cass-ooflicttin example
Sample weight w low
Bias Experts
high
O AO
o A
fe Class
Weighting function larget label:
Main Model
Training set
A uxiliary Model
fa
A
O
fb
Biased Prediction
fa
A
difficulty
Class
Clesss
Classs
Whichh class?
Else: w
Class
difficulre
Training Inputs
Class
Positive
Negative
samples
samples
Figure The overall pipeline of our method. Each bias expert trained by assigning high weights to biased examples belonging the target class and bias-conflictigg examples not belonging to the target class. As result, each bias expert focuses on biased examples of its target class.
framework is illustrated conceptually in Figure 3.
bias expert is required to predict only the biased examples of the target class as the target class with high confidence, while, on bias-conflicting exam- ples, the bias expert required predict with low confidence. However, by increasing the weight all easy examples, the bias expert can learn the bias attributes of the non-target class and predict bias-conflicting examples the target class with high confidence, which is not in line with our pectations. For example, in the the bias-confficting example of neutral class with neither lexical overlap nor negation words can be identified as biased with high confidence".
3.1 One-vs-Rest
In Section observe that the multi-class learn ing objective impedes the auxiliary model from being biased. Thus, we introduce One-vs-Rest (OvR, also called One-vs-All) approach to train the auxiliary model on the binary learning objective.
The OvR is an approach in machine learning to making binary classification algorithms (e.g., logistic regression model) capable of working multi-class classification dataset 'D by split- ting D into multiple binary datasets B1, B2, ..., Bk- For example, the MNLI dataset, which includes "contradiction", "entailment" and "neutral" classes, is divided into the following three binary classifi- cation datasets (target vs non-target) through the OvR approach:
of the MNLI,
To discourage bias experts from learning bias attributes of non-target classes, we assign large weight to bias-conflicting examples of non-target classes. Therefore, inspired by Zhang and Sabuncu (2018) and Nam et al. (2020), we train bias experts by down-weights weightiig the loss of biased examples in the non-target class based on the confidence of the auxiliary model as follows:
By: Contradiction vs Non-Con Contradiction
By: Entailment vs Mon-Emtailment
By: Neutral Non-Neutral
(1 Fu 1og(1-a(()()))) (a.
Caon-target.i
This strategy enables the model to individually learn the bias attributes of the target class without the competition across different classes, which is caused by the softmax normalization.
|
(1)
where is the target class, In denotes the weight of the example (z, y), and o(-) is the sig- moid activation function. is hyperparameter that controls the degree of amplification. At the same time, we down-weight bias-conflicting exam-
3.2 Training bias experts
In this work, define the bias expert fo be binary classifier that identifies biased examples within specific class. We expect fo to learn the bias attributes of the target class and identify bi- ased examples by relying on them. Specifically, the
"Lexical overlap and the presence of negation words are the hias attrihutes of entailment and contradiction classes, respoctively.

---

ples of the target class:
(Williams et al., 2018). For evaluation, eval- uate the in-distribution performance of the mod- els on the MNLI validation set and the out-of- distribution performance on HANS (McCoy et al., 2019), an evaluation set designed to determine whether models have adopted three predefined syn- tactic heuristics such lexical overlap.
Ln.y log o (f((z)) |Dj
Largett Σ
(2)
(E.y)ED;
We expect that these losses encourage bias ex- perts to learn the bias attributes of the target class rather than those of the non-target class. Thus, minimize them over bias experts:
Fact verification Fact verification involves de- termining if the evidence supports refutes claim, or it lacks sufficient information. We train our proposed model and baselines on the FEVER training set (Thorne et al., 2018). For evaluation, we evaluate the in-distribution performance of the models on the FEVER validation set and the out- of-distribution performance on FEVER Symmetric (Schuster et al., 2019), an evaluation set designed to test whether models rely spurious cues in claims.
1 ((06) K E Cargerttt. mo (3)
where Å1 and 22 are the hyperparameters that con- trol the class imbalance aroused by OvR. At this time, backpropagation is not performed for the aux- iliary model.
3.3 Training debiased model
Paraphrase identification Paraphrase identifica- tion is the task of detecting whether given pair of questions is semantically equivalent. We use the QQP dataset and divide this dataset into training and validation sets so that the validation set con- tains 5k examples, following Udomcharoenchaikk et al. (2022). We train proposed model and baselines the resulting QQP training set. For evaluation, we evaluate the in-distribution perfor- mance of the models on the QQP validation set and the out-of-distribution performance on PAWS (Zhang et al., 2019) to test whether models learn to exploit lexical overlap bias.
We train a debiased model fa using product-of- experts (PoE) which is widely used in NLU debi- asing research (Clark et al., 2019; Mahabadi et al., 2020; Sanh et al., 2021). In PoE, fa is trained in an ensemble by combining the softmax outputs of fa and fo- The ensemble loss for each example is:
((O.) y y log softmax(Iog Pa log Pt). (4)
With this loss function, examples that bias experts predict correctly are wewn-weighted, thereby dis- couraging fa from learning attributes exploited by bias experts. Since we use multiple bias experts, we apply softmax to the final outputs of bias experts and use this as ps- During the training, we freeze the parameters of fo.
4.2 Baselines
We compare multiple baselines, which are gener- ally classified into two categories:
4 Experiments
Methods using prior knowledge of bias: In such methods, auxiliary models are trained by us- ing the hand-crafted features which indicate how words in one sentence shared with another sen- tence (Clark et al., 2019; Mahabadi et al., 2020; Utama al., 2020a), or whether a certain n n-gram occurs a sentence (Schuster et al., 2019).
4.1 Evaluation Tasks
We evaluate our model and baselines on three natu- ral language understanding tasks that have out-of- distribution evaluation sets: natural language infer- ence, fact verification, and paraphrase identifica- tion. We use accuracy the performance metric for each task. The detailed information for each task is as follows:
Methods without targeting a specific bias: This group of methods forces auxiliary models to learn bias attributes either by reducing the training dataset size (Utama et al., 2020b), restricting the model capacity (Sanh al., 2021; Ghaddar et al., 2021), without targeting specific bias type.
Natural language inference Natural language inference (NLI) is the task of determining whether a premise entails, contradicts, or is neutral to a hypothesis given a sentence pair. We train our pro- posed model and baselines on the MNLI training
hhttps: http::///ooraaaaaaa// www.u.uurroooooooo homra.ccm// First- First-Quora- Quora-Dataset- Reelease-Question

---

Known biases
MNLI
FEVER
QQP
Method
dev
HANS
Gap
dev
symm. Gap
dev
PAWS
Gap
BERT-base (Devlin al., 2019)
84.5
62.4
22.1 85.6
63.1
22.5
91.0
33.5 57.5
Reweighting (Clark et al., 2019)
83.5
69.2
14.3
Reweighting (Schuster et al., 2019)
84.6
66.5
18.1
PoE (Clark et al., 2019)
82.9
67.9
15.0
Conf-reg (Utama et al., 2020a)
84.3
69.1
15.7
86.4
66.2
20.2
Reweighting (Utama et al., 2020b)
82.3
69.7
12.6
87.1
65.5
21.6
85.2
57.4
27.8
PoE (Utama et al., 2020b)
81.9
66.8
15.1
85.9
65.8
20.1
86.1
56.3
29.8
PoE (Sanh al., 2021)
83.3
67.9
15.4
84.8
65.7
19.1
88.0
46.4
41.6
Conf-reg (Utama et al., 2020b)
84.3
67.1
17.2
87.6
66.0
21.6
89.0
43.0
46.0
Self-Debiasing (Ghaddar et al., 2021)
83.2
71.2
12.0
90.2
46.5
43.7
Bias Experts (ours)
X
82.7
72.6
10.1
85.6
68.1
17.5
86.8
58.1
28.7
Performances of models evaluated on MNLI, FEVER, QQP, and their corresponding challenge test sets. We also report the difference between in- and cit-of-distribution performances as Gap. The results of baselines using prior ledge of biases are from the original paper. We mark the best and the second-best performance bold and underline, respectively.
Table I:
4.3 Implementation details
MNLI
HANS
Ablative Setting
Ent
NEnt
Avg
In all tasks, we employ the BERT-base model with 110M parameters for the main model and use the
Bias Experts (ours)
82.7
93.5
51.8
72.6
(1) w/o Amp.
82.8
94.9
45.0
70.0
BERT-tiny (Ture et al., 2019) model with 4M pa- rameters for the auxiliary model and the bias ex- perts We use Adam W as the optimizer for both the main model and the biased model, with a batch size of 32 and epochs of training. In order to train the main model, we use a coefficient of 0.3 for the cross-entropy loss and a coefficient of 1.0 for the PoE loss, following Sanh et al. (2021). For the NLI task, set the learning rate to 3e-5 and to 0.2. For the fact verification task, 2e-5 and the a 0.01. For the paraphrase identification task, we set the learning rate 2e-5 and the a to 0.3.
(2) w/o OvR
83.0
94.4
42.7
68.5
w/o Amp. and OvR
83.1
94.5
41.4
67.9
Table 2: Ablation studies using accuracy on MNLI and
HANS. We mark the best performance in bold.
smaller than the other methods, suggesting that our framework achieves more out-of-distribuutin performance gains with less in-distribution perfor- mance losses. On the other hand, that the proposed framework shows the lowest per- formance improvement PAWS, the challenge test set QQP. This is because only bias amplification affects the performance improvement since QQP is originally a binary problem.
the learning
observe
4.4 Main results
We report experimental results on three natural lan- guage understanding tasks in Table 1. Each result is the average of the scores across 5 different runs. We observe that our model consistently outperforms the state-of-the-art three out-of-distribution eval- uation sets. Specifically, model shows perfor- mance improvement of 1.4%p, 2.1 %p, and 0.7%p in HANS, FEVER Symmetric, and PAWS over the best- pest-performing model, respectively. These results indicate that our framework is effective identify- ing biased examples regardless of the type of the dataset. In addition, the gap between in- and out- of-distribution performances of our model is much
4.5 Ablation study
Effect of each module in our method We com- pare different ablative settings of our method to demonstrate the effect each part in our method. (1) w/o Amp. denotes omitting bias amplification of biased models by setting 0 in Eq. 1 and Eq. 2, (2) w/o OvR denotes that we train the main model with ensemble prediction of three auxil- iary models trained with the multi-class learning objective, (3) w/o Amp. and OvR denotes that we remove both bias amplification and OvR approach. Table indicates that all the components are
//ttps://github.. http.///wwggggggeeee/eaannnrrrrm

---

100
Method
MNLI
HANS
BERT-base
84.5
62.4
PoE (Utama al., 2020b)
81.9
66.8
PoE (Sanh et al., 2021)
83.3
67.9
PoE Ours (T 2k)
83.8
68.0
MNLI
HANS Ent
PoE Ours (T = 10k)
83.3
71.0
HANS Non-Ent
Bias Experts (ours)
82.7
72.6
Table 3: Trade-off results between in- and out-of- distribution performances evaluated MNLI and HANS, T denotes the number of training steps for bias experts. We mark the best and the second-best perfor- mance in bold and underline, respectively.
40
Figure 4: Performances main models evaluated on MNLI and HANS with different
4.6
In- and Out-of-distribution Performances Trade-off
important in the improvement of debiasing perfor- mance. Amplifying the target class bias of bias experts (w/o Amp.) is crucial to boost the out-of- distribution performance, indicating that focusing learning biased examples of the target class improves the bias experts' ability to identify bi- ased examples. In addition, omitting the OvR ap- proach (w/o OvR) further decreases the perfor- mance, which suggests that training bias experts with the binary learning objective is key to im- proving the performance of bias experts. Further- more, using both OvR and Amp. causes larger im- provement in out-of-distributie performance com- pared to using of them. These results show that the two components work complementary each other, improving different aspects of bias experts. Additional results on the fact verification task are provided in Appendix A.6.
To verify that our method achieves a better trade-off between in- and out-of-distriutioon performances than baselines, we conduct an analysis by tuning the number of training steps for bias experts. The intuition behind tuning the number of training steps is that the biased models with smaller training steps are likely to less focus on bias attributes, resulting in a main model with higher in-distribution per- formance, compared to the setting where biased models are fully fine-tuned (e.g., for epochs as in Section 4.3). The results are presented in Table 3. By comparing the results of PoE Ours (T = 2k) in 4 with those of Sanh al. (2021) in row 3, where T denotes the number of training steps for bias experts, can see that our model can achieve the similar duiti--disttibution performance gain in HANS with the smaller -idiiitribttinn performance degradation in MNLL In addition, the comparison results between PoE + Ours (T' = 10k) in row 5 and the method in Sanh et al. (2021) demonstrate that our model improves the out-of-distribution per- formance while preserving the in-distribution per- formance. These results indicate that method facilitates a better balance between in- and out-of- distribution performances.
Degree of bias amplification We vary the degree of bias amplification for the target class (i.e., a) and compare the performances of the correspond- ing main models. Figure 4 shows the results. We first observe that debiasing with a = shows the lowest accuracy. This indicates that bias amplifica- tion helps bias experts focus on the target class bias, leading to better out-of-dissee difdrriitbbtionn performances of the main model. On the other hand, as exceeds 0.2, both in-distribution and out-of-distributie per- formances start decrease. We speculate that this is because the main model receives a smaller gra- dient update as a increases. This indicates that we should train the bias experts either by using mod- erate level of a or by increasing the learning rate.
4.7 Analysis Model Confidence
To further investigate the effectiveness of our bias experts, we use the kernel density estimation to draw the distributions of model confidence two groups of examples: biased and bias-conflicting ex- amples. Specifically, we experiment on the MNLI training set and compare our bias experts with the weak learner used in Sanh et al. (2021). The result illustrated Figure 5. It is observed that our bias

---

Ent Subseq
NEnt
Method
Lexical
Const 99.4
Lexical
Subseq
Const
BERT-base
99.5
99.2
52.4
10.0
17.8
PoE. (Sanh al., 2021)
90.2
97.0
96.3
66.0
18.0
40.1
Bias Experts (ours)
88.4
96.6
95.5
83.7
26.4
45.2
Table 4: Performances of main models evaluated on HANS for each heuristic. The columns Lexical, Subseq, and Const mean lexical overlap, subsequence, and constituency, respectively. We mark the best performance in bold.
Method
BAR
NICO
Bias attribute: Lexical overlap
ERM
35.3
42.6
Bias Experts
ReBias (Bahng et al., 2020)
37.0
45.2
12
Weak Learner
LfF (Nam al., 2020)
48.2
40.2
Density
9-
LWBC (Kim et al., 2022)
62.0
52.8
Bias Experts (ours)
64.2
54.1
6-
3
Table 5: Performances of models evaluated BAR and NICO. We compare our method with baselines that do not use bias labels. We mark the best and the second- best performance in bold and underline, respectively.
0
0.6
1.0
Confidence
(a) Biased examples
Bias attribute: Lexical overlap
experts show higher confidence on biased examples and lower confidence on bias-conflicting examples average, compared to the weak learner in Sanh et al. (2021). This indicates that the bias experts identify biased and bias- bias-confficting examples more precisely than the biased model applied in the pre- vious work. As result, we can observe in Table that the main model trained in an ensemble with the bias experts achieves significant improvement over the baselines on the non-entailment subset of HANS, for all three heuristics, with only small degradation on the entailment subset of HANS.
Bias Experts
Weak Learner
8
Density
0.0
0.2
0.4
0.6
Confidence
(b) Bias-conflicting examples
Figure 5: The kernll density estimation of models'
fidence on (a) biased examples and (b) bias-conflicting examples in MNLI. The weak learner the auxiliary model used in Sanh et al. (2021). Each plot includes dashed lines representing the average confidence each model.
4.8 Experiment on Image Classification
We conduct experiments on computer vision tasks to verify that our framework is model-agnostic, and be applied to other debiasing algorithms.. For computer vision tasks, we apply bias experts the bias committee algorithm (Kim et al., 2022). We experiment on two image classification datasets: I) Biased Action Recognition (BAR): a real-world image dataset crafted to contain examples with spu- rious correlations between human action and place, and 2) NICO: real-world dataset for simulating image classification scenarios. We compare our model with baselines without plicit supervision on bias: ReBias (Bahng al.. 2020), L.F (Nam et al., 2020), and LWBC (Kim et al., 2022). Table shows the results on both datasets. Our model achieves performance improve-
ments of 2.2%p and 1.3%p BAR and NICO over LWBC, respectively. The results show that our framework can be used complementary with other debiasing algorithms in computer vision tasks.
5 Related Work
5.1 Debiasing with prior knowledge of bias
Earlier bias mitigation methods typically combine human prior knowledge with data-driven training. They define a specific bias type in advance and train the auxiliary model to learn the predefined bias

---

type. For example, the auxiliary model is trained learn a word overlap bias or hypothesis-only bias (He et al., 2019; Mahabadi et al., 2020) in NLI, a question-only bias (Cadène et al., 2019)
effect of the multi-class learning objective on the bias identification ability of an auxiliary model.
Conclusion
in VQA, and a texture bias (Bahng et al., 2020)
We have proposed novel framework that intro- duces a binary classifier between the main model and the auxiliary model, called bias experts, for improving the bias mitigation method. The main intuition of this work is to mitigate the incorrect bias identification caused by the naive application of the multi-class learning objective to train the auxiliary model. We train our bias experts via the One-vs-Rest approach, pushing each bias expert to foeus more on the biased examples in its tar- get class. Through various experiments and abla- tion studies, we demonstrate our framework effec- tively alleviates the aforementioned problem and improves existing bias mitigation methods.
in image classification. Then, the auxiliary model reduces the importance of biased examples when training the main model by reweighting examples (Schuster et al., 2019), regularizing the main model confidence (Utama et al., 2020a; Mahabadi al., 2020), or combining with the main model in the PoE manner (Clark et al., 2019). As a result, the main model focuses more on bias-conflicting ex- amples in which exploiting solely bias attributes is not sufficient to make correct predictions. While these methods provide remarkable improvement in the duttrf-distriiuttinn performance, manually identifying all bias types in tremendous datasets is a cime-consuming and costly process.
Limitations
5.2 Debiasing without targeting a specific bias
Although we have demonstrated the efficacy of our method in improving bias mitigation, there are two limitations that should be addressed in the future: (1) Since we have to train individual binary clas- sifiers for each class, a limitation of our work is that it may lead to large memory usage as the num- ber of classes increases. However, to the best of our knowledge, most NLU tasks consist of only moderate number of classes. For example, we found that there typically 2-3 classes in NLI, 2 classes in paraphrase identification, 2-5 classes in sentiment analysis, and 4-20 classes in topic classi- fication. In addition, our work has validated wide applicability in datasets with a moderate number of classes across various NLU and image classi- fication tasks. In the future, we plan to alleviate this limitation by exploring the way to adopt the parameter-efffcient fine-tuning method for training bias experts.
To address these limitations, several attempts were conducted to train the auxiliary model without hu- man intervention. These works force the auxiliary model to learn the bias attributes in datasets by restricting the size of training data (Utama et al., 2020b), limiting the capacity of the model (Bras et al., 2020; Clark et al., 2020; Sanh et al., 2021; Yaghoobzadch et al., 2021: Ghaddar et al., 2021), training the model with GCE loss (Nam et al., 2020), or reducing the training epochs (Liu al., 2021). Then, they consider the examples for which the auxiliary model struggles to predict the cor- rect class as bias-confficting ones and assign large weights to them when training the main model. Recently, to further improve debiasing methods, Kim et al. (2022) use the consensus of multiple auxiliary models to identify biased examples and determine their weights, Ahn et al. (2023) use the norm of the sample gradient determine the im- portance of each sample, Lee et al. (2021) augment bias-confficting features with feature-mixing tech- niques, and Lee et al. (2023) propose a method which removes the binscconn examples from a training set to amplify bias for a biased model. Lyu et al. (2023) take contrastive learning ap- proach to mitigate bias features and incorporate the dynamic effects of biases. Although these prior ef- forts achieve promising results, to our knowledge, none of them attempted to improve the effective- ness of debiasing methods by tackling a detrimental
(2) We introduce hyperparameters in our work, which could be problematic in debiasing works since most nut-of-disttibuutin datasets do not pro- vide a validation set for tuning hyperparameters, as noted by Utama et al. (2020a). With this in mind, we have sought to minimize the number of hyper- parameters additionally introduced in our work. Specifically, there are three additionally introduced hyperparameters, λT. A2. and a. Following Wen et al. (2022), we set A (k: 1)/k k: and X2 = 1/k, where k denotes the number of classes. Thus, A and X are not involved in the tuning process.

---

addition, compared with previous works, the per- formance of the proposed method is less sensitive to the hyperparameter a. Specifically, in HANS, the performance of our method varies with a vari- ance of 0.99, depending on the value of a. This variance is lower than 1.10 - 8.80, the variances of performance observed in prior works (Utama et al., 2020b; Sanh et al., 2021; Kim et al., 2022; Lyu al., 2023). In the future, we plan to investi- gate how to tune hyperparameters without leaking information about the bias.
of dataset biases. Proceedings of the 37th Interna- tional Conference on Machine Learning, 1CML 2020. volume 119 of Proceedings of Machine Learning Research, pages 1078-1088, PMLR
Rémi Cadène, Corentin Dancette, Hédi Ben-Younes, Matthicu Cord, and Devi Parikh. 2019. Rubi: Reduc- ing unimodal biases for visual question answering. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, pages $39-8500 Curran Associates, Inc.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't 't take the casy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pages 4067-4080. Association for Computational Linguis- tics.
Ethics Statement
We acknowledge that the high out-of di-diitrinttion performance of our method is achieved at the ex- pense of the in-distribution performance (i.c., at the expense of the performance in the majority group). One of the risks that can stem from this trade-off is that errors in the majority group could increase in applications such as toxicity classification,, fa- cial recognition, and medical imaging, potentially leading to reverse discrimination against samples in such a majority group
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2020. Learning to model and ignore dataset bias with mixed capacity ensembles. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3031-3045, Association for Computa- tional Linguistics.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutamova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLI, 2019, pages 4171-411 7776-416..40 As- sociation for Computational Linguistics.
Acknowledgements
This work was supported by the Basic Research Program through the National Research Founda- tion of Korea (NRF) grant funded by the Korea government (MSIT) (2021R1 R021RIA2C30111 A2C3010430) and In- stitute of Information & Communications Technol- ogy Planning& & Evaluation (IITP) grant funded by the Korea government (MSIT) (N0.2019-0-0009, Artificial Intelligence Graduate School Program (Korea University)).
Vitaly Feldman, Roy Frostig, and Moritz Hardt. 2019. The advantages of multiple classes for reducing over- fitting from test set reuse. In Proceedings of the 36th International Conference on Machine Learning, ICML. 2020, volume 97 of Proceedings of Machine Learning Research, pages 1892-1900, PMLR
References
Abbas Ghaddar, Philippe Langlais, Mehdi Reza- gholizadeh, and Ahmad Rashid. 2021. End-to-end self-debiasing framework for robust NLU training. In Findings of the Association for Computational Linguistics: ACL/ICCNPP 2021, pages 1929-1929. Association for Computational Linguisties.
Sumyeong Ahn. Seongyoon Kim, and Se- Young Yun. 2023. Mitigating dataset bias by using per-sample gradient. In The Eleventh International Conference on Learning Representations, ICLR 2023. OpenRe- view.net.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy. Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018 Annotation artifacts in natural lan- guage inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIL 2018, pages 107-112. Association for Computational Linguistics.
Hyojin Bahng. Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. 2020. Learning de-bia representations with biased representations. In Pro- ceedings of the 37th International Conference on Ma- chine Learning, 1CML 2020, volume 119 of Proceed- ings of Machine Learning Research, pages 528-539. PMLR.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bha- gavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop

---

On Deep Learning Approaches for Low-Resource NLP, DeepLo@ SOMNLP-UCCLL 2019, pages 132- 142. Association for Computational Linguisties.
of the 57th Conference of the Association for Com- putatioonal Linguistics, ACL 2019. pages -428-3448. Association for Computational Linguisties.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pages 770- 778. IEEE Computer Society.
Jun Hyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. 2020. Learning from failure: De- biasing classifier from biased classifier. In Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, pages 19673-20688 Curran Associates, Inc.
Byungju Kim. Hyunwoo Kim. Kyungsu Kim, Sungjin Kim, and Junmo Kim. 2019. Learning not to learn: Training deep neural networks with biased data. In IEEF/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2019, pages 9012-9020. Computer Vision Foundation/ / IEEE
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language in- ference. In Proceedings of the Seventh Joint Con- ference on Lexical and Computational Semantics, "SEM @ @NACNNAAHLLL 2018, pages 180-191 191. Associa- tion for Computational Linguistics.
Nayeong Kim, Sehyun Hwang, Sungsoo Ahn, Jaesik Park, and Suha Kwak. 2022. Learning debiased clas- sifier with biased committee. In Advances in Neu- ral Information Processing Systems 35: Anmual Con- ference on Neural Information Processing Systems 2022, NeurlPS 2022, pages 18403-18415. Curran Associates, Inc.
Marco Tálio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should trust you?": Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD 2016, pages 1135 1155-114 1144. ACM.
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lec, and Jacgul Choo. 2021. Learning debiased rep- resentation via disentangled feature augmentation. In Advances in Neural Information Processing Systems 34; Annual Conference on Neural Information Pro- cessing Systems 2021, NeurlPS 2021, pages 25122 -5123- 25133. Curran Associates, Inc.
Ryan M. Rifkin and Aldebaro Klautau. 2004. In defense of one-vs-all classification. J. Mach. Learn. Res., 5:101-141.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M. Rush. 2021. Learning from others' mistakes: Avoiding dataset biases without model- ing them. In The Ninth International Conference on Learning Representations, ICIR 2021. OpenRe- view.net.
Jungsoo Lee, Jeonghoon Park, Daeyoung Kim, Juyoung Lee, Edward Choi, and Jaegul Choo. 2023. Revisit- ing the importance of amplifying bias for debiasing. In Shirty-Seventh AAA/ Conference on Artificial In- telligence, AAAI 2023, pages 14974-14981. AAAI Press.
Tal Schuster, Darsh J. Shah, Yun Jie Serene Yeo. Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pages 3417-3423. Association for Computational Linguis-
Evan Zheran Liu, Behzad Haghgoo. Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. 2021. Just train twice: Improving group robustness without training group information. In Proceedings of the 38th Interna- tional Conference on Machine Learning, 1021,2021, volume 139 of Proceedings of Machine Learning Research, pages 6781-6792. PMLR.
tics.
Yougang Lyu, Piji Li, Yechang Yang, Maarten de Rijke, Pengjie Ren, Yukun Zhao, Dawei Yin. and Zhaochun Ren. 2023. Feature-leve debiased natural language understanding. In Thirty-Sevnnth AAAI Conference on Artificial Intelligence, AAAI 2023, pages 13353- 13361. AAAI Press.
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. 2018. FEVER: a large-scale dataset for f'ct extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-III.T 2018, pages 809-819. Association for Computational Linguistics.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, ACL 2020, pages 8706-8716 8716. Association for Computational Linguistics.
Iulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.088
Tom McCoy, Ellic Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In Proceedings
Can Udomcharoenchaikit, Wuttikorn Ponwitayarat, Patomporn Payoungkhandee, Kanruethai Masuk, Weerayut Buaphet, Ekapol Chuangsuwanich, and

---

Sarana Nutanong. 2022. Mitigating spurious cor- relation in natural language understanding with coun- terfactual inference. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing. EMNLP 2022, pages 11308-11321. As- sociation for Computational Linguisties.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020a. Mind the trade-off: Debiasing
NLU models without degrading the in- distribution performance. Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, pages 8717-8729 Association for Computational Linguistics.
Prasctya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020b. Towards debiasing NLU models from unknown biases. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing, EMNLP 2020, pages 7597-7610. Association for Computational Linguistics.
Yandong Wen, Weiyang Liu, Adrian Weller, Bhiksha Raj, and Rita Singh. 2022. Sphereface2: Binary clas- sification is all you need for deep face recognition. In The Tenth International Conference on Learning Representations, ICLR 2022. OpenReview.net.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. broad-coverage challenge corpus for sentence understanding through inference. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- HLT 2018, pages 1112-1122. Association for Com- putational Linguisties.
Yadollah Yaghoobzudeh, Soroush Mehri, Remi Tachet Combes, Timothy Hazen, Alessandro Sor- doni. 2021. Increasing robustness to spurious corre- lations using forgettable examples. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, pages 3319-3332. Association for Computational Linguistics.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 1298-1380 1308. Association for Computational Linguistics.
Zhilu Zhang and Mert R. Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances Neural Information Processing Systems 31: Annual Conference on Neu- ral Information Processing Systems 2018, NeurIPS 2018, pages 8792 8802. Curran Associates, Inc
Zhuotun Zhu, Lingxi Xie, and Alan L. Yuille. 2017. Object recognition with and without objects. In Pro- ceedings of the Twenty-Sixt International Joint Con- ference on Artificial Intelligence. IJCA/ 2017, pages 3609-3615. ijcai.org.

---

A Appendix
MNLI
HANS
Merging Method
Ent NEnt Avg
A.1 Dataset Statistics
PoE (Sanh al., 2021)
83.1
94.5 41.4 67.9
Table 6 shows data statistics for the datasets used in our experiments.
PoE. (Utama et al., 2020b)
81.9
90.9 42.8 66.8
(1) Softmax
82.7
93.5 51.8 72.6
(2) Softplus
82.7
94.0 45.2 69.6
Dataset
#samples #classes
MNLI
401k
3
Table Performances of main models evaluated
ID
FEVER
258k
3
MNLI and HANS with different strategies for merging expert predictions. We mark the best performance
QQP
399k
2
HANS
30k
2
bold.
OOD FEVER Symm.
712
3
PAWS
677
2
MNLI
HANS
Balancing Method
Ent
NEnt
Avg
Table Detailed dataset statistics. ID and OOD denote in-distribution and out-of-distribution data, respectively.
(1) Reweighting
82.7
93.5 51.8 72.6
(2 Over-sampling
82.7
95.3 46.6 71.0
Under-sampling
81.7
94.5
48.0
71.2
(4) w/o Balancing 82.7
94.6
44.2 69.4
A.2 Details of Preliminary Experiments
this section, we present the details of the pre- liminary experiments discussed in Section 2. We collect minsed/bias-comfiii
Table 8: Performances of main models evaluated on MNLI HANS with different strategies for balanc- ing positive/negative samples. We mark the best perfor- mance in bold.
examples from the training set of MNLI and FEVER. For MNLI, we select examples where all the hypothesis words oc- cur in the premise in terms of exact match. Among those selected examples, classify examples with the entailment class as biased and others as bias- conflicting. For FEVER, we select examples where any of the Top-10 LMI-ranked bigrams, listed in Schuster et al. (2019), appear in the claim also in terms of exact match. We then classify the exam- ples with the refute class biased and others as bias-con bias-confficting..
classes except the target class non-target class. In this analysis, alleviate this problem, com- pare three widely used strategies for mitigating the imbalance class problem: (1) reweighting, (2) over- sampling, and (3) under-sampling. Table 8 shows the accuracy of the main models on MNLI and HANS. All three methods, the reweighting, over- sampling, and under-sampling methods show better performance than the method without applying the imbalance mitigation method (i.c., w/o Balancing). Also, when using reweighting, the performance of the main model is the highest.
A.3 Effect of merging method
We use as many bias experts as the number of classes in multi-class classification, thus we need to merge their predictions into one probability dis- tribution to train the main model in PoE manner. We consider two different strategics for merging predictions from different bias experts: (1) softmax and (2) softplus. From Table 7, we observe that both strategies show higher duttof-distributton per- formances compared to the vanilla PoE baseline. particular, merging with softmax outperforms merging with softplus: 72.6% vs 69.6% on HANS, This suggests that applying softmax is best suited for our debiasing method.
A.5 Qualitative Results
Table illustrates examples to show how our pro- posed framework improves the identification per- formance of biased examples. The examples show that bias experts trained with the binary learning objective make predictions with higher confidence, compared to the weak learner in Sanh et al. (2021), the biased model trained with the multi-class learn- ing objective. This indicates that each bias expert individually learns the bias attributes (e.g.. high lexical overlap, or the presence of negation words such as "no" and "never") of the target class without being affected by the confidence of other classes.
A.4 Handling class imbalance
The OvR approach we introduced causes an im- balance between classes because classifies all

---

Bias Attribute
Weak
Bias
Premise
Hypothesis
Class
Learner
Experts
Contradiction
18.9%
7.2%
We have Kroger but not Skaggs.
We have Kroger but no Skaggs.
Entailment
75.6%
91.4%
Lexical overlap
Neutral
5.5%
1.4%
Contradiction
28.9%
17.2%
And no surprise, the bugger was already inside.
The bugger was inside.
Entailment
66.2%
81.5%
Neutral
4.9%
1.3%
Contradiction
64.2%
83.6%
I didn't hear what Mr. Inglethorp replied.
Mr. Inglethorp said anything back.
Entailment
5.3%
1.6%
Presence
Neutral
30.5%
14.8%
of Negation
Contradiction
65.4%
86.2%
That's the only way to keep you from being punished.
There's no way to keep you from being punished.
Entailment
29.7%
12.7%
Neutral
4.9%
1.1%
Table 9: Qualitative comparison of the models' confidence. We compare bias experts with the weak learner used in Sanh et al. (2021). The bold class denotes the true label.
FEVER
Ablative Setting
dev
symm.
Bias Experts (ours)
85.6
68.1
(1) w/o Amp.
85.1
66.9
(2) w/o OvR
84.9
66.0
(3) w/o Amp. and OvR
84.8
65.7
Table 10: Ablation studies using accuracy FEVER
and FEVER Symmetric. We mark the best performance in bold.
A.6 Additional Ablation on Fact Verification
To further study the efficacy of each component in method, conduct additional ablation stud- ies on FEVER and FEVER Symmetrie datasets. The results are shown Table 10, and the abla- tive settings are the same as those Section 4.5. As shown in the table, using both OvR and Amp. obtains the best out-of-distributiin performance, demonstrating the importance of both components. In addition, OvR contributes more to improving the out-of nut-of-distribution performance, compared to bias amplification. These results are consistent with those reported in Table


Reviews:
[{"review_id": "QaAc0RN2ZY", "reviewer": "Reviewer_JToF", "summary": "", "questions": "While the code for the experiments is easily accessible, I wonder whether there is an easy way of integrating your idea into existing systems. Right now, the ease of access outside of your experiment is limited. ", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reason to reject 1.** The trade-off between out-of-distribution and in-distribution performances could be a concern.\n\n**Answer.** We agree that the trade-off could be a concern in debiasing works, as in previous works. However, our method achieves the best trade-off between out-of-distribution and in-distribution performances. The results are shown in the table below. Specifically, the proposed method shows the largest OOD (HANS) performance improvement at the expense of the same or less ID (MNLI) performance.\n|                          | ID (MNLI) | OOD (HANS) |\n|------------------------|:---------:|:----------:|\n| BERT-base                |    84.5   |    62.4    |\n| PoE (Utama et al., 2020) |    81.9   |    66.8    |\n| PoE (Sanh et al., 2021)  |  **83.3** |    67.9    |\n| PoE + Ours (10k)         |  **83.3** |  **71.0**  |\n\nHere, \"10k\" in the 4th row means training the main model for 10k steps.\n\n&nbsp;\n\n**Reason to reject 2.** I also see the memory usage limitation, but without knowing it in detail, I would guess that applications exist where this approach is still usable and helpful.\n\n**Answer.** Yes, there could be a memory usage limitation (lines 219-225) since we need as many bias experts as the number of classes of the task. However, most NLU tasks, such as fact verification and natural language inference, have a moderate number of classes (2~20, See footnote 2), and the bias experts are used only in training. We demonstrate the applicability of our method to various NLU tasks in the paper.\n\n&nbsp;\n\n**Question 1.** While the code is easily accessible, the ease of access outside of your experiment is limited.\n\n**Answer.** Thanks for the comment. We will revise the code to easily integrate the proposed method into existing systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper focuses on a critical and timely issue in Natural Language Understanding (NLU) - debiasing models. The topic is of significant interest given the increasing awareness of biases in machine learning models and their potential societal implications.\n\n2. The paper introduces a method of learning debiased classifiers with the help of biased committees. This approach seems to be a novel contribution to the field, offering a different perspective on tackling biases.\n\n3. The paper evaluates the proposed model and baselines on three NLU tasks with out-of-distribution evaluation sets: Natural Language Inference (NLI), fact verification, and paraphrase identification. Such a comprehensive evaluation provides a robust understanding of the model's performance across different tasks.\n\nThe paper offers substantial reproducibility, is thoroughly edited, and contains a valid and important ethical statement. \n", "reasons_to_reject": "The paper acknowledges that their method's high performance on out-of-distribution data (new or unseen data) comes at the expense of in-distribution performance (data it was trained on). This trade-off could be a concern, especially if the in-distribution performance drop is significant. I also see the memory usage limitation, but without knowing it in detail, I would guess that applications exist where this approach is still usable and helpful. \n\n", "questions_to_authors": "While the code for the experiments is easily accessible, I wonder whether there is an easy way of integrating your idea into existing systems. Right now, the ease of access outside of your experiment is limited. ", "paper_topic_and_main_contributions": "This document discusses a debiasing framework for natural language understanding (NLU) models. It addresses the issue of biases in datasets that can lead to high performance on in-distribution data but poor performance on out-of-distribution data. The proposed framework introduces bias experts and binary classifiers trained between the main and auxiliary models. These bias experts bridge the gap between the auxiliary model's learning objective and purpose, resulting in improved performance on various challenge datasets.\nKey points are:\n\n1. Biases in datasets can cause NLU models to perform well on certain data distributions but poorly on others.\n2. Previous debiasing methods down-weight biased examples identified by an auxiliary model trained with explicit bias labels.\n3. The proposed framework introduces bias experts, and binary classifiers trained between the main and auxiliary models.\n4. Each bias expert is trained on a binary classification task derived from the multi-class classification task using the One-vs-Rest approach.\n5. Experimental results show that the proposed strategy effectively reduces the gap between in-distribution and out-of-distribution performance and improves performance on various challenge datasets.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 315: \"Natural language inference (NLI) is the task of determining whether a premise entails, contradicts, or is neutral with a hypothesis given a sentence pair.\" - The phrase \"neutral with a hypothesis\" might be clearer as \"neutral to a hypothesis\".\nLine 324: \"Fact verification is the task of determining whether the information in the evidence supports a claim, refutes a claim, or does not contain enough information.\" - Consider rephrasing for clarity: \"Fact verification involves determining if the evidence supports or refutes a claim, or if it lacks sufficient information.\"\nLine 303: The \"Experiments\" section could benefit from subheadings for each task (NLI, Fact Verification, Paraphrase Identification) to make it clearer and easier to follow.\nLine 335: The description of the \"Paraphrase identification\" task ends abruptly, and the reader is left without a clear understanding of the evaluation metrics or datasets used. This section should be completed for clarity.\nOverall: I feel PoE could be detailed with another sentence or two. Not everyone is familiar with it. "}}, {"review_id": "ApD9uBGFbL", "reviewer": "Reviewer_Vam3", "summary": "", "questions": "a. While the preliminary study confirms that bias experts can effectively reflect the degree of bias in samples, the relationship between these bias experts and a better-debiased model remains underexplored. For instance, does a \"more biased\" auxiliary model lead to a more robust main model? From my perspective, a more biased auxiliary model might introduce bias in the opposite direction to the main model. Consequently, the debiased model may inaccurately predict biased test samples due to potential excessive penalization, adversely affecting performance on in-domain test sets.\n\nb. Some recent comparison methods published after 2021, such as Ghaddar et al., 2021, and Lyu et al., 2023, are absent in the current study.\n\nc. The preliminary study includes experiments on biased/bias-conflicting examples. However, I couldn't find any description regarding how these examples were obtained. It would be helpful to include information about the methodology used to gather these examples for the experiments.\n\n[1] Ghaddar, A., Langlais, P., Rezagholizadeh, M., & Rashid, A. (2021). End-to-end self-debiasing framework for robust NLU training. ACL/IJCNLP (Findings) 2021\n\n[2] Lyu, Yougang, et al. \"Feature-level debiased natural language understanding.\" AAAI 2023", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1.** The motivation should be further verified. The relationship between these bias experts and a better-debiased model remains underexplored. For instance, a more biased auxiliary model might introduce bias in the opposite direction to the main model. Consequently, the debiased model may inaccurately predict biased test samples, adversely affecting performance on in-domain test sets.\n\n**Answer.** Thanks for the constructive comment. The motivation of this paper is the development of a more effective \"training methodology\" for the biased model to improve the trade-off in the debiased model between OOD performance and ID performance. Developing better training methods can mitigate the ID performance degradation you're concerned about. Specifically, as shown in the attached table below, our model can achieve the same OOD performance gain at the expense of less ID performance degradation. Degradation of ID performance can be controlled in various ways, by reducing the weight of PoE loss in a multi-loss [1], or by training the biased model with fewer steps.\n\n|                          | ID (MNLI) | OOD (HANS) |\n|------------------------|:---------:|:----------:|\n| PoE (Utama et al., 2020) |    81.9   |    66.8    |\n| PoE (Sanh et al., 2021)  |    83.3   |    67.9    |\n| PoE + Ours (2k)          |  **83.8** |    68.0    |\n| PoE + Ours (Full)        |    82.7   |  **72.6**  |\n\nHere, \"2k\" in the 3rd row means training the main model for 2k steps, and \"Full\" means in the 4th row means training the main model for 3 epochs, as detailed in the section 4.3.\n\n[1] Sanh et al., Learning from others' mistakes: Avoiding dataset biases without modeling them, ICLR 2021.\n\n&nbsp;\n\n**Question 2.** Some baselines are missing, such as Ghaddar et al., 2021, and Lyu et al., 2023.\n\n**Answer.** Ghaddar et al. (2021) has been compared with our model in Table 1 of the submitted version, and is denoted by Self-Debiasing [2]. Meanwhile, we excluded DCT [3], due to the scope difference. While we focus on improving the learning of the biased model, DCT focuses on improving the utilization of the existing biased model.\n\n[2] Ghaddar et al., End-to-end self-debiasing framework for robust NLU training, Findings of ACL-IJCNLP 2021.\\\n[3] Lyu, et al., Feature-level debiased natural language understanding, AAAI 2023.\n\n&nbsp;\n\n**Question 3.** Details about the methodology for classifying biased/bias-conflicting examples in the preliminary study would be helpful.\n\n**Answer.** Thank you for the comment. We have collected biased/bias-conflicting examples from the training set of MNLI and FEVER. For MNLI, we have filtered examples where all the hypothesis words occur in the premise by exact match, then have classified the entailment class as biased and others as bias-conflicting. For FEVER, we have filtered examples where any of the Top 10 LMI-ranked bigrams (listed in Schuster et al. (2019) [4]) appear in the claim (also by exact match), then have classified the refute class as biased and others as bias-conflicting. We will attach details about the methodology for classifying examples in the camera-ready version.\n\nAdditionally, we attach a Python code named “train_bert_poe_track.py” to Github, which has been used in our preliminary study. Please see lines 713-733 of this code for the detailed implementation of classifying biased/bias-conflicting examples.\n\n[4] Schuster et al., Towards Debiasing Fact Verification Models, EMNLP 2019.\n\n&nbsp;\n\n**Missing Reference.** We will update the related work section with the missing references you pointed out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a. The introduction of a set of binary classifiers is novel in the field of debiasing.\n\nb. The experimental results provide evidence of the proposed method's superiority when compared with the listed baselines.", "reasons_to_reject": "a. The motivation should be further verified. (See question a)\n\nb. Some recent comparison methods are missing. (See question b)", "questions_to_authors": "a. While the preliminary study confirms that bias experts can effectively reflect the degree of bias in samples, the relationship between these bias experts and a better-debiased model remains underexplored. For instance, does a \"more biased\" auxiliary model lead to a more robust main model? From my perspective, a more biased auxiliary model might introduce bias in the opposite direction to the main model. Consequently, the debiased model may inaccurately predict biased test samples due to potential excessive penalization, adversely affecting performance on in-domain test sets.\n\nb. Some recent comparison methods published after 2021, such as Ghaddar et al., 2021, and Lyu et al., 2023, are absent in the current study.\n\nc. The preliminary study includes experiments on biased/bias-conflicting examples. However, I couldn't find any description regarding how these examples were obtained. It would be helpful to include information about the methodology used to gather these examples for the experiments.\n\n[1] Ghaddar, A., Langlais, P., Rezagholizadeh, M., & Rashid, A. (2021). End-to-end self-debiasing framework for robust NLU training. ACL/IJCNLP (Findings) 2021\n\n[2] Lyu, Yougang, et al. \"Feature-level debiased natural language understanding.\" AAAI 2023", "paper_topic_and_main_contributions": "The central aim of this paper is to address data bias by introducing Bias Experts, which are a set of binary classifiers. The authors conducted a preliminary study to validate that the predictions made by the Bias Experts can more accurately indicate whether an example is biased or not. By simultaneously training the Bias Experts with the main model using PoE, the proposed method achieves improved model performance on biased test sets. The experimental results substantiate the effectiveness of this approach. However, further verification of the motivation is warranted, and some recent comparison methods are missing.\n\n------ Post Rebuttal ------\nThe authors' response resolved my concerns, prompting me to raise the soundness score from 3 to 4.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Ghaddar, A., Langlais, P., Rezagholizadeh, M., & Rashid, A. (2021). End-to-end self-debiasing framework for robust NLU training. ACL/IJCNLP (Findings) 2021\n\n[2] Lyu, Yougang, et al. \"Feature-level debiased natural language understanding.\" AAAI 2023"}}, {"review_id": "rIpVqnjXQS", "reviewer": "Reviewer_R64f", "summary": "", "questions": "Any reason to not have the auxiliary model used to get the weights for 3.2 not be also be one-vs-all?\n\nWhat model is being used for the image classification experiments? The appendix at least should include some additional details.\n\nThe explanation of why one vs. all bias-only models are better in 080-090 was not very clear to me. Presumably it is due to some kind of regularizing effect in multi-class models that reduces bias but I don't quite have an intuition on what that would be. Is there any other intuition that would be make things more clear?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reason to reject 1.** Introduces several hyper-parameters, which is always problematic in de-biasing works since one shouldn't assume there is a validation set of non-biased to hyper-parameters since using such a set would \"leak\" information about the bias. The work also does not discuss how the parameters are tuned for their experiments.\n\n**Answer.** Yes, we agree that introducing several hyper-parameters is problematic in debiasing works. With this in mind, we have sought to minimize the number of hyper-parameters in our work. Specifically, there are three additionally introduced hyper-parameters, $\\lambda_{1}$,  $\\lambda_{2}$, and $\\alpha$. We have introduced $\\lambda_{1}$ and $\\lambda_{2}$ to mitigate the class imbalance problem aroused by OvR. Following [1], we set $\\lambda_{1}=(k-1)/k$ and $\\lambda_{2}=1/k$, where $k$ denotes the number of classes. Thus, $\\lambda_{1}$ and $\\lambda_{2}$ are not involved in the tuning process.\n\nIn addition, compared with previous works, the performance of the proposed method is less sensitive to the hyperparameter $\\alpha$ introduced to control the degree of bias amplification. Specifically, in HANS, the performance of our method varies from 70.0 to 72.7 (±0.99) depending on the hyperparameter value. On the other hand, the performance of [2] ranges from 66.5 to 72.0 (±1.70, Figure 5), the performance of [3] ranges from 64.9 to 68.3 (±1.10, Table 6) depending on the hyperparameter value.  In addition, [4] shows that the performance of the model varies from 34.0 to 57.5 in PAWS (±8.80, Figure 4), and the performance of [5] varies from 77.5 to 85.5 in CelebA (±2.60, Figure 3) depending on the hyperparameters. \n\nHowever, we still agree that introducing hyper-parameters in debiasing works, where most out-of-distribution datasets do not provide a validation set, is problematic. Therefore, in the future, we plan to investigate how to tune hyper-parameters without leaking information about the bias. We will also add a discussion regarding this issue in the final version.\n\n[1] Wen et al., SphereFace2: Binary Classification is All You Need for Deep Face Recognition, ICLR 2022. \\\n[2] Sanh et al., Learning from others' mistakes: Avoiding dataset biases without modeling them, ICLR 2021. \\\n[3] Lyu, et al., Feature-level debiased natural language understanding, AAAI 2023. \\\n[4] Utama et al., Towards Debiasing NLU Models from Unknown Biases, EMNLP 2020. \\\n[5] Kim et al., Learning Debiased Classifier with Biased Committee, NeurIPS 2022.\n\n&nbsp;\n\n**Reason to reject 2.** I am left a little uncertain how much the bias amplification vs. the OvR is contributing. The ablation helps but it is only on HANS which can be a pretty unreliable evaluation set. Ablation results on the other other tasks would help.\n\n**Answer.** Thanks for the constructive comment. Therefore, based on your comment, we conducted an additional ablation experiment on FEVER and attached the results as below for presenting more reliable results. As shown in the table, OvR consistently contributes more significantly to the OOD performance on both commonly used evaluation dataset in the debiasing domain. We will include these results in the final version.\n\n|   Ablative Setting  | FEVER | FEVER-Symmetric | MNLI | HANS  |\n|-------------------|:-----:|:-----:|:------:|:------:|\n| Bias Experts (ours) |  85.6 |  **68.1** | 82.7 | **72.6** |\n| w/o Amp.            |  85.1 |  66.9 | 82.8 | 70.0 |\n| w/o OvR             |  84.9 |  66.0 | 83.0 | 68.5 |\n| w/o Amp. and OvR    |  84.8 |  65.7 | 83.1 | 67.9 |\n\n&nbsp;\n\n**Reason to reject 3.** Overall seems like a more incremental change relative prior work.\n\n**Answer.** We believe that our work has brought significant contribution to the debiasing domain. To the best of our knowledge, while most of the previous works have been studied to improve the way in utilizing the trained biased model, the way on how to effectively train a biased model remains under-explored. That being said, they generally follow the training strategies proposed in [6,7]. Meanwhile, our work presents a novel and effective way to train the biased model and has achieved significant performance improvement on various challenging datasets.\n\n[6] Sanh et al., Learning from others' mistakes: Avoiding dataset biases without modeling them, ICLR 2021.\\\n[7] Utama et al., Towards Debiasing NLU Models from Unknown Biases, EMNLP 2020.\n\n&nbsp;\n\n**Reason to reject 4.** The evaluation approach using BERT-base is what was used in prior work, but is starting to feel like a very out-of-date setup.\n\n**Answer.** We have performed experiments on the BERT-base for comparison with previous works.\n\n&nbsp;\n\n**Question 1.** Any reason to not have the auxiliary model used to get the weights for 3.2 not also be one-vs-all?\n\n**Answer.** Our design choice of the auxiliary model is selected by the empirical observation that the change of performances of the main model is negligible when the one-vs-all approach is applied to train the auxiliary model. Specifically, when one-vs-all is applied to the auxiliary model, the MNLI performance has increased from 82.7 to 82.9, while the HANS performances have remained the same, 72.6. Thus, we adopt a multi-class classifier as an auxiliary model, which requires relatively less memory.\n\nBesides, in the case of using the probability of each bias expert itself for amplifying biased examples, the variance of the performance has been largely increased (1.47 compared to 0.77), while only showing slight improvement of the main model performance in the OOD data (73.0 compared to 72.6). Therefore, our methodology design uses the output (logits) of the auxiliary model, instead of the output from bias experts when calculating the weights.\n\n&nbsp;\n\n**Question 2.** What model is being used for the image classification experiments? The appendix at least should include some additional details.\n\n**Answer.** Thanks for your thoughtful feedback. We use ResNet-18 as a backbone network for the image classification experiments for a fair comparison with [8] and we replace the multi-class biased classifiers with biased experts. Specifically, in the BAR task, which has six classes, we assign five bias experts to each class, instead of using 30 multi-class classifiers. We will attach these details in the appendix.\n\n[8] Kim et al., Learning Debiased Classifier with Biased Committee, NeurIPS 2022.\n\n&nbsp;\n\n**Question 3.** The explanation of why one vs. all bias-only models are better in 080-090 was not very clear. Is there any other intuition that would make things more clear?\n\n**Answer.** To elaborate further, the multi-class learning objective (e.g., softmax cross-entropy) has several drawbacks in training biased classifiers: First of all, it has a competitive nature across classes [9]. In the multi-class learning objective, increasing the confidence of one class will necessarily decrease the confidence of some other classes, thus the model may predict bias-conflicting examples as biased when logits of other classes are relatively lower than the correct one. Secondly, as you mentioned above, it has a regularization effect. The multi-class learning objective could reduce the overfitting of the auxiliary model on the biased features due to the regularization effect [10]. Therefore, considering the aforementioned characteristics of the multi-class learning objectives, the one-vs-all approach is more suitable to train the biased model.\n\n[9] Wen et al., SphereFace2: Binary Classification is All You Need for Deep Face Recognition, ICLR 2022.\\\n[10] Feldman et al., The advantages of multiple classes for reducing overfitting from test set reuse, ICML 2019.\n\n&nbsp;\n\n**Missing Reference.** We will update the related work section with the missing references you pointed out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Evaluate on many well studied de-biasing tasks with positive results\n- Section 2 helps motivate the method by providing empirical evidence for the hypothesis that the multi-class bias-only model are worse at capturing bias then one-vs-rest models.\n- Ablation study helps verify the value of the individual components.", "reasons_to_reject": "- Introduces several hyper-parameters, which is always problematic in de-biasing works since one shouldn't assume there is a validation set of non-biased to hyper-parameters since using such a set would \"leak\" information about the bias. The work also does not discuss how the parameters are tuned for their experiments.\n- I am left a little uncertain how much the bias amplification vs. the OvR is contributing, I see the authors even report SoTA on QQP despite it being a binary problem. The is particularly important since I understand the bias amplification part mostly follows prior work. The ablation helps but it is only on HANS which can be a pretty unreliable eval set (e.g. \"BERTs of a feather do not generalize together: Large variability in\ngeneralization across models with similar test set performance\"  McCoy, et al.), ablation results on the other other tasks would help.\n- Overall seems like a more incremental change relative prior work\n- The evaluation approach using BERT-Base is what was used in prior work, but is starting to feel like a very out-of-date setup.", "questions_to_authors": "Any reason to not have the auxiliary model used to get the weights for 3.2 not be also be one-vs-all?\n\nWhat model is being used for the image classification experiments? The appendix at least should include some additional details.\n\nThe explanation of why one vs. all bias-only models are better in 080-090 was not very clear to me. Presumably it is due to some kind of regularizing effect in multi-class models that reduces bias but I don't quite have an intuition on what that would be. Is there any other intuition that would be make things more clear?", "paper_topic_and_main_contributions": "This paper consider the problem of de-biasing a model without prior knowledge of the bias. Similar to other works they train a bias-only model the reflects the bias and us it to de-biasing another model. They propose using a one-vs-all classifier instead of a multi-class classifier, as well as a bias amplification re-weighting scheme, will result in a bias-only model that better captures the bias.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "\"Increasing Robustness to Spurious Correlations using Forgettable Examples\" Yaghoobzadeh et al.\n\n\"Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles\" Clark et al.\n\nThere is a lot of additional work on de-biasing that has focused on image classification or VQA that is not cited, I don't think they need to be comprehensively covered but I think the covering them in a bit more detail would be good. Some examples:\n\n\"Revisiting the Importance of Amplifying Bias for Debiasing\" Lee at al.\n\n\"EnD: Entangling and Disentangling deep representations for bias correction\" Tartaglione et al.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "I know from the cited works that the idea in 270-300 is to amplify the bias, but the explanation in 3.2 appears to be completely circular since it reads like one should use knowledge of the bias to discover the bias. "}}]

Author rebuttal:

Meta-review: This paper presents a debiasing framework without prior knowledge of the bias. This is a challenging but realistic setting. In particular, they proposed to use one-against-all (i.e., multiple binary classifiers) to construct a bias-only model that better captures the bias in the dataset. 

Pros: 
- The paper focuses on a critical, challenging, and realistic problem. Specifically, several prior work in bias mitigation assumes type of biases is known but this paper does not make such an assumption. 
- The proposed approach using a set of binary classifiers to capture biases is reasonably novel.
- Experiments are sufficient to support the claim. 

Cons: 
- It is unclear whether the approach is practical given the tradeoff between in-distribution and out-distribution performance.
- Experiments on the BERT-based model are a bit out-of-date. Although it's understandable that the authors want to compare with prior work, it would be interesting to show performance on more recent LLMs.
Decision: Accept-Main