Uncertainty Calibration for Ensemble-Based Debiasing Methods
Authors: Ruibin Xiong
Yimeng Chen
Liang Pang
Xueqi Cheng
Zhi-Ming Ma
Yanyan Lan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Debiasing
Calibration
bias-only model
NLI

Abstract: Ensemble-based debiasing methods have been shown effective in mitigating the reliance of classifiers on specific dataset bias, by exploiting the output of a bias-only model to adjust the learning target. In this paper, we focus on the bias-only model in these ensemble-based methods, which plays an important role but has not gained much attention in the existing literature. Theoretically, we prove that the debiasing performance can be damaged by inaccurate uncertainty estimations of the bias-only model. Empirically, we show that existing bias-only models fall short in producing accurate uncertainty estimations. Motivated by these findings, we propose to conduct calibration on the bias-only model, thus achieving a three-stage ensemble-based debiasing framework, including bias modeling, model calibrating, and debiasing. Experimental results on NLI and fact verification tasks show that our proposed three-stage debiasing framework consistently outperforms the traditional two-stage one in out-of-distribution accuracy.

Uncertainty Calibration for Ensemble-Based Debiasing Methods
Ruibin Xiong 1,,,+ Yimeng Ch-n Liang Pang 2.5, Xueqi Cheng! Zhiming Ma and Yanyan Lan CCAS Key Laboratory of Network Data Science and Technology,
Institute of Computing Technology, Chinese Academy of Sciences
University Chinese Academy of Sciences Baidu Inc.
Academy of Mathematics and Systems Science, Chinese Academy of Sciences
'Data Intelligence System Research Center,
Institute of Computing Technology. Chinese Academy of Sciences
IIstitttee for Al Industry Research, Tsinghua University (xiongruibin1 8, chenyimengi4| /mailss ucas. ac on, (exq, Bangliang)@iit..o cn, mazm@ ac en, Lanyanyan@tsingheee edu.cn
Abstract
Ensemble-based debiasing methods have been shown effective in mitigating the reliance of classifiers on specific dataset bias, by exploiting the output bias- only model to adjust the learning target. In this paper, we focus on the bias-only model in these ensemble- based methods, which plays an important role but has not gained much attention in the existing literature. Theoretically, we prove that the debiasing performance can damaged by inaccurate uncertainty estimations the bias-only model. Empirically, we show that existing bias-only models fall short in producing accurate uncertainty estimations. Motivated by these findings, we propose to conduct calibration the bias-only model, thus achieving three-stage ensemble-basee debiasing framework. including bias modeling, model calibrating, and debiasing. Experimental results on NLI and fact verification tasks show that our proposed three-stage debiasing framework consistently outperforms the traditional accuracy.
two-stage one in
1 Introduction
Machine learning models have achieved remarkable performance on natural language understand- ing [10 7 28 and computer vision [16] [7]. However, observations have shown that these models have difficulties in generalizing well in out-of-distribution settings [30) 40; [ which limits their applications real-world scenarios. A major cause of this failure is the reliance of the model on specific dataset bias [3 For instance, McCoy et al. 23 have shown that sentence pairs with high word overlaps in MNLI easy to be classified the label 'entailment', even if they have different relations.
A growing body of literature recognizes debiasing important direction in machine learning and natural language processing (EBD) methods [15
34). Within these works, ensemble-base debiasing have caused considerable interest within the community, as shown
39
"Equal contribution.
i Work done while Yimeng Chen was interning Institute for Al Industry Research, Tsinghua University.
I Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

promising improvements on the cut-of-distributie performance. EBD methods, e.g., PoE [] DRiFt [[S], and Inverse-Reweight B9) usually adopt two-stage framework. Firstly, a biased predictor is trained based on the bias features only, namely the bias-only model. Its output is then utilized to adjust the learning target of the main model by using different ensembling strategies. Previous works are mainly limited to designing different ensembling strategies, without considering the bias-only model, which clearly plays an essential role in the whole process.
In this paper, we focus on investigating the bias-only model in the EBD methods. We theoretically reveal that the quality of the predictive uncertainty estimation given by the bias-only model is crucial for the debiasing performance of EBD methods. Specifically, we prove that the out oot-of-dee of-distribution accuracy of the debiased model is monotonically decreasing with the calibration error of the bias-only model when such error exceeds threshold Moreover, by theoretically analyzing the decline of in-distribution performance caused by debiasing, we show the existence of the case when uncertainty calibration can also mitigate such a side-effect. Empirically, we show that bias-only models employed by existing methods on both natural language inference and fact verification tasks fail to produce accurate uncertainty estimations. These findings indicate the critical role of the calibration property of current bias-only models for further improvement of EBD methods.
Motivated by the theoretical analysis and empirical study, we introduce an additional calibration stage into the previous EBD methods. In this stage, the bias-only model is calibrated with model- agnostic calibration methods to obtain more accurate predictive uncertainty estimation. Specifically, two typical calibration methods are used in this paper, i.c. temperature scaling [12] and Dirichlet calibration [19]. After that, the calibrated bias-only model is used to train the main model with off-the-shed shelf ensembling strategies. In this way, we extend the traditional two-stage EBD framework to stree-stage one, including bias Modeling, model Calibrating, and Debiasing, named MoCaD for short.
To demonstrate the effectiveness of our proposed framework, we conduct experiments on four challenging benchmarks for two NLU tasks, i.e. natural language inference and fact verification. Experimental results show that our framework significantly improves the out-of-distributis perfor- mance, as compared with the traditional two-stage one. Moreover, our theoretical results are well verified by the empirical observations in real scenarios.
Our main contributions be summarized as the following three folds.
We explore, both theoretically and empirically, the effect of the bias-only model in the EBD methods. Consequently, a critical problem is revealed: existing bias-only models are poorly calibrated, which will hurt the debiasing performance.
We propose agdessggi three-stage EBD framework to tackle the above problem.
We conduct extensive experiments on four challenging datasets for two different tasks, and experimental results show the superiority of our proposed framework as against the traditional two-stage one.
2
Related Work
Dataset Bias. Various biases have been found in different NLU benchmarks. For example, models with partial input can perform much better than majority-clan baselines in NLI and fact verification datasets [13] 27 (3 Many multi-hop hop questions
be solved by just using single-hop models in the recent multi-hop hop QA datasets (24 6) Similar phenomena have been observed in many other tasks, such as reading comprehension [18] and visual creestio--nnseriig [1] Many models have used
such superficial cues to achieve remarkable performance instead of capturing the underlying intrinsic principles in these biased datasets, leading poor generalization out-of-distribution datasets, when the relation of bias features and labels are changed [23 30 21].
Ensemble- debiasing (EBD) methods. EBD methods are kind of model-agnostic debias- ing method to reduce the reliance of models on specific dataset bias. In these methods, a bias-only
*This condition is more general when the ground truth labeling based on the signal features has certainty. Such cases exist in natural language understanding (NLU) tasks, where the ground-truth label for a sample is not unique but inherently forms a distribution, as shown by recent empirical studies (26] 25 That is why we focus empirical study on NLU tasks.

---

model is used to assist the debiasing training of the main model. Most EBD methods, e.g., PoE 8 DRiFt [15]. and Inverse-Reweight [39], can be formalized as a two-stage framework. It is commonly assumed that the dataset bias is known a-priori. In the first stage, the bias-only model is trained to capture the dataset bias by leveraging the pre-defined bias features. Then the bias-only model is used to adjust the learning target of the main model with different ensembling strategies. Recently, some works start to improve the EBD methods by exploring the bias-only models. For example, Utama et al. [35], Sanh et al. [29], and Clark et al. [9] focus on relaxing the basic assumption of many EBD methods, i.e., the dataset bias is known a-priori. They exploit different prior knowledge to obtain bias-only models, e.g., models that shallow [35 or with limited capacity [29] 9 are considered to be biased. Unlike these works, we theoretically study the essential effect of the bias-only model on the final debiasing performance and show how to improve it in the algorithm design process. Please note that some works [2 9 have been proposed to jointly learn the bias-only model and the debiased main model in an end-to-end manner. However, Since it is difficult to quantify the impact of the bias-only model in this scheme, we mainly focus on the typical two-stage methods [ 15 35 29
3 Formalization of EBD Methods
In this section, we formalize EBD methods with an introduction to some related notations. Consider general classification task, where the target is to map an input value x E X' of an input random variable X to a target label y E y of a target random variable Y. We denote features of æ that have invariant relations with the label as signal x², e.g.. the sentiment words in sentiment analysis. Conversely, features whose correlation with label Y is spurious and prone to change in the out- of-distribution setting are denoted as bias x², e.g., the length of input sentences in the NLU tasks. The corresponding random variables are respectively denoted as X and X Now suppose that on training dataset DD where (X, Y) PD x V), XB and Y are spuriously correlated. The goal of debiasing is to learn a classifier that models P X) with invariant out-of-distribbttin performance.
The following decomposition forms the theoretical basis for EBD methods: for Væ E X, with its corresponding features X xb X x²
PD = x) x Po = =²)) PY(
x
where (r is the conditional probability distribution of Y given the value of bias features X PD = x") represents the true principle would like to learn, and P> z) is the conditional distribution of Y given all features, which is usually approximated by directly applying statistical machine learning methods
the training data. This decomposition can be deduced under the constraint that X 1L X "I,,, as shown in [8 [5] 9]. We further prove that it also holds with the assumptions in [39 (See the appendix). The theoretical analysis in this paper is conducted based on
the same constraint as in 8 15.
From this decomposition, the true principle (Y can be achieved by adjusting the learning target with (Y ").. This is exactly the basic idea of EBD methods.
Most EBD methods belong to a two-stage framework. In the first stage, a bias-only model fn : X + RRI is trained to approximate [P B).. Then it is employed to adjust the learning target in a direct or indirect way. Direct methods such Inverse-Reweight [39] reweight the distribution by the inverse of the probability induced by the bias-only model to approximate the true principle. The objective function of the main model fa X + RD1 becomes:
min EX fM
p (X)
where p'(X) = (x)
p/y (X)) TT (X) (PT (x)
... (X)) (X)) denote
the uncertainty estimations, i.e. the prediction probabilities given by fB and fur respectively. Le represents the cross-entropy loss function. On the other hand, indirect methods usually utilize the output of the bias-only model to adjust the loss function of the main model, and the learning target becomes:
min fM
(X, (X)),,
(3)
3

---

where m is the normalization function, and (x)) (X) are vectors in proportion to p" p"( and px"(X) respectively. Specifically, PoE [5] directly uses the probability output, DRiFt [5] and
Sanh et al. (29 utilizes exponential of the logits output. In Learned- g variant of PoE, q'(X) is changed to (p'(X) ())))(x,, where g(X) ) is trainable gate function.
For both direct and indirect methods, by the property of the cross-entropy loss [14], the optimal main model fM satisfies pr* x Pp(Y|xX/pp. Therefore, we have p p"* & P when p" x Pp(Y [xx"), which guarantees the effectiveness of the existing EBD methods. Please note that Learned-Mixie does not satisfy this property due to the trainable gate function.
4 Analysis of the Bias-only Model
Bias-only models are critical to EBD methods, since their outputs are used to help recover the unbiased distribution. However, far too little attention has been paid to them in previous research. In this section, we theoretically quantify the effect of bias-only outputs on the final debiasing performance and empirically show the weakness of existing bias-only models.
4.1 Theoretical Analysis
According to the discussion in Section 3 the optimal main model f& induces the following condi- tional probability:
PD -x
PD,
(4)
PP Ej Po
For arbitrary x E X. we define
Y (x) = argmax,ey Pp(Y i|X = argmax PD. (Y = i|X = x). (5)
Here (x) stands for the predicted label given by the intrinsic principle, and ((x) the label prediction given by the debiased main model. With these notations, the debiasing performance defined as (xx(Po(x (X) (X)).. As the major factor related to the bias-only model is p (X). i.e. the uncertainty estimation, in the concerned quantities ((X), we investigate the effect of the bias-only model on the debiasing performance from this aspect.
be
Without loss of generality, we consider the binary classification problem with y {0, 1} and balanced label distribution. To divide and conquer, we conduct the theoretical analysis on a set of samples, where the bias-only model generates the same uncertainty estimation, i.e. Sy2 (1) {xlpš (z) 1), Vl E [0, 1]. Specifically, the quality of the uncertainty estimation of the bias-only model on f (4) can be measured by the calibration error defined as [ - PD(Y Y = O (f)) The debiasing performance on Si (l) is defined as (f E S (1)) (x) Y (=))), }), i.e. the probability of the subset of S (1) on which the main model gives the same prediction as the intrinsic principle.
The following theorem formalizes a precise result. Specifically, the debiasing performance is a monotonically decreasing function of the calibration error when exceeds deviation threshold 5(lo, 6, a). Here == minx s (ax PY x|X*) denotes the global certainty level of the true principle Po {))
Theorem 1. For any 1 E [0, 1), assume that 3lg s.t. PD = 0) E - cc + +c) when X
takes values in Stn. (I). If the calibration error [[ PD olS,,, (1)) 2 5(l,, c,a) > 0, the
debiasing performance Fo((x E S1 (z) = Y(x))) declines as || - Pp = OfS(a(0)
increases, where 6(lo,,,a) a) is a constant dependent with lo. e and a. When a < 2 + 21(1 20))-
0 S 8(1o, a, <2e, where 2E S 2o11-10+222 < J. Otherwise C 5(l, < 2e + C, where 0 C to
latt (1o+))+(--oo0))
which increases as increases.
The threshold in this theorem depends on latent constants lo, and a. Here Ly and e define the range of PP = o| B) on St2 (I). As these constants are related to the posterior characteristics of Sn, we verify the generality of such condition by empirical facts in Section 6 Note that the deviation threshold decreases as the certainty level a decreases. That means the same calibration error is more likely to exceed the threshold under smaller a, resulting in more considerable decrease in debiasing

---

1.0
1,0
Observed Clast Portion
Observed Class Portion
Gap (ECE-9.883
Gap (BCE-7.12)
0.8
0.8
Frequency
0.6
Frequency
0.6
0.4
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Predict Probubility
Predict Probability
(a) MNLI
(b) FEVER
Figure 1: Reliability diagrams of the bias-only models on MNLI and FEVER. The x-axis is the predictive probability of the bias-only model, and the y-axis the frequency. The wide blue bars show the weighted average of the observed class portion to all classes within each bin, and the narrow red bars show the gap between observed class portion and the predictive probability bias-only model.
performance. As result, the condition in Theorem I is more general and significant when the true principle I PD ) has low certainty, for example, in the NLU tasks supported by empirical
evidence in 26 25
We also theoretically analyze the effect of the bias-only model the in in-distribution performance, which is defined (x)) (Ý (X)
Y())). where Y(x)) argmax,gy IP i| x) denotes the label given by the ideal predictor on D. The result is shown the following theorem.
Theorem 2. For any X, (X) + (X) if and only if pf (x) PD(Y (x) =
Theorem 2 gives possible explanation for the decrease of in-distribution performance of EBD debiased models: the in-distribution error occurs when predictive uncertainty estimation of the bias-only model on (x) is higher than the conditional probability of (x)). That indicates that the in-distribution error is non-decreasing as the range of the uncertainty estimation of bias-only models increases. As an important case, when the bias-only model is over-confident [12], decreasing its calibration error improve both the in-distribution and outtof-distribution performance of the debiased model according to the two theorems.
To sum up, our theoretical study shows that both debiasing and in-distribution performances of the EBD methods are affected by the uncertainty estimation the bias-only models. Please note that both Theorem and 2 be generalized to multi-class scenarios, with more complex form. For simplicity, we only discuss the binary class case.
4.2 Empirical Analysis
According some recent machine learning studies, the uncertainty estimations of many widely
used machine learning classifiers are not reliable (20) 12 32 36). This indicates that the existing
bias-only classifiers may fail to produce a good uncertainty estimation, which can hurt the debiasing performance, as demonstrated by our theoretical results.
To quantify the effect, we further conduct an empirical study to demonstrate the quality of the existing bias-only models with respect the uncertainty estimation. Specifically, we experiment on two typical public datasets, MNLI and FEVER. Their experimental settings and detailed analysis can be found in Section 6.I For MNLI, we consider the syntactic bias (33 and
features to train a bias-only model, the same as in [8] For FEVER, we consider the claim-onl, bias [30 and train claim-only model the bias-only model, as 34 After that, we use the classwise reliability diagram
19 to check its calibration error based on data binning. We adopt the classwise expected calibration error [19] as a measure to quantify the quality of the uncertainty estimation, denoted as ECE for short, with its lower value indicates betler-calibrated uncertainty estimation.
5

---

Now we introduce our experimental results. The classwise reliability diagrams on MNLI and FEVER training sets are plotted in Figure[Taa and Figure respectively. For perfectly calibrated predictions, the curve in a reliability diagram should be as close as possible to the diagonal. Therefore, the deviation from the diagonal represents the calibration error. From the results, we can see that existing bias-only models suffer from inaccurate uncertainty estimation problems on both datasets.
The MoCaD Framework
To overcome the unreliable predictive uncertainty problem, we introduce calibration operation to the bias-only model, achieving a Modeling, Calibrating and Debiasing framework, named MoCaD for short. Our framework consists of three stages. Firstly, train bias-only model to model PD Secondly, we use the model-agnostic calibration methods to improve the calibration error of the bias-only model. The calibrated bias-only model is finally employed to conduct the debiasing process through the existing ensembling strategies.
5.1
Bias Modeling
In the first stage, we train bias-only model approximate IY similar to previous works (8 [5]. When the dataset bias is identified, i.c. bias features X B are known a-priori [8] [5]. the bias-only model can be obtained by only using the pre-defined XB predict label y with cross-entropy loss. For example, in NLI, many specific linguistic phenomena in hypothesis sentences such as negation are highly correlated with certain inference classes [27). In this case, hypothesis sentences are used as inputs to train an NLI model as bias-only model. When the dataset bias is unknown, 'shallow' model or 'weak" model can be built as the bias-only model, as 29
5.2 Model Calibrating
We propose to utilize model-agnostic calibration methods to improve the calibration error of the bias- only models. Specifically, two typical calibration methods, temperature scaling [12] and Dirichlet calibrator [9] are used in this paper. The calibrated bias-only model denoted as fB.
Temperature scaling is a simple-but-effectvy calibration method. It learns a single scalar parameter 'temperature' which applied the last softmax layer. Specifically, denote zx(X as the logit output of the bias-only model on sample (X, Y), abbreviated for z temperature scaling will correct the output follows: pa softmax(z) zz/ /T), where T is the temperature, which is learned with the cross-yntropy loss.
Dirichlet calibrator is derived from the Dirichlet distribution likelihood. The transformed probability computed as pd softmax (W In pn + b'), where W and b' stand for the linear transformation matrix and intercept term, which are optimized by the cross-entropy loss equipped with ODIR (Off-Diagonal and Intercept Regularisation) to prevent over-fitting [9]
Please note that temperature scaling does not change the predicted label because the maximum of the softmax function remains unchanged. In other words, only changes the uncertainty estimation and maintains the model's 's accuracy. Unlike temperature scaling, the Dirichlet calibrator can change the prediction accuracy. Empirically, we observed that the Dirichlet calibrator improves the accuracy of all bias-only models in our experiments (See the Appendix for details). In both methods, the calibration error is expected be reduced by learning the parameters with the cross-entropy loss.
5.3 Debiasing
The final step to train the main model fo with the calibrated bias-only model fB. Specifically, fe is applied with the existing ensembling strategies to make the main model fo approximate the true principle (Y by adjusting the learning target of the main model, as described in Section 3
The design of the main model is highly dependent on the concerned task, as indicated by previous works. For example, a BERT-based classifier is usually used in NI. [55, and a BottomUppToopl VQA model is usually adopted in VQA [8].

---

6 Experiments
In this section, we conduct experiments different real-world datasets answer two questions: (1) whether our proposed MoCaD framework improves the debiasing performance of the EBD methods; (2) whether the experimental results are consistent with the theoretical findings.
6.1 Experimental Settings
We describe our experimental settings, including datasets, models and some training details. More details are provided in the Appendix.
Datasets and bias-only models. We conduct experiments on both fact verification and natural language inference, which are commonly used tasks in debiasing [ 34 35]. to choose the datasets and design the bias-only models.
We follow these works
Fact verification requires models to validate a claim in the context of evidence. For this task, we use the training dataset provided by the FEVER challenge [31]. The processing and split of the dataset into training/developmen set are conducted following Schuster et al. [30 It has been shown that FEVER has the claim-only bias, where claim sentences often contain words highly indicative of the target label (3).. So the bias-only model is trained to predict labels by only using claim sentences. Finally, Tever-Symment
datasets 330 (both version and 2) are used as the test sets for evaluation.
Natural language inference aims to infer the relationship between premise and hypothesis. Recent studies have shown that various biases exist in the widely used NLI datasets (27 13 23 In this
paper, we conduct our experiments on MNLI (37 and consider both known bias and unknown bias. For known bias, firstly, we consider the syntactic bias, e.g. the lexical overlap between premise and hypothesis sentences is strongly correlated with the entailment label [23]. So the bias-only model is a classifier using hand-crafted features indicating how words are shared between the two sentences as the input, the same as that in [8). Finally, HANS (Heuristic Analysis for NLI Systems) (23) is utilized as the challenging dataset for evaluation. Then we consider the hypothesis-only bias, which means that we can only use the hypothesis to predict the relation between premise and hypothesis. So the bias-only model defined as classifier trained to predict labels by only using hypothesis. In the experiment, we still use MNLI as the training set and employ two hard MNLI datasets [13] 21 for evaluation. The 'hard' subsets are derived from the MNLI Mismatched dataset with two different strategies: (I) a neural classifier is trained on hypothesis sentences and the wrongly classified instances are treated as 'hard' instances. (2) patterns in hypothesis sentences that are highly correlated to the specific labels are extracted as surface patterns, and samples which against those surface patterns' indications are recognized as 'hard' samples. Therefore, the two challenging dataset are referred to as Hard-CD (Classifier Detected) and Hard-SP (Surface Pattern), corresponding to their creation strategies. For unknown bias, following Utama et al. [35), we build a 'shallow' model as the bias-only model, which has the same architecture the main model and is trained on a subset of the MNLI training set. Then we use HANS as the challenging dataset for evaluation as Utama et al. [3).
Baselines and configurations. We experiment with 8 implementations of MoCal, i.e. two different calibrators combined with four different ensembling strategies. The two calibrators are temperature scaling and Dirichlet calibrator, and the four ensembling strategies are those in Product-of- Experts (PoE), Learned-Mixis (LMin), DRiFt, and mnverse-Reweipe (Inv-R). We compare the performances of these implementations with their corresponding two-stage EBD methods. We denote different implementations of MoCaD by the name of corresponding EBD methods with the calibrator name as the subscript. We use TempS and Dirichler to denote the implemented methods with temperature scaling and Dirichlet as the calibrator, respectively.
In our experiments, we adopt the BERT-based classifier as the main model and follow the standard setup for sentence pair classification 110. The cross-entropy trained model (denoted as CE) is also included a baseline, to show the difference between the debiased and un-debiased model. To tackle the high performance variance on challenging datasets as observed by Clark et al. (8), we run each experiment five times and report the mean scores and the standard deviations. For each task, we utilize the training configurations that have been proven to work well in previous studies and keep the same bias-only model for all methods. For Leamed-MMxin, the entropy term weight is set to the value
7

---

Table 2: Classification accuracy MNLI.
Syntactic Bias
Hypothesis-only Bias
Unknown Bias
Method
ID
HANS
ID
Harded
Hardsp
ID
HANS
CE
84.2 + 0.2
61.2 + 3.2
84.2 0.2
76.8 i 0.4
72.6 2.0
84.2 + 0.2
61.2 3.2
PoE
82.8
68.1 3.4
83.2 0.2
79.4 +:
76.8 24
80.7 = 0.2
69.0 24
PeEns
83.9 + 0.3
69.1 22
82.9 0.3
79.6 1.4.
77.4 224
82.1 0
69.9 1.6
PiEpirichlet
84.1 0.3
70.7 1.5
8.4 0.4
19.4+00
77.6 2.1
82.3 + 0.3
70.7
DRiFt
81.8 0.4
66.5
83.5 0.4
79.5
76.3
80.2 0.3
69.1
DRiFt
83.0 = 0.4
69.7 1.8
83.1 #.2
79.6 0.2
77.4 3.3
81.5 0.3
70.0 + 0,9
Dirihhiee
83.6 + 0.3
69.8
82.8 0.3
79.6
79.0 1.6
81.9 0.6
69.4
InvR
82.5 0.1
68.4 = 1.2
83.1 0.2
78.4 0.5
77.1 2.0
78.7 = 4.8
64.7 2.6
InvR InvRremps
83.6 +
69.4 1.6
82.8 0.0.2
78.6::00
77.9
81.4 1:
65.8 0.9
Inv Diriihtet
83.7 + 0.4
69.4 + 1.3
82.5 4.0.2
78.9 0.4
80.8 2.0
81.5 + 0.2
68.2 0.888
LMin
84.1 + 0.3
65.5 37
80.5 0.3
80.0 0.4
78.2 2.0
83.1 0.3
66.5 + 1.1
LMinTemps
84.1 0.2
63.2 27
80.5 0.6
80.3 0.2
80.8 3.6
83.3 0.2
66.2
LMinpiriehlet
84.3 + 0.3
62.7 2.6
80.1 0.5
79.8 0.4
83.2 2.2
82.7 0.2
66.4 + 1.2
suggested by Utama et al. [4]. For the Dirichlet calibrator, we set = 0.06 for all experiments, based on the in-distribution performance on the development sets. sets.
6.2 Experimental Results
Now we show our experimental results answer the aforementioned two questions.
Table shows the experimental results on FEVER. We can see that for both calibrators, MoCal) outperforms the corresponding EBD methods, including Learned-Mxxin, both Fever-Symmetrric v1 and v2 datasets. Comparing different calibrators, Di chlet consistently performs better than TempS. Please note that the label distribution of the development training set on FEVER, which explains why sometimes Dirichlet obtains better in in-distribution performance than the cross-entropy loss.
different from that of the
Table shows the experimental results on MNLI with respect to known bias and unknown bias. The main results similar that FEVER, i.e. cal- ibration brings benefit to the debiasing performance, and Diri siricheet chlet obtains better results than Temps, for all EBD HANS.
Table 1: Classification accuracy on FEVER.
Method
ID
Symm. v1
Symm. v2
CE
87.1 +: 0.6
56.5 0.9
63.9 :: 0.9
PoE
84.0
62.0 1.3
65.9 0.6
PoE
82.0 + 0.9
63.3 0.9
66.4 + 0.8
methods except Learnd-Mixin indicates that for both known and un- known dataset bias, MoCaD outperforms corresponding EBD methods. Please note that, as a trainable gate function is added in Learned- model of different from others and does not fit our theoretical assumptions. Specially, the performance gap between baselines and our methods is relatively small on Hard-CD. This may due to the fact that the construction of Hard-CD is dependent on a specific biased model.
PoEpirichlet
87.1
65.9 1.1
69.1 +
DRiFt
84.2 1.2
62.3
65.9 + 0.7
DRiFt
81.7 + 0.9
63.5 1.3
66.5 = 0.7
DRiFt PRrichirichlee
87.4 1: 1.2
65.7 1.4
69.0 1: 1.3
the optimal bias-only
InvR
84.3 - 0.8
60.8 1.2
65.2 + 1.0
Inv Tempss
83.8
61.5 0.9
65,4 +: 0.7
Inv Dirichhet
87.0 1: 0.8
63.8 2.2
68.2 : 1.7
LMin
84.7 + 1.8
59.8 4.7.
65.3 + 1.1
LMin
84.9 1.7
60.0 4.5
65.6 + 1.5
LMinpirichlet
87.5 +
61.5 424
67.1 + 1.3
6.2.1 Empirical Verification of Theorem
I
Now we analyze whether the improvement of debiasing performance agrees with our theoretical study in Theorem[] [ That is, calibrated models achieve better uncertainty estimation, leading to better debiasing performance results.

---

FEVER
MNLI
Hypoahesis-olly
72
0.55
0.64
71
66
69
64
68
0.82
0.82
Symm
62
Symm V2
57
HANS
0.7
0.9
1.3
0.7
0.9
1.1
1.
Claswins-BB.
Temperstare
Figure 2: Debiasing performance of bias- only model the quality of predictive un- certainty measured by classwise- (lower is better).
Figure 3: In-distribution performance (accu- racy) of the main model vs temperature.
To facilitate the study, we demonstrate the classwise-ECE of the calibrated bias-only models on different training datasets, as shown Table the table, Un-Cal, Dirichlet, and TempS denote the bias-only model without calibration, with temperature scaling and Dirichlet calibrator, respec- tively. From the results, we can see that calibrated bias-only models on different datasets achieve better uncertainty estimation, for both calibrators. Comparing the two calibrators, the Dirichlet calibrator performs better because its higher expressive power. Further considering the debiasing that the empirical findings consist with theory.
improvement in Table and 2
Furthermore, we conduct more detailed experiment on MNLI and FEVER, regarding syntactic bias and claim-only bias respectively. Specifically, we adopt the ensembling strategy in PoE, and calibrate bias-only models with the Dirichlet calibrator and save models at different checkpoints. Then we consider the debiasing performances of bias-only models with different uncertainty estimation qualities, measured by classwise-Eos ECE. The results are plotted in Figure 2 We can see that when the classwise- ECE grows, i.e. the calibration error of the bias-only model grows, the accuracy the test set decreases, i.c. the debiasing performance drops. These results precisely prove Theorem
6.2.2 Empirical Verification of Theorem
Table 3: Classwise-ECE. of the calibrated bias-only models on different training datasets.
Theorem reveals the relation between the confidence of the bias-only model and the in- distribution error of the main model. That is, if the confidence, i.e. the uncertainty estimation of the bias-only model on the predicted label is reduced, the in-distribution error of the main model will decrease. Since the label distribution changes on the development of FEVER, we only consider the results on MNLL. From Ta-
FEVER
HANS MNLI Unknown
Un-Cal
7.11
9.83
3.01
7.41
TempS
6.23
7.70
2.38
3.07
Dirichlet 1.73
4.47
0.87
1.45
ble 2 the in-distribution performance increases in the scenario of syntactic and unknown bias and decreases in the scenario hypothesis-onle bias, for most implementations. That is because the syntactic and unknown bias-only model is over-confident, and the hypothesis-only bias-only model is confident, as shown in our Appendix. These results are accordant with our theory.
We provide detailed experiment to further explain the relationship revealed in Theorem 2 Specially, we adopt the ensembling strategy PoE. and take temperature scaling as the calibrator, because the temperature parameter controls the confidence of the calibrated model. The bigger the temperature, the less confident the obtained model. We manually set the temperature parameter from 0.7 1.5 with step 1, and record the in-distribution accuracy model with PoE. The results are plotted in Figure B It shows that when the bias-only model less confident, the in-distribution performance of the main model improves, which verifies Theorem
the development set for the calibrated bias-only
Conclusions and Future Work
This paper theoretically and empirically reveals an important problem, which ignored in previous studies, that existing bias-only models in the EBD methods coor-calibrated, leading to unsatis-

---

factory debiasing performances. To tackle this problem, we propose three-stage EBD framework (MoCaD), including bias modeling, model calibrating and debiasing. Extensive experiments on natural language inference and fact verification tasks show that MoCal) outperforms corresponding EBD methods, regarding known and unknown dataset bias. Furthermore, our detailed empirical analyses verify the correctness of our theorems. We believe that our study will draw people's s attention to the bias-only model, which has the potential to become an interesting research direction in the debiasing study. A limitation of this paper is that our empirical studies focus on NLU tasks. Further experimental results on image classification show inconsistent improvements (See the appendix). A possible reason is that image classes (e.g., birds or elephants) are less disputed than language concepts (e.g., entailment or neutral). Thus the invariant mechanism for image classification has a higher certainty, reducing the impact of calibration eπror debiasing according analysis. In the future, we plan to extend our investigations to end-to-end EBI) methods and more tasks besides NLU.
theoretical
Acknowledgments and Disclosure of Funding
This work is supported by the National Key R&D Program of China under Grants No. 2020AAA0105200, the National Natural Science Foundation of China (NSFC) under Grants No. 61773362, and 61906180.
References
[1] Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1955-1960, 2016.
[2] Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. Proceedings of the European Conference on Computer Vision (ECCV), pages 456-473, 2018.
[3] Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander Rush. On adversarial removal of hypothesis-only bias in natural language inference. NAACL HLT 2019, page 256, 2019.
[4] Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding the age of data. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 5185-5198, Online, July 2020. Associa- tion for Computational Linguistics. doi: 10. 111866///122222ac-mmaaa
URL https
//www. acl web. org/ aothology/ wtttchology/222cccom acl.c.ccc--cc.. manecl-mann... 463
[5] Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. In Advances in neural information processing systems, pages 841 1512, 2019.
[6] Jifan Chen and Greg Durrett. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4026 4032, 2019.
[7] Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced Istm for natural language inference. In Proceedings of the 55th Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pages 1657-1668, 2017.
[8] Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLE) pages 4060-4073, 2019.
[9] Christopher Clark, Mark Yatskar, and Luke Zetlemoyer. Learning to model and ignore dataset bias with mixed capacity ensembles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3031 1-305,, 2020.
10

---

[10] Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
[11] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained emns are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
[12] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. arXiv preprint
2017.
[13] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, 2018.
[14] T. Hastie, R. Tibshirani, and J.H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer series in statistics. Springer, 2009. ISBN 9780387848846. Itttps:: gwogle..ggoooooooooooooo com. hk. /ttpoo/ss..-aBBooAAAAAAAA
[15] He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132-142, Hong Kong, China, November 2019. UtLhtttss:/ //www
Association for Computational Linguistics. doi: 10.18653/v 10.18653/1//D1--111 I/D19-6115,
aclwcb. www.bb.oggaaae wttp///ncooooogoooo--l///
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Divyansh Kaushik and Zachary C Lipton. How much reading does reading comprehension require? critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015, 2018.
[19] Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in neural information processing systems, pages 12316-12326.
2019.
[20] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett editors, Advances in Neural Information Processing Systems, volume 30, pages 6402-6413. Curran As- sociates, Inc., 2017. URL Sef2ed4b7 fd2c8 www7Ffa5faBibceeeee.aaaaaaaaaaae
ngs neuripss.. wwtpp//ppep///////e/e.... ..pprr.p.......
[21] Tianyu Liu, Zheng Xin, Baobao Chang, and Zhifang Sui. Hyponli: Exploring the artificial patterns of hypothesis-only bias in natural language inference. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6852-6860, 2020.
[22] Rabech Karimi Mahabadi, Yonatan Belinkov, and James Henderson. End-to-end bias mitigation by modelling biases in corpora. In Annual Meeting of the Association for Computational Linguistics, 2020.
[23] Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristies in natural language inference. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448 2019.
11

---

[24] Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannanch Hajishirzi, and Luke Zettlemoyer. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4249 4257, 2019.
[25] Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 9131-9143, 2020.
[26] Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677- 2019.
[27] Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191, 2018
[28] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[29] Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander Rush. Learning from others' mistakes: Avoiding dataset biases without modeling them. In International Conference on Learn- ing Representations, 2021.
not /
[30] Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 3410-3416, 2019.
[31] James Thome, Andreas Vlachos, Christos Christodoulopoollss, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv: 180,00555, 2018.
[32] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 32, pages 13888-13899 Curran Associates, Inc., 2019. httLtttts//:poeeediiii www.iinnnnnnnuiirneeeeeee www.nps......m pctps////pppp//1///////// File/ Gtlee3GadBb33442da (AAe449948877777707o2o Paw-4499755c222-Paa---- .dd
[33] A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528, 2011. doi: 10. 109/CCPR.2011..0..0 1.599537..
[34] Prasetya Ajic Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Mind the trade-off: Debiasing nlu models without degrading the in in-distribution performance. arXiv preprint arXiv:2005.00315, 2020.
[35] Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing nlu models from unknown biases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7597-7610, 2020.
[36] Juozas Vaicenavicius. David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of Machine Learning Research, volume 89 of Pro- ceedings of Machine Learning Research, pages 3459-3467. PMLR, 16-18 Apr 2019. URL mir.. press/ / wws/vaicenavieius1110.. 9tml
[37] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, 2018.
[38] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
12

---

[39] Guanhua Zhang, Bing Bai, Jian Liang, Kun Bai, Shiyu Chang, Mo Yu, Conghui Zhu, and Tiejun Zhao. Selection bias explorations and debias methods for natural language sentence matching datasets. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4418 4429, 2019.
[40] Yuan Zhang. Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1298-1308, 2019.
13


Reviews:
[{"review_id": "u4_5cW1q_Rt", "reviewer": "Reviewer_CYB7", "summary": "This paper shows an increasing connection between the uncertainty estimation of the biased only models and proposed MoCaD, a three-stage EBD approach considering model calibrating.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: High. The combination of uncertainty calibration and EBD is a very novel idea and the impressive results shown in this paper validates the effectiveness of such combination.\n\nQuality: The submission is technically sound, with the major claims in Section 4 being backed up by both theoretical analysis and experimental results in Section 5. however, I do feel that the experiments section can be improved to reveal more insight into why MoCaD works well. \n\nClarity: This paper is well organized, although I struggled to remember the complicated notation system used in the paper. The authors should spend some time on improving the nomenclature even at the cost of abusing some notations, consider that various kinds of probabilities are needed in both main body and appendix. The paper does provide enough information to reproduce its results.\n\nSignificance: High. Debiasing is an increasingly important area in ML, and this work has the potential to serve as a foundational piece for future development. \n\nSome areas where improvement can be done:\n\nExperiments. From Theorem 1-2, the authors nicely connected the uncertainty calibration of bias-only models to the final performance of the debiased models, and motivated the use of calibration methods. In particular, in Section 6.2.1, the Dirichlet scaling method seems to work much better than the temperature scaling method. This is understandable, as Dirichlet scaling is more expressive, but it can also introduce some accuracy change that might hurt the debiasing model. Therefore, I would like to see  if more expressive calibration methods are indeed helpful, for example, the more expressive temperature scaling (http://proceedings.mlr.press/v119/zhang20k.html) and the nonparametric calibration <http://proceedings.mlr.press/v108/wenger20a.html>. Besides the class-wise ECE, the authors might also want to look into the confidence-based ECE (http://proceedings.mlr.press/v70/guo17a.html), which is related with theorem 2, as well as the accuracy of the calibrated bias-only model (to check if the calibration method introduces non-negligible hurt on that). Section 6.2.2 is indeed very informative and completely backs up Theorem 2, good job!\n\nThe title might be confusing. It might be interpreted to how to calibrate the uncertainty of final debiased model, while the uncertainty is in fact only calibrated for the bias-only model.\n\nSome typos, e.g., Figure 3 'Syntatic Bias'.\n\n------------After Rebuttal------------\nI have read the authors rebuttal and appreciate the new experiments comparing accuracies. This moves the scores upward. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "ePiJtn6-Std", "reviewer": "Reviewer_7E1r", "summary": "The paper reveals that uncertainty calibration of the biased model is important for debiasing performance in EBD methods. By calibrating the biased model, the proposed method improves the performance of existing methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: to the best of the reviewer's knowledge, this paper is the first to discuss calibration in EBD methods even though the topic of calibration has been studied extensively.  To some extent, the paper lacks novelty because the paper uses existing techniques without modification to improve existing EBD models. \n\nClarity: the paper is well organized and easy to follow. The method starts with the formalization of the EBD (Sec.3) method and theoretical (Sec.4.1) and empirical analysis (Sec.4.2) of the bias-only model. Then it introduces calibration techniques and the complete pipeline (Sec.5). However, there are some technical details that require further clarification. Please see the questions. \n\nQuality: the paper conducts theoretical analysis and verifies assumptions empirically. \n\nSignificance: the paper points out an overlooked problem in EBD methods. The work can inspire further investigation of calibration problems in ensemble-based debiasing methods. \n\nQuestions:\n\n1, Is the predicted label $Y(s)$ from the intrinsic model (line 136) different from the label $\\tilde{Y}(x)$ predicted by the ideal predictor (line 166)? Should they be the same? \n\n2, The global certainty level $\\alpha:=min_{X^s} max_{i\\in\\{0,1\\}} P_D(Y=i|X^s)$ (line 149) is the least confidence prediction from the true principle.  Can the author justify its role as a \"global\" certainty level? It's not clear why this quantity conveys a \"global\" view of certainty. \n\n2, Regarding Figure 3: the plot shows monotonically increasing performance with temperature. As the temperature increases,  the biased models will become more and more uncertain. At some point, this must hurt the debiasing performance because when the distribution of the biased model is almost flat, it does not provide any information to the unbiased model.  Can the authors comment on this thought? \n\n**--------------------post-rebuttal review--------------------**\n\nI appreciate the authors' response and discussion on my questions. I think the paper uncovers an overlooked problem in EBD methods and can inspire further research. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have addressed the limitations of the paper especially the inconsistent results on image data. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hour"}, "extra_text": {}}, {"review_id": "Kukex9DH3u", "reviewer": "Reviewer_P1n2", "summary": "This paper provides theoretical and empirical proof that the bias-only model used for two-step debiasing methods does not provide adequate and accurate uncertainty estimation. Via both mathematical proofs and empirical experimentation on two NLP tasks, the authors show that bias-only models have been over-estimated and overlooked, and that further work is needed to improve them. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a clear and interesting critique of the bias-only model used in two-step debiasing methods. \n\nSome comments that I had while reading the paper: \n\nPlease note that  some recent works [22; 9] propose to jointly learn the bias-only model and the debiased main model  in an end-to-end manner. Since it is difficult to quantify the impact of the bias-only model in this scheme, we mainly focus on the two-stage methods  == How can your methodology be applied to these kinds of methods?\n\nFor Figure 1, wouldn't it be worth discussing the difference between the two plots (and the two tasks?)\n\nLine 228-229: \"In other words, it only changes the uncertainty estimation229 and *maintains* the model’s accuracy? \n== not sure what 'remain' means\n\nWhy were fact verification and NLI the tasks chosen?\n\nIs using a hand-crafted classifier for the NLI task comparable to a data-driven one for the fact verification one? Are they equivalent and comparable?\n\nAny hypotheses why the method doesn't work as well on Learnd-Mixin on HANS? Is there a difference between that approach and others?\n\nLine 359: \"poorly calibrated\"\n\n\"Further experimental results on image classification show inconsistent improvements\" == maybe it depends on the image task? would it be worth trying tasks that are more complex and multi-modal, as opposed to simple classification? e.g. captioning or VQA?\n\nWhat are some ways in which the bias-only model can be improved? I feel like there should be a lot more discussion of this, as well as the advantage of the joint bias and debiased model approach that was not studied at all in this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The fact that the paper focusses purely on NLP tasks and shows inconsistent improvement on image tasks is worrisome for me. What if the phenomena that were observed during the emprical experimentation were intrinsic to the nature of the two NLP tasks chosen (which is actually very few)? The fact that this method doesn't apply in image domains is worth exploring and explaining.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "Fh-Nk03fTs", "reviewer": "Reviewer_x4rW", "summary": "- The paper analyzes ensemble-based debiasing (EBD) methods for training models that don’t rely on dataset biases and thus has good out-of-distribution performance\n- The theoretical analysis shows that the debiased model performance decreases monotonically as calibration error increases, and properly calibrating the model can improve not just out-of-distribution performance, but also in-distribution performance under certain conditions\n- Based on the theoretical analysis, the authors propose a modification to EBD methods, where the bias-only model is calibrated. Empirically, this improves the performance of debiased models.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper highlights the importance of calibrating bias-only models for EBD methods. Although this is not so surprising given that EBD methods rely on probabilistic estimates outputted by bias-only models, the findings (both theoretical and empirical) are novel to my knowledge.\n- The theoretical analysis clearly illustrates the benefits of calibrating the bias-only model both for out-of-distribution and in-distribution model, and this directly motivates the proposed modification to the EBD methods. The modification (adding a calibration step) is simple, but well-motivated, practical, and novel to my knowledge.\n- The paper includes thorough, systematic experiments showing that the proposed modification improves the debiased model performance, in line with the theoretical analysis. The empirical improvement in performance is small, especially relative to the empirical gains yielded by the standard EBD methods, and this could make the key takeaway of the paper limited in its impact. However, the empirical gains are consistent across multiple datasets, calibration methods, and debiasing methods. They also have additional empirical analysis, which are also consistent with the theoretical results.\n- The paper is clearly written, and I enjoyed reading it! \n\nMinor comments\n- For Theorem 1, what kind of values should I expect for the threshold? I understand it depends on latent constants, but it’d be nice to get a rough range.\n- It took me some time to parse some of the theoretical notation and statements. For example, I found the notation for debiasing performance in Theorem 1 unclear.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the uncertainty calibration properties of the bias-only model in ensemble-based de-biasing methods. The authors argue that the accurate estimation of uncertainty in the bias-only model has been overlooked in prior research, but can have an important influence on the overall de-biasing process. The method proposed in the paper applies calibration methods to the output of the bias-only model. Thus, from a novelty perspective, the method itself is a novel combination of existing approaches, while at a conceptual level the paper is addressing a previously overlooked issue and thus has stronger novelty at this level. The reviewers judged the approach and the theoretical development to be technically correct. The experiments were judged to provide adequate support for claims. The writing was judged to be clear. The reviewers indicated that all of their questions had been adequately addressed following the author response. As a result, the paper is recommended for acceptance. The authors should be sure to incorporate the discussed clarifications and updates in the final manuscript.
Decision: Accept (Poster)