Statistically Profiling Biases in Natural Language Reasoning Datasets and Models
Authors: Shanshan Huang
Kenny Q. Zhu
Venue: EMNLP 2023 Findings (2023)
Keywords: nature language processing
model robustness
bias analysis

Abstract: Recent studies have shown that many natural language understanding
and reasoning datasets contain statistical cues that can be exploited by NLP models,
resulting in an overestimation of their capabilities. Existing methods, such
as “hypothesis-only” tests and CheckList, are limited in identifying these cues
and evaluating model weaknesses. We introduce ICQ (I-See-Cue), a lightweight,
general statistical profiling framework that automatically identifies potential biases
in multiple-choice NLU datasets without requiring additional test cases. ICQ
assesses the extent to which models exploit these biases through black-box testing,
addressing the limitations of current methods. In this work, we conduct a
comprehensive evaluation of statistical biases in 10 popular NLU datasets and 4
models, confirming prior findings, revealing new insights, and offering an online
demonstration system to encourage users to assess their own datasets and models.
Furthermore, we present a case study on investigating ChatGPT’s bias, providing
valuable recommendations for practical applications.

Statistically Profiling Biases in Natural Language Reasoning Datasets and Models
Shanshan Huang
Kenny Q. Zhu" University of Texas Arlington Arlington, Texas, USA kenny. zhu0 couuuua.een edu
Shanghai Jiao Tong University
Shanghai, China
Abstract
Hypothesis:
Someone is swimming in the sea.
Recent studies have shown that many natural language understanding reasoning datasets contain statistical cues that be exploited by NLP models, resulting in an overestimation of their capabilities, Existing methods, such "hypothesis-only" tests and CheckList, limited identifying these cues and evaluating model weaknesses. We introduce ICQ (I-See- Cue), a lightweight, general statistical profil- ing framework that automatically identifies po- tential biases in multiple-choice NLU datasets without requiring additional test cases. ICQ assesses extent which models exploit these biases through black-box testing. address- ing the limitations of current methods. In this work, we conduct comprehensive evaluation of statistical biases in 10 popular NLU datasets and models, confirming prior findings, re- vealing new insights, and offering an online demonstration system to encourage users to assess their own datasets and models. Further- more, present study investigating ChatGPT's bias, providing valuable recommen- dations practical applications.
Label: a) Entailment. b) Contradiction. c) Neu- tral.
In tasks akin to Example I, humans typically rely on the logical relationship between the premise and hypothesis. Contrarily, some NLP models might bypass this logical reasoning, focusing instead on the biases embedded within the dataset and, more specifically, within the hypotheses (Naik et al., 2018; Schuster et al., 2019). These biases -usc as sentiment shallow n-grams could provide misleading cues for correct predictions.
We refer to these biases as "artificial spurious cues" when they pervade both the training and test datasets, maintaining a similar distribution over predictions. An example of such cue is a model's disproportionate dependence the word "some- one" in Example 1. These cues, when absent altered, can significantly impair a model's perfor- mance, underlining the importance of identifying them to enhance model robustness in the future.
Labelt
Label2
1 Introduction
Distribution certain
Deep neural models have made remarkable strides in a broad spectrum of natural language understand- ing (NLU) tasks (Bowman al., 2015; Wang et al., 2018; Mostafazadeh et al., 2016; Roemmele et al., 2011; Zellers al., 2018). These tasks often em- ploy multiple-choice framework, as illustrated in Example However, the inherent sensitivity of these models to minute variations calls for a robust and precise evaluation mechanism (Jurafsky et al., 2020).
Training Data
Possible distribbtion
Test Data
not
not
Example Natural language inference in the SNLI dataset, with the correct answer bolded.
Figure 1: Example of cue.
Premise:
swimmer playing in the surf watches low flying airplane headed inland.
To tackle the issue of cues, it's crucial to distin- guish between cues embedded in the dataset and
"The corresponding author.
those learned by the model. Conventional bias

---

detection and mitigation tools, such as the Al Fair- ness 360 toolkit (Bellamy et al., 2018), primarily
We execute comprehensive evaluation of sta- tistical bias issues across ten popular NLU datasets and four models, corroborating previ- ous findings and unveiling new insights. We also offer an online demonstration system to showcase the results and invite users to evalu- ate their own datasets and models.
target dataset biases, inadequately addressing those learned by models during training.
While existing methods like "hypothesis-only" tests and Lisc bilitics, they're re not expressly designed to identify model-learned cues. "Hypothesisoonly" tests can highlight dataset issues where the hypothesis alone can provide a correct answer but fail to realisti- cally depict the model's capabilities as they don't tt evaluate the model using the full data context used during both training and prediction.
uncover model vulnera-
Through a case study, we delve into how Chat- GPT learns potential biases, offering valu- able recommendations for its practical appli- cations.
Preliminary
Drawing from the tenets of black-box testing in software engincering, CheckList scrutinizes model weaknesses without detailed knowledge of the model's s internal architecture. It achieves this by de- livering additional stress test cases premised on pre- defined linguistic features. However, CheckList's dependence on meticulously designed templates limits its scope, and it also falls short of illuminat- ing the knowledge the model has actually gleaned from the data.
2.1 Task Definition
We define an instance x of an NLU task dataset X
as
x = (p, 1) X,
(1)
where p is the context against which to do the rea- soning (p corresponds "premise" in Example I); the hypothesis given the context p; E is the label that depicts the type of relation between and h. The size of the relation set L varies with tasks.
To address these limitations, we introduce ICQ ("I-see-cue"), resilient statistical analysis frame- work engineered identify model-learned cues. Diverging from traditional methods, ICQ identifies biases in multiple-choice NLU datasets without ne- cessitating additional test cases. Employing black- box testing, ICQ assesses how models utilize these biases, delivering a comprehensive understanding of the bias in NLU tasks.
2.2 Linguistic Features
As demonstrated in previous work (Naik et al.. 2018; Jurafsky et al., 2020), we consider the fol- lowing linguistic features:
Word: The existence of specific word in the premise or hypothesis of a dataset instance.
We authenticate ICQ's efficacy by deploying on various NLU datasets to probe potential cues learned by models during training. ICQ facilitates an in-depth understanding of how models like Chat- GPT learn potential cues, and it offers illustrative examples to guide the selection of suitable prompts, providing invaluable guidance for model optimiza- tion.
Sentiment: The sentiment value of an instance, calculated as the sum of sentiment polarities of individual words.
Tense: The tense feature (past, present, future) of an instance, determined by the POS tag of the root verb.
Negation: The existence of negative words (e.g., (c.g.."no", "not", or'never") in an instance, deter- mined by dependency parsing,
In summary, this paper contributes the following:
Overlap: The existence of at least word (ex- cluding stop words) that occurs in both the premise and hypothesis.
We unveil ICQ, a lightweight yet potent method for identifying statistical biases and cues in NLU datasets, proposing simple and efficient tests to quantitatively and visually evaluate whether a model leverages spurious cues in its predictions.
NER: The presence of named entities (e.g., PER, ORG, LOC, TIME, or CARDINAL) in an instance, detected using the NLTK NER toolkit.
Typos: The presence of least typo in in- stance, identified using a pretrained spelling model. For multiple-choice datasets, all features except Overlap are applied exclusively to hypotheses.
'The code and dataset are available at https:// github. cachuu..cmm.. cwm/ hcem/ELora33//1c Phttos://chat.opgee openai..co

---

3 Approach
(MSE) and Jensen-Shannon Divergence (JSD) (Lin, 1991). The cueness score indicates the extent which dataset X biased against feature
The ICQ framework, depicted in Figure 2, consists three phases: data extraction, cue discovery, and model probing. In the data extraction phase, in- stances containing a specific linguistic feature are extracted from the dataset. The cue discovery phase identifies potential cues among pre-defined features. Finally, the model probing phase conducts two tests: the "accuracy test" and "distribution test". We will discuss these phases in more detail below.
MSE(F) = [ Σ y T
(2)
Here, Y4 represents the number of instances with label I; in the extracted dataset F, and yi is the mean number of instances for each label. A larger M SEE(E ()) implies and greater bias. If the extracted training set (Ry) and the extracted test set (Sy) exhibit similar biases, the JSD between their distributions will small:
pointed label distribution
R
Cue discovery
JSD (Q(R)) ))) A) + (Q(s,) 1), (3)
Distribution test
Rf
Flatten
18r
Saj
where A
The function
Q( denotes the label distribution of the extracted dataset. We define the cueness score
M
M
MSF(R,) cue(S, x) eXp(JSD(RR,,S))
Stress data
(4)
Accurucy
This cueness score represents the degree to which dataset X is biased against feature
Figure 2: ICQ Workflow. ①: data extraction phase; 2: cue discovery phase; 3: model prohing phase. f=a specific feature. R-training data, S=test data, =e=extracted training data, Sy=xxtracted test data, Suy=rimannigg test data without feature Sy=flatten test data, M=a specific model.
3.3 Model Probing Phase
In the previous section, we established that a dataset X could be influenced by cue f. However, model trained on this dataset may not necessarily exploit that cue, as models depend on both data and architecture. In this section, introduce framework to probe any model instance trained the biased dataset, assessing if it utilizes cue and to what degree. We achieve this through two tests: the accuracy test and the distribution test.
3.1 Data Extraction Phase
After defining the linguistic features, our system's fundamental step is constructing a data extractor for each feature value f. An extractor processes dataset and retrieves set of instances associated with the specified feature value. More specifically, for specific feature, if instance contains that feature, then that instance will be singled out as part the subset bearing that feature.
3.3.1 Accuracy Test
The accuracy test examines the model's perfor- mance on data subsets with and without specific features. By comparing the model's ' accuracy on these two subsets, we can understand the model's generalization capabilities under different condi- tions. If the model's performance shows notice- able improvement the subset containing spe- cific feature, it may suggest that the model has exploited that feature for prediction.
3.2
Cue Discovery Phase
For each feature /. we apply its extractor to both the training and test data of dataset X, denoted R and S in Figure 2. This results in clustered subsets of training instances (Ry) and test instances (Sy). A feature is considered a possible cue for dataset only if is present in both the training and test data.
In the accuracy test, we assess the prediction ac- curacies of the model on the extracted test set (with feature f) and on the remaining test set (with- out feature f), denoted as ace(Sy) and acc(Snf).
The bias the label distribution for an extracted set computed using the mean squared error

---

A positive negative value of AAce indicates the direction of the model's performance change when comparing its accuracy data subsets with and without specific features. A positive value suggests that the model exploits the feature for pre- diction, while a negative value implies struggles to generalize or detrimental sensitivity to the feature. respectively. The accuracy test computes the differ- ence between these two accuracies:
the impact of feature distribution changes on model performance, while accuracy testing evaluates the model's performance on data subsets with and with- out specifie features. By combining these two test- ing methods, a model's 's sensitivity to particular fea- tures can be more comprehensively assessed. If both tests determine that the model is sensitive to certain feature, can have higher degree of confidence in this conclusion.
AAce(f) acc(S)) ac)(Sn))
(5)
Evaluation
We first present the experimental setup, followed by results on cue discovery, model probing, and analysis. The entire framework is implemented an online demo
The magnitude of the absolute value of A Acc reflects the degree to which model's performance is affected by the presence absence of specific features in the data subsets. larger absolute value indicates a stronger reliance on or sensitivity to the feature, whereas a smaller absolute value suggests a more robust model that is less affected by the presence or absence of the feature.
4.1 Setup
We evaluate this framework 10 popular NLR datasets in Table and 4 well-known models, namely FASTText (FT) (Joulin er al., 2017), ESIM (ES) (Chen et al., 2016), BERT (BT) (Devlin et al., 2018) and RoBERTA (RB) (Liu et al., 2019) on these datasets. All these datasets except for SWAG (Zellers et al., 2018) and RECLOR (Yu et al., 2020) are collected through crowdsourcing. SWAG is generated from an LSTM-bbsed language model. Specifications of the datasets are listed in Table 1.
3.3.2 Distribution Test
The distribution test is a visual examination that focuses on how changes in specific feature distri- butions within datasets affect model's predictive performance.
First, we create a "stress dataset" Sy by "flatten- ing" the label distribution in Sy. We achieve this by removing random instances from all labels except for the one with the smallest number of instances, stopping when balance is reached. In other words, we retain the minimum number of instances present in each label, while randomly discarding the excess instances. This approach effectively eliminates bias in the extracted test set, and challenges the model. Next, we apply the model to the stress test set and obtain prediction results. We then compare the label distribution of the prediction results on the stress test set with the label distribution of the extracted training data (Ry). The rationale is that if the extracted training data contains cue, its label distribution will be skewed towards a specific label. the model exploits this cue, it will prefer to predict that label as much as possible, amplifying the skewness of the distribution, despite the input test set being neutralized. We aim to observe such amplification in the output distribution to identify the model's weaknesses.
Dataset
Type
Size
Train/Test
Acc
Racio
(%)
SNLI
CLS
570K
56:1
80.0
QNLI
CLS
11k
19:1
80.0
CL.S
ROC
MOQ
3.9k
1:1
100.0
MCQ
SWAG
MCQ
113k
4:1
88.0
RACE
MCQ
100K
18:1
RECLOR
MCQ
6k.
9:1
63.0
ARCT
MOQ
2K
79.8
ARCT
MCQ
Table 1: Datasets. Data size refers to the num- ber of questions in each dataset. CLS-Classification. MCQ=Multiple Choice Questions. By our definition, k-way MCQs will be split into & instances in prepro- cessing.
These datasets can mainly be classified into two types of tasks. SNLI, QNLI, and MNLI (Williams et al., 2018) ROC, COPA (Roemmele al., 2011), SWAG, RACE (Lai al., 2017), RECLOR, ARCT (Haber- nal et al., 2017) and ARCT (Schuster et al., 2019) are multiple-choice reasoning tasks. Fea- tures appearing a minimum of five times in training
classification tasks, while
In summary, the accuracy test and distribution test are related in terms of assessing a model's sensitivity to specific features, but they emphasize different aspects. Distribution testing focuses on

---

testing sets are considered cues.
influenced by these features, with larger values sug- gesting stronger reliance or sensitivity and smaller values indicating a more robust model.
4.2 Cues in Datasets
In this section, we showcase the cues identified in each dataset using the cueness metric, as described in Section 3.
The bottom of Table 2 shows that, across all 10 datasets, the sum of the absolute values of A fol- lows the order: RoBERTA BERT < ESIM FastText. This is consistent with earlier hypothesis- only tests and the community's s common perception of these popular models. However, examining indi- vidual datasets and features reveals a more nuanced situation. For instance, Fast tends to pick up individual word cues rather than semantic cues, while more complex models such as BERT and RoBERTA appear more sensitive to structural fea- tures like NEGATION and SENTIMENT, which are actually classes of words. This is well explained by FastText's 's design, which focuses more on mod- eling words than syntactic or semantic structures. Interestingly, FastText exhibits strong negative correlation with TYPO. We speculate that FastText might have been trained with a more orthodox vo- cabulary, making less tolerant of typos in the text.
We first filter the training and test data for each dataset using all the features defined in this paper. The left half of Table 2 displays the top 5 cues dis- covered for each of the 10 datasets, along with their cueness scores. ARCT_an __dv,, an adversarial dataset, is intentionally well-balanced. Consequently, only found one cue, OVERLAP, with very low cueness score. This is unsurprising since OVER- LAP the only "second-order" feature in our list of linguistic features that considers tokens in both the premise and hypothesis, and likely evaded data manipulation by the creator.
Mostly, the top 5 cues discovered are word fea- tures. However, besides OVERLAP, we also see NEGATION and TYPO appearing in the lists. In fact, SENTIMENT and NER features would have emerged if we expanded the list to the top 10. In- terestingly, several features previously reported as biased by other works, such as "not" and NEGA- TION in ARCT, "no" in MNLI and SNLI, and "like" in ROC, are also found. Particularly in MNLI, all five discovered cues are related to negatively toned words, suggesting significant human artifacts in this dataset that can lead to model fragility.
Hounal
ga
Cue "no" MNLI
Cue "speaking" in SNLI
Additionally, we observe that some word cues are indicative of certain syntactic, semantic, or sen- timent patterns in the questions. For example, "be- cause" in SNLI implies cause-effect structure; "like" in ROC indicates positive sentiment; "proba- bly" and "may" in RACE suggest uncertainty, and so on. These features can serve as clues for revising datasets.
5x
2-
Modile
Cue "above" in ARCT
Cue "threw" in ROC
4.3 Biases in Models
Figure 3: Four test examples for distribution comparison with different models
To investigate whether model is affected by specific cue or feature in dataset, we train four models on their original training sets and evaluate them using accuracy and distribution tests.
Distribution Test: We highlight three interest- ing findings in Figure 3. The bars for the four models represent the distribution percentage based each predicted label. Ry denotes the extracted training data distribution with specific feature. We observe that all models on the cue "no" in MNLI achieve positive A Table 2, particularly Fast Consistent with the "Accuracy Test," we find that the prediction label distribution skewness
Accuracy Test: The results are presented in Table 2. As mentioned in Section 3. posi- tive or negative A value indicates the direction of model's performance change when comparing its accuracy on data subsets with and without spe- cific features. The absolute value of A reflects the degree to which the model's performance is

---

Dotaset
Top Cues
Cueness
FT
BT
RB
%
(A)
(A)
(A)
(A)
"sleeping"
13.95
30.3
6.81
5.34
4.87
"no"
13.33
18.09
3.32
2,05
2.6
SNLJ
"hocause""
9.24
18.89
4,88
5.11
4.31
"friend"
"movic"
7,73
16.64
0.06
9,47
-0.19
"dioxide"
4.52
9.78
-0.06
4,97
10.56
4.26
QNLI
"kllometee"
4.24
4.85
6,43
4.67
2,55
"mile"
3.95
7.16
15.64
1.65
6.65
"newcastle"
3.8
3.44
0.89
-1.23
"never"
10.4
29.15
26.41
9.86
10.6
8,98
3.32
MNLI
"nothing"
8.98
25.5
26.84
5.11
4.32
"any"
6.79
20.4
19.39
7.76
3.74
"anything"
"threw"
12.99
1.28
4,69
10.88
0.97
"now"
8.68
-10.01
14.51
1.75
5.60
"found"
"woe"
7,71
2.43
0.74
1,05
5.51
"wont"
3.61
-0.83
7.92
1.04
"goc"
2.74
5.45
-9.89
-12.22
10.3
"Ioe"
"with"
1.38
-15.64
-6.98
3.3
13.82
TYPO
0.84
12.46
2233
3.8
-8.22
"football"
7.38
1,55
"ansious"
6.65
7.55
-.67
-6.66
-1.67
SWAG
"concemed""
6.19
12.6
4.58
8:27
-5.66
"skall"
5,73
-2777
0.49
8.43
3.49
"cop"
5,01
2.79
5.3
-0.92
-0.04
"above"
13.74
8.73
-8.43
-0.22
-1.92
"b"
12.84
16.97
-..8
3.52
+3,45
RACE
11.83
15.69
-6.94
8.6
-7.6
"prohably"
2.86
"mazy"
4.2
7.75
-3.45
-6.67
-1.8
"over"
2,07
1.76
-2.44
-1.35
-4.12
-3.7
RECLOR
"explanation"
1.81
6.33
-.73
.2.76
-7.44
"proportion"
1.68
"praduce"
1.4
4.54
-2.88
-14.36
-3.7
"not"
3.74
2.54
7.45
-0.97
-11.96
NEGATION
-8.23
ARCT
"n'
2.52
10.3
5,89
9.49
4.84
"always"
2.25
-4,66
38.21
-4.35
-8.26
2,06
ARCT_adN
OVERLAP
1.96e 10
1.65
2.73
0.57
(Model weakness)
469.8
361.4
227.7
216.2
Table 2: Datasets, their top 5 cues and 4 models biases A on them.
is amplified in Figure 3 for FastText and ESIM. With the "no" cue, they prefer predict "Contra- diction" even more than the ground truth in the training data. contrast, BERT and RoBERTA moderately follow the training data. While the cue
lier for BERT, the distribution test result is in- consistent with the accuracy test: the accuracy de- viation is very high for BERT, but its prediction distribution flat. We have not encountered many such contradictory cases. However, when they do occur, in this example, we give BERT the benefit of the doubt that it might not have exploited the cue "threw".
"no" is effective at influencing the models, the cue "above" is not as successful. Figure 3 shows that the distribution of predicted results for ESIM in ARCT is entirely opposite the training data, explaining the A -.443 in Table 2 and demonstrating that models may not exploit cue even if present in the data. Similarly, "speaking" in BERT and RoBERTA can also explain their low A values, which are not shown Table 2.
Case Study
Recently, ChatGPT, a large language model (LLM) released by OpenAI, has garnered significant inter- est from the NLP community. ChatGPT, GPT-
The example of the cue "threw" presents an out-

---

Prompt 1
Prompt 1 CoT
Analyze the logical relationship between the following
Analyze logical relationship between the following
premise: 'premise)" and the hypothesis: (hypothesis)".
premise: "(rremise]" and the hypothesis: "hhypohhssis"
Determine whether hypothesis is an entailment, contrudiction, or nentral respect to the premise.
Determine whether the hypothesis is an entailment, contradiction, or nentral respect the premisc.
Let 5 think step by step.
Prompt 2
Prompt CoT
Please identify whether the premise entails, contradicts, is neutral with respect to the hypothesis, The answer should be exactly "entarlment," "contrudiction, premise: 'Ipremise]" hypothesis: "hypothesiss"
Please identify whether the premise entails, contradicts, or neutral with respect the hypothesis. The answer should be exactly "eniaiimentt" "contrudiction," or "neutral." premise: 'Ipremise]"
"neutral."
hypothesis: "hypothesis]" Let think, stop by step.
Figure 4: Prompts.
x series model, is trained through Reinforcement Learning from Human Feedback (RLHF) (Chris- tiano al., 2017) similarly InstructGPT (Ouyang et al., 2022). In this section, we investigate zero- shot ChatGPT is influenced by bias features, using a case study focused on the word "no" the MNLI dataset. We aim to compare the effectiveness of different prompts and select the best one for miti- gating bias based on single bias feature.
ond prompt is inspired by previous work (Qin al., 2023), The third and fourth prompts are created by adding "Let's think step by step" (Kojima et al.) prompt and prompt 2 with the "chain of thought" (CoT) thinking, respectively. This modification has been shown to significantly improve the perfor- mance of InstructGPT on reasoning tasks (Ouyang et al., 2022).
5.3 ICQ Results
5.1 Dataset
We selected test instances from the MNLI dataset to study the influence of the word "no" on ChatGPT's performance. The original test set has Contradic- tion: 3240, Entailment: 3463, and Neutral: 3129 instances. Instances containing "no" are distributed follows: Contradiction::22, Entailment: 38, and Neutral: 46.
Prompt
Ace ("au")
Ace (wio "no")
A.Ace
P2
75.42
74.18
1.24
P2 CUT
76.67
76.40
0.27
Table 3: Accuracy test results (%). P1=Prompt 1. P2=Prompt 2.
For the accuracy test, we used all 313 instances with "no" and an equal number of instances without "no", randomly chosen from the remaining test set, This ensures balanced evaluation of ChatGPT's performance.
60
Contraciction
50
Neutral
Eoo
30
For the distribution test, we selected 38 instances per label containing "no", resulting in total of 14 instances.
20
10
P11 007 10
5.2 Prompts
Figure 5: Distribution test results.
We have four prompts in Figure 4: The first prompt is proposed by ChatGPT itself. We ask, "What is the best prompt for the MNLI task according to you?". ChatGPT returns prompt for us. The sec-
We evaluated the model's accuracy using differ- ent prompts on instances with and without the word "no." " The results are shown in Table Pl demon- strates a negative Acc, indicating difficulty in generalizing when "no" is present. P2 exhibits a positive Acc, suggesting better generalization.
Currently, x is either 3.5 or 4, and in the subsequent experimental process, the Cha we use is hased on GPT- 3.5.

---

Adding "CoT"" to both prompts reduces bias risk. Pl CoT shows the most significant improvement in Ace ("no"), but P2 + CoT has the smallest abso- lute A Ace, indicating the least sensitivity to "no" and the lowest bias risk among the tested prompts.
classified into lexicalized and unlexicalized (Bow- man al., 2015): lexicalized features mainly con- tain indicators of i-gramm tokens and cross-ngram tokens, while unlexicalized features involve word overlap, sentence length, and BLEU score between the premise and the hypothesis. (Naik et al., 2018) refined the lexicalized classification to Nega- tion, Numerical Reasoning, Spelling Error. (Mc- Coy et al., 2019) refined the word overlap features to Lexical overlap, Subsequence, and Constituent which also considers the syntactical structure over- lap. (Sanchez et al., 2018) provided unseen tokens an extra lexicalized feature.
Besides, we analyzed the model's prediction dis- tribution Figure 5 for the different prompts on the stress test set containing the word "no" with bal- anced label distribution. Distribution test results reveal imbalances in Pl and P2 prediction distribu- tions, with PI and P2 leaning towards predicting contradictions. Adding "CoT"" mitigates these im- balances, leading to more balanced distributions. P2 CoT presents the most balanced distribution among the labels, supporting its lowest bias risk.
In conclusion, our case study, particularly when focusing on the feature "no", indicates that zero- shot ChatGPT can be influenced by bias features. The choice of prompt significantly affects its per- formance. When assessing the feature "no", P2 + CoT showed the lowest bias risk across both tests. The "CoT" strategy appears effective in reducing bias risk for this specific feature. However, it's important to note that our findings, particularly re- garding the feature "no", suggest that ChatGPT's seofrrecomeeddd prompt (PI) might not always be optimal. This underscores the importance of human intervention and ongoing exploration to op- timize performance and minimize bias risks. Future studies and conclusions would benefit from more nuanced and feature-specific analysis.
Bias calculation is concerned with methods to quantify the severity of the cues. Some work (Clark et al., 2019; He et al., 2019; Yaghoobzadeh et al., 2019) attempted to encode the cue feature implic- itly by hypothesis-only training by extracting features associated with a certain label from the embeddings. Other methods compute the bias by statistical metrics. For example, (Yu et al., 2020) used the probability of seeing a word conditioned on specific label to rank the words by their bias- ness. LMI (Schuster et al., 2019) was also used to evaluate cues and re- re-weight in some models. How- ever, these works did not give the reason to use these metries, one way or the other. Separately, (Ribeiro et al., 2020) gave a test data augmentation method, without assessing the degree of bias.
6 Related Work
Dataset filtering is one way of achicving higher quality in datasets by reducing artifacts. In fact, datasets such as SWAG and RECLOR evalu- ated in this paper were produced using variants of this filter approach which iteratively perturb the data instances until a target model longer fit the resulting dataset well. Some meth- ods (Yaghoobzadeh et al., 2019), instead of pre- processing the data by removing biases, leave out samples with biases in the middle of training ac- cording to the decision made between epoch to epoch. (Bras et al., 2020) investigated model-based reduction of dataset cues and designed an algo- rithm using iterative training. Any model can be used in this framework. Although such an approach is more general and more efficient than human an- notating, it heavily depends on the models. Unfor- tunately, different models may catch different cues. Thus, such methods may not be complete.
Our work is related to three research directions: spurious features analysis, bias calculation, and dataset filtering.
Spurious features analysis has been increas- ingly studied recently. Much work (Sharma et al., 2018: Srinivasan et al., 2018; Zellers et al., 2018) has observed that some NLP models can surpris- ingly get good results on natural language under- standing questions in MCQ form without even looking at the stems of the questions. Such tests are called "hypothesis-only" tests in some works. Further, some research (Sanchez et al., 2018 8) dis- covered that these models suffer from insensitivity certain small but semantically significant alter- ations in the hypotheses, leading to speculations that the sypothesis-only performance due to sim- ple statistical correlations between words in the hypothesis and the labels. Spurious features can be

---

7 Conclusion
Armand Joulin, Édouard Grave, Piotr Bojanowski, and Tomášs Mikolov. 2017. Bag of tricks for efficient text classification. In ECAL, pages 427-431.
Our lightweight framework, ICQ, identifies biases and cues in multiple-choice NLU datasets, illumi- nating model behaviors from statistical perspec- tive. Extensive experimentation on diverse tasks validates ICQ's efficiency in revealing dataset and model biases. Using a case study on ChatGPT, we explore its cues, offering practical guidance. ICQ advances our understanding and optimization of large language models, promoting the creation of robust, unbiased Al systems.
Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel R. Tetreault, editors. 2020. ACL 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In ICML 2022 Work- shop on Knowledge Retrieval and Language Models.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. 2017. RACE: Large-scale read- ing comprehension dataset from examinations. In EMNLP, pages 785-794.
References
Lin. 1991. Divergence measures based on the shan- non entropy. JEEE Transactions on Information The- ory, 37(1):145-151
Rachel KE Bellamy, Kuntal Dey. Michael Hind, Samuel C Hoffman, Stephanic Houde, Kalapriya Kannan, Pramay Lohia. Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, et al. 2018. Ai fair- ness 360: An extensible toolkit for detecting, under- standing, and mitigating unwanted algorithmic bias. arXiv preprint 181001110.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 1900.1197..999
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In ACL, pages -428-344..
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP, pages 632-642.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In NAACL-HLI, pages 839- 849.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bha- gavatula, Rowan Zellers, Matthew E Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. arXiv preprint arXiv:2002.0 04108.
Qian Chen, Xiaodan Zhu Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced Istm for natural language inference. arXiv preprint orXiv:1609.06038
Aakanksha Naik, Abhilasha Ravichander, Norman Sadch, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In COLING, pages 2340-2353.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg. and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Ad- vances neural information processing systems, 30.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-277 277744.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't 't take the easy way out: Ensemble based methods for avoiding known dataset biases. In EMNIP-IJCNIIP, pages 4060-4073.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpont natural language process- ing task solver? arXiv preprint arXiv:2302.066 06476.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1rXiv:1810.04880 04805.
Marco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin. and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NL. models with checklist. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2020, Online, July 5- 10, 2020, pages 4902-4912.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2017. The argument reason- ing comprehension task: Identification and recon- struction implicit warrants. arXiv preprint 118.02570.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI Spring.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias natural language inference by fitting the residual. EMNLP-IJCNLP, page 132.

---

Ivan Sanchez, Jeff Mitchell, and Sebastian Riedel. 2018. Behavior analysis of nli models: Uncovering the in- fluence of three factors robustness. In NAACI- HLT, pages 1975-1985.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. arXiv preprint arXiv:1908.05267.
Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadch. 2018. Tackling the story end- ing biases in the story cloze test. In ACL, pages 752-757.
Siddarth Srinivasan, Richa Arora, and Mark Riedl. 2018. simple and effective approach to the story cloze test. In MACC-HLI, pages 92 96.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. EMNLP. page 353.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. Associa- tion Computational Linguistics.
Yadollah Yaghoobzadeh, Remi Tachet, TJ Hazen, and Alessandro Sordoni. 2019. Robust natural language inference models with example forgetting. arXiv preprint
Weihao Yu, Zihang Jiang, Yanfci Dong, and Jiashi Feng. 2020. ReClor: A reading comprehension dataset requiring logical reasoning. arXiv preprint arXiv:2002.04326.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In EMNLP. pages 93-104.


Reviews:
[{"review_id": "R0Izt1BltJ", "reviewer": "Reviewer_cXYc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response to Reviewer Comments:\n\nFirst and foremost, we would like to express our gratitude to the reviewer for the insightful comments and suggestions. We genuinely value the feedback and strive to enhance our work accordingly.\n\n1. Computational Cost Concerns:\nWe appreciate your concern about the computational costs, particularly for models with extensive test sets. Indeed, the time required for feature partitioning is linearly dependent on the test set size. However, the process of feature extraction can be efficiently parallelized. From a temporal perspective, the testing process is swift. Regarding computational costs, we haven't introduced additional inferences to the model. Our approach fundamentally revolves around analyzing the original prediction results from diverse angles to ascertain the features that intrigue the model. This methodology does not augment the model's running costs. The primary temporal cost arises from feature extraction and test feature partitioning, which are expeditious processes.\n\n2. Distinctiveness of ICQ vs. CheckList:\nThank you for recognizing the name 'ICQ' and its uniqueness. In our introduction, we emphasized that the objective of CheckList tests is to unearth tasks the model struggles with. In contrast, our framework seeks to illuminate the root causes of model vulnerabilities. This differentiation indeed imbues our method with a distinct essence.\n\n3. Broadening the Discussion & Missing References:\nWe genuinely appreciate the guidance on the scope and missing references. We plan to venture deeper into interpretability research in a broader sense in our subsequent work. We will incorporate the examples you've pointed out to enrich our discussion and reference list.\n\nIn summary, we believe that our revisions, incorporating the constructive feedback, will further solidify the contributions of our paper. Once again, we are grateful for the invaluable insights and guidance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper is very well written and easy to follow. \n2) The authors will make their code public upon acceptance, accompanied by a demo, which will help other researchers to reproduce their findings. \n3) ICQ is a straightforward method (in a good way) that the authors motivate pretty well throughout the paper. \n4) The experimental setup is quite comprehensive, and the added case study to examine more recent LLMs is very timely. \n ", "reasons_to_reject": "1) I searched for a discussion about the computational costs of doing such an examination and didn't find one (maybe I missed one?). If I understand correctly, your methods required multiple inferences for different subsets of the test set, which is not as expensive as training the model but might be significant for large models with many test set variants. \n2) Showing some of the cues ICQ (a great name, BTW) can identify and other methods, e.g., CheckList, couldn't, would make this paper even more robust. \n3) I believe there are many attempts to mitigate this problem beyond NLU that you can discuss in this paper in order to make its scope broader (I wrote a few examples in the missing references section). ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors present ICQ, a method for automatically identifying cues that can affect model decisions. \nThe authors show their method's usefulness by applying it to multiple models and NLU benchmarks. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "EQ 3 number should be better aligned. "}, "extra_text": {"Missing_References": "A recent study has shown that models tend to use shallow patterns to match mathematical theorems and their proofs [1], I believe your method can be applied for such matching tasks as well.\nThere are multiple works showing that multimodal models tend to use textual cues and ignore visual ones [2]\n\n1. https://arxiv.org/pdf/2302.09350.pdf\n2. https://arxiv.org/pdf/2110.14375.pdf"}}, {"review_id": "oAHc67wFEe", "reviewer": "Reviewer_o4hf", "summary": "", "questions": "1. Regarding reproducibility and replicability, ChatGPT seems to keep updating. How do you make this work more reproducible? Is there a way to define ChatGPT’s version?\n2. Can this work be extended to include unknown biases as well?  (Unknown biases as defined by Towards Debiasing NLU Models from Unknown Biases (Utama et al., EMNLP 2020))\n3. How did the authors obtain the percentage of human accuracy?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "I'd like to extend my heartfelt gratitude for highlighting the unique attributes of our work. Your recognition of our novel statistical analysis framework—designed to probe biases in NLU models—and our rigorous evaluations spanning diverse datasets, notably ChatGPT, is deeply cherished.\n\nWith respect to your comments and concerns:\n\n1. On the nature of our task: We've taken care to demarcate this as a \"reasoning task\" rather than a mere classification task. This approach seeks to capture the nuanced interplay between premise and hypothesis, which we believe is pivotal to understanding bias. While the same methodology might be applied to CLS tasks, the outcomes would deviate towards highlighting \"interest cues\" over \"spurious cues\". This flexibility speaks to our framework's adaptability. I'm eager to venture deeper into this domain in subsequent studies.\n\n2. Elaborating on the \"OVERLAP\" feature: Within multiple-choice contexts, even with a uniform premise, we've honed in on the subtle shifts in the hypothesis. The OVERLAP feature signifies this intricate bond between the premise and its counterpart hypothesis. Such nuanced attributes aren't inherently predisposed to model bias. I pledge to refine and emphasize these aspects further should our manuscript be chosen for publication.\n\n3. Regarding reproducibility: Our framework, in essence, transcends the confines of any single model, including ChatGPT. Should there be interest, I can offer detailed evaluations across ChatGPT's different iterations. This, I hope, stands as testament to our method's consistent reproducibility, even if certain case studies exhibit variations. It's all about illustrating the potential.\n\n4. Turning to the matter of unknown biases: Your suggestion resonates deeply. While the current study is geared towards articulating biases, I'm concurrently exploring avenues to enhance model resilience against more elusive biases. Preliminary insights suggest data augmentation and distribution balancing as potent tools. Still, the quest for perfect interpretability amidst unknown biases remains. Your thoughts have further ignited my enthusiasm to navigate this terrain, and I'm keen to share and discuss future findings.\n\nIn wrapping up, I sincerely hope that our discourse brings clarity and reassures the committee of the merit and novelty in our work. Your feedback has been nothing short of enlightening, and I'm grateful for the guidance it offers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Simple and test-case-free statistical analysis framework that allows us to detect and quantify known biases in NLU datasets/models.\n2. Extensive experiments on a variety of datasets and models, including ChatGPT", "reasons_to_reject": "1. Accuracy test: It is a good idea to compare the accuracies on the test set with the bias feature of interest and the test set without. However, the authors did not control for the number of samples in each class. For the CLS tasks,  the results might measure the sensitivity toward a specific class also.\n\n2. A failure to quantify “OVERLAP” bias with the cue-ness score is a significant flaw of the cue-ness score metric.  Maybe a more elaborate explanation is needed to justify why the proposed metric has such flaws. The authors only state that the reason for such a low cue-ness score is that “OVERLAP” is the only feature considering both premise and hypothesis tokens. This explanation does not give me an intuition on why the cue-ness score does not detect such bias. I worry that the authors define the OVERLAP bias as “The existence of at least one word (excluding stop words) that occurs in both the premise and hypothesis.” This definition is not the same or even similar to the definition given in many NLU literature, such as Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (McCoy et al., ACL 2019) define OVERLAP as a sample where all words in the hypothesis are also in the premise. \n\n3. Presentation issue:  The way the authors present some notation is a bit confusing. For instance, y_i is the number of instances with labels y_i. How did the authors get these numbers? It could be a count of instances in the dataset or a count of instances predicted by the model. Or is it something different since it is used to calculate the MSE?", "questions_to_authors": "1. Regarding reproducibility and replicability, ChatGPT seems to keep updating. How do you make this work more reproducible? Is there a way to define ChatGPT’s version?\n2. Can this work be extended to include unknown biases as well?  (Unknown biases as defined by Towards Debiasing NLU Models from Unknown Biases (Utama et al., EMNLP 2020))\n3. How did the authors obtain the percentage of human accuracy?", "paper_topic_and_main_contributions": "The paper studies spurious correlation or “artifact spurious cues” in natural language understanding (NLU) datasets. It argues that many NLU datasets contain cues/biases that can be exploited by NLP models, resulting in an overestimation of their capabilities. The main contribution of this paper is ICQ (I-See-Cue), a simple and general statistical analysis framework that automatically identifies model-learned biases in multiple-choice NLU datasets without requiring additional test cases. The experimental results reveal and quantify the existing known “artifact spurious cues” in NLU.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n- Figure 2: “sepecific” → specific\n- Line 307: “eatures” → features\n\nSuggestion on presentation improvement: \n1. Some notations are not well defined—for example, y_i.\n2. Cue-ness score seems to be an important contribution to this work. I think the authors should use more space to explain the intuition behind each component of the cuteness score.\n3. The tables are quite hard to read. The texts are so small I have to zoom in to read them. It probably doesn’t look good if someone prints the paper out and reads it."}}, {"review_id": "5j0fNR8mZL", "reviewer": "Reviewer_eyRB", "summary": "", "questions": "A: How exactly are the linguistic features from Section 2.2 used to create the splits of the data used in the Accuracy test?\n\nB: Can the absolute value of the Accuracy test be interpreted? Can we state that one bias is larger than the other, without the significance test?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, we would like to express our sincere gratitude to the reviewer for recognizing the merits of our work. We are deeply encouraged by your affirmation of the reasonableness of our proposed method, the coherence of our results, and the potential contribution of our study to future research. Your appreciation for the comprehensive discussion in Section 4.3 gives us further motivation.\n\nIn response to the concerns and questions raised by the reviewer, we wish to address them as follows:\n\n1. Concerning the conclusions drawn from the results, especially in Table 3: We understand the reviewer's apprehensions regarding the premises of some of our claims. We'd like to emphasize that our choice of the feature “no” in MNLI was methodical. In table 2, the cueness score for this particular feature in the MNLI dataset was notably high. Moreover, it has been demonstrated to have a significant impact on other models like FastText and ESIM. We acknowledge that we may have presented our conclusions with undue definitiveness. Specifically, regarding the feature “no”, where P2 + CoT showcased the lowest bias risk based on both tests, our claims might have been overarching. If given an opportunity, we commit to articulating this point more prudently in our revised version.\n\n2. Regarding the term “extracted training data” mentioned between lines 259-271: This term was introduced in Section 3.1 during the Data Extraction Phase, where we defined Rf. This can be corroborated in Figure 2’s caption. Nevertheless, in light of your suggestion, we concede that it might be beneficial to utilize precise symbols to eliminate any ambiguity.\n\n3. On the topic of data splitting outlined in Section 3.1: We realize that our description may have inadvertently glossed over some intricacies. To enhance clarity: for a specific feature, if an instance contains that feature, then that instance will be singled out as a part of the subset bearing that feature. We will ensure that this is elucidated better in our revisions.\n\n4. In relation to the absence of source code and an anonymous repository for evaluation: We have invested considerable effort into developing an online platform hosted on our university's website. This platform allows for the uploading of data and models and provides diagnostic results for various features. However, due to the double-blind review process, we were hesitant to include the link. We genuinely apologize for any inconvenience this may have caused. Upon acceptance of our paper, we intend to provide links to all associated codes, including our platform.\n\n5. About the absolute value in the Accuracy Test: We reaffirm that the absolute value can indeed depict the extent to which a model is influenced by a specific feature. We will underscore this in our subsequent submission.\n\nIn wrapping up, we earnestly hope for an opportunity to make amends and present a more polished version of our work. Your feedback is invaluable in this endeavor, and we remain enthusiastic about incorporating these improvements. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed method is reasonable and easy to interpret \n* The results consistently complement the previous work in this direction, suggesting the methodical correctness of the reported results\n* The surveyed database of possibly biased linguistic features used in evaluation can be a useful building stone for future work\n* Related work that links a wide variety of work in spurious features identification and mitigation\n* Solid discussion of the results in Section 4.3.", "reasons_to_reject": "* I am concerned with the grounding of some of the claims based on the results. The results are often similar, but no significance tests are provided. An outstanding example of this is Table 3, from which the authors conclude that prompts have a \"substantial\" impact on robustness (L479) and that chain-of-thoughts prompting \"exhibits lower bias risk\" (L482). However, one should note that all these conclusions are based on two samples. Regardless of that, *all* $\\Delta$ Acc values reported in Table 3 might as well be insignificantly different from zero.\n* The description of the proposed evaluation methodology is hard to follow and might benefit from further details and more consistent use of terminology. \n  * For instance, on L259-271, authors use the term \"extracted training data\", which I can not disambiguate. \n  * Similarly, I have trouble following the notation of the MSE formula on L188. Here, I can not figure out what $L$ is. In the description that follows this formula, I do not understand the explanation of two instances of $y_i$ on L189-191. Thus, while I understand the idea, this disallows me to assess the absolute values reported for specific biases.\n  * I can not find how exactly the two compared subsets of data in the Accuracy test (Section 3.3.1) are created, based on the assessed feature. I believe the details should be exhaustively described somewhere.\n* The motivation for the specific method design could be better justified. We find the comparison to existing methods on L64-77 and in the Related work (L532-534), but these are too broad (\"they don’t evaluate the model using the full data context\"), or hard to follow (\"these works did not give the reason to use these metrics, one way or the other\").\n* Given that the authors propose an evaluation technique, it would be great to be able to assess its practical usability. However, no source code is attached, nor anonymised repository is available for assessment.", "questions_to_authors": "A: How exactly are the linguistic features from Section 2.2 used to create the splits of the data used in the Accuracy test?\n\nB: Can the absolute value of the Accuracy test be interpreted? Can we state that one bias is larger than the other, without the significance test?", "paper_topic_and_main_contributions": "The paper proposes a method for quantifying models' reliance on possibly spurious features (i.e. cues) in the training datasets. The authors identify a set of possibly biased features (specific words, sentiment, negation tokens, or named entities) and use their occurrences to split the test datasets into two segments on which they subsequently compare the performance of the models. In this manner, the authors evaluate four models (FastText, ESIM, BERT, RoBERTa) over ten NLI and multiple-choice QA. \nThe paper concludes that newer models (BERT, RoBERTa) learn to rely on dataset cues less than the older models. Further, they evaluate ChatGPT and conclude it to be more agnostic to dataset cues and state that the agnositicity can be further enhanced by prompt design and chain-of-thought prompting.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "I am aware of the following additional work evaluating language models for bias in prediction. It might be useful for the reader to differentiate your work from it.\n* Shikha Bordia and Samuel R. Bowman; 2019. Identifying and reducing gender bias in word-level language models.\n* Alicia Parrish et al.; 2022. BBQ: A hand-built bias benchmark for question answering.\n* Lukas Mikula et al.; 2023. Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models.", "Typos_Grammar_Style_and_Presentation_Improvements": "* L295:  \"on these datasets\" - which datasets? Missing reference?\n* L307: \"F\"eatures\n* The labels in Figure 3 are very small\n* L512: Perhaps you meant \"BLEU\" instead of \"BLUE\"\n* After reading the Introduction, I still did not get any idea about the functioning of the method. It would be beneficial to provide at least a high-level peek into what is going to follow."}}]

Author rebuttal:

Meta-review: The focus of this paper is on the issue of spurious correlation in NLU datasets. The argument presented is that biases or cues in NLU datasets can be exploited by NLP models, resulting in an overestimation of their capabilities. The paper is well-crafted and easily understood, with only minor notation revisions required. The proposed method is straightforward and straightforward to implement, and the experimental setup is relatively thorough. However, a few reviewers have expressed concerns about the validity of some claims based on the results. They claim that the results lack statistical significance tests, which undermines the credibility of the proposed method's impact. Also, some reviewers have observed that some of the accuracy scores might not be significantly difference from zero and that some conclusions drawn are based on a limited number of samples.
Decision: Accept-Findings