Explore Spurious Correlations at the Concept Level in Language Models for Text Classification (2024)

Abstract: Language models (LMs) have achieved notable success in numerous NLP tasks, employing both fine-tuning and in-context learning (ICL) methods. While language models demonstrate exceptional performance, they face robustness challenges due to spurious correlations arising from imbalanced label distributions in training data or ICL exemplars. Previous research has primarily concentrated on word, phrase, and syntax features, neglecting the concept level, often due to the absence of concept labels and difficulty in identifying conceptual content in input texts. This paper introduces two main contributions. First, we employ ChatGPT to assign concept labels to texts, assessing concept bias in models during fine-tuning or ICL on test data. We find that LMs, when encountering spurious correlations between a concept and a label in training or prompts, resort to shortcuts for predictions. Second, we introduce a data rebalancing technique that incorporates ChatGPTgenerated counterfactual data, thereby balancing label distribution and mitigating spurious correlations. Our method’s efficacy, surpassing traditional token removal approaches, is validated through extensive testing. 1

heading: 1 Introduction
text: Pre-trained language models (LMs), leveraging extensive text corpora in their pre-training phase, have demonstrated remarkable effectiveness in a variety of natural language understanding tasks (Wei et al., 2022; Devlin et al., 2018). Nevertheless, LMs encounter issues with spurious correlations during fine-tuning or instruction-following stages (Zhang et al., 2022; Wang et al., 2022; Tang et al., 2023). These correlations involve specific associations between features and labels that, while prevalent in training data, are erroneously generalized as rules, leading to reduced performance.
1Our code and data are available at https://github. com/Tonyzhou98/concept-spurious-correlation
Current research on spurious correlations in LMs spans various dimensions, such as token-level shortcuts in text classification (Wang et al., 2022; Tang et al., 2023; Chew et al., 2023), syntactic heuristics in natural language inference (McCoy et al., 2019), sentence triggers in text classification (Tang et al., 2023; Jia and Liang, 2017), and topic shortcuts in machine translation (Borah et al., 2023). Moreover, spurious correlations with demographic concepts like race or sex, raise fairness concerns (Kleinberg et al., 2018). Yet, studies seldom address semantic spurious correlations at a broader concept level.
We define spurious correlations at the concept level as: Most texts featuring a certain concept in training data (or prompts) are linked with a specific label, leading LMs to inappropriately rely on this association for predictions. For instance, in Figure 1, terms like “salsa,” “fast food burgers,” or “Thai steak” denote the concept “food.” A prevalent association between “food” and label 1 in training data
478
or prompts results in LMs forming a concept-level spurious correlation, mistakenly assigning some “food”-related texts to label 1.
The tendency of LMs to learn concept-level shortcuts might stem from the formation of similar embeddings for expressions related to the same concept during fine-tuning or pre-training, driven by their semantic similarities. As Figure 2 suggests, various expressions of a concept cluster closely in the embedding space of fine-tuned or pre-trained LMs. When similar embeddings frequently coincide with a label in training or demonstrations, LMs tend to adopt the shortcut. We offer an in-depth analysis using a specific dataset in Section 3.2.
In the first part of our study, we assess and quantify concept-level spurious correlations in LMs across both fine-tuning and ICL scenarios within text classification tasks. Initially, we employ the advanced large language model (LLM), ChatGPT, to identify relevant concepts in each dataset (Ouyang et al., 2022) and to predict the presence of these concept labels. In the fine-tuning setting, we train LMs on both the original dataset and a conceptbiased counterpart. Our findings indicate that LMs exhibit concept-level spurious correlations in standard benchmarks, with more pronounced prediction biases emerging from increasingly imbalanced data. In the ICL setting, we compare the performance of LMs on concept-balanced and conceptbiased prompts, demonstrating that biased prompts lead to more skewed inferences.
The second part of the paper explores the use of data rebalancing techniques to counteract these spurious correlations in a fine-tuning framework. We introduce an upsampling strategy that incorporates counterfactual texts generated by ChatGPT, which effectively reduces bias while maintaining the utility (i.e., accuracy) of the LMs. In summary, our research makes three significant contributions:
• We are the first to investigate spurious correlations at a general concept level and introduce a metric to quantify these correlations.
• Through experiments on various benchmark data for text classification, we demonstrate that LMs are prone to adopting learned concept-level shortcuts in both fine-tuning and ICL settings.
• We introduce an effective upsampling approach, incorporating counterfactuals generated by LLMs, to mitigate concept-level bias.

heading: 2 Exploring Concept-level Spurious Correlations

heading: 2.1 Obtaining the Concept Labels
text: Due to the lack of human-annotated metadata indicating concepts in most text classification datasets, and considering the superior capabilities of LLMs in text annotation tasks over human annotators (Gilardi et al., 2023), we utilize ChatGPT (GPT-3.5) to annotate concept labels for sentences in text classification datasets (Ouyang et al., 2022). Our annotation process involves an annotation prompt Pa that contains the annotation instruction and five demonstrations, a text input x, LLM Ma, and a candidate concept set C = {C1, C2, · · · , Ck} (we describe how we curate the candidate set in Section 3).
The annotation process is formalized as: a(x) = Ma(Pa∥C∥x), where a(x), the set of concept labels for text x, may contain zero or several concepts selected from the pre-defined concept set C (a(x) ⊂ C), and ∥ denotes the concatenation operation. To ensure reliability, we repeat the annotation process twice with a temperature setting of 0.7 and retain only those examples and labels that are consistently identified by both LLM annotators.

heading: 2.2 Measuring Concept Spurious Correlations
text: For the text classification task, we consider an input x ∈ X accompanied by concept labels a(x) ⊂ C. Each input is associated with a ground truth classification label y = l from the output label space Y , l ∈ {0, 1, · · · , n}. Given a LM classifier M : X → Y , if the model avoids utilizing potential concept-level shortcuts from c → y, c ∈ C, the following condition is satisfied:
Ex[pM (ŷ = l|x, c ∈ a(x), y = l)] (1) =Ex′ [pM (ŷ = l|x′, c /∈ a(x′), y = l)] ∀l ∈ Y.
Here, ŷ denotes the predicted label, while pM represents the probability predicted by model M . The inputs x and x′, belonging to the space X , contain the concept c or do not contain it, respectively.
Equation 1 implies a critical condition: regardless of the presence of concept c in the input, the models should maintain an unbiased estimate of the predicted probability on average. The expression Ex[pM (ŷ = l|x, c ∈ a(x), y = l)] can be interpreted as the model’s accuracy on texts that are labeled l and incorporate the concept c.
Denote ∆ci as the difference in model accuracy between texts with or without concept c that have label i ∈ Y . We further infer from Equation 1 that:
∆ci =Ex[pM (ŷ = i|x, c, y = i)] (2) −Ex′ [pM (ŷ = i|x′,¬c, y = i)] = 0,
where ¬c denotes concept c is not in input x. We hypothesize, if there exists a spurious correlation in models between concept c and label i, the following conditions would hold:
Ex[pM (ŷ = i|x, c, y = i)] > Ex′ [pM (ŷ = i|x′,¬c, y = i)] Ex[pM (ŷ = j|x, c, y = j)] < Ex′ [pM (ŷ = j|x′,¬c, y = j)]
Then we have ∆ci > 0 > ∆cj . Otherwise, if the spurious correlation is between c and j, then ∆cj > 0 > ∆ci . We propose to measure the discrepancy between ∆ci and ∆cj to quantify the spurious correlation. Hence, considering the output space Y , we quantify the model’s reliance on shortcut mapping as the average discrepancy in the accuracy difference ∆ci −∆cj across all label combinations.
Bias@C = 1( n 2
) ∑
i,j∈Y (∆ci −∆cj ), i > j
For the binary classification task, the bias measurement is simplified to Bias@C = ∆c1 −∆c0
A Bias@C approaching 0 indicates minimal reliance on concept shortcuts. Conversely, a positive Bias@C value suggests that model is more likely to predict larger labels when the input includes concept c, and the opposite for a negative value.

heading: 2.3 Evaluation of Model Robustness to Concept Shortcut in Fine-tuning
text: To assess LMs’ robustness against spurious correlations for concept c across varying scales of concept bias during fine-tuning, we fine-tune models on the original dataset Dori and a concept-biased dataset
Dcbiased separately. To further demonstrate the impact of concept-level spurious correlation, we construct Dcbiased of concept c by filtering Dori, where, for each data point, we only keep those with the majority labels under concept c. After fine-tuning on Dori or Dcbiased, we evaluate models on test data using Bias@C to quantify spurious correlations.
We report accuracy on the test data for utility performance. However, label distributions with or without the concept c may be imbalanced. Following previous work (Chew et al., 2023), we rebalance the test set by downsampling and report the inference accuracy (robust accuracy) on the balanced subset for examples with concept c (Acc@C) and without concept c (Acc@NoC), respectively.

heading: 2.4 Evaluation of Model Robustness to Concept Shortcut in ICL
text: As LLMs have shown outstanding performances with the ICL setting, we are interested in investigating the concept shortcut in the demonstrations. The prompt P for ICL contains three parts: 1) the instruction s, 2) the demonstrations with h exemplars (text + labels), and 3) the test input xtest.
We consider the sentiment classification task and concatenate the h exemplars together with the form “Input: x. The sentiment label is v(y)”. The label verbalizer v(y) will transfer 0 to “negative” and 1 to “positive” when the label is binary and will maintain the original numerical rating scales when multiple classes (n ≥ 3). The ICL process is formulated as f(xtest) = M(P∥xtest), where f(xtest) is a categorical variable belonging to Y .
We create two types of prompts: the biased prompt Pbiased and the balanced prompt Pbalanced by changing the label distributions in the demonstrations. For Pbiased, we insert h2 numbers of exemplars containing the concept c with the label l ∈ {l1, l2, · · · , lk} (the majority ground truth labels) and h2 numbers of exemplars without c with the other labels. For Pbalanced, we split the exemplars with concept c or not half by half, but ensure balanced labels. We compare the results of Bias@C and robust accuracy with two types of prompts. Since the ICL are sensitive to the exemplars, we repeat the experiments three times with differently selected exemplars and report the average values.

heading: 3 Dataset Construction and Analysis
text: Models We assess and mitigate concept-level bias in DistilBERT and LLAMA2 7B in fine-tuning
setting (Sanh et al., 2019; Touvron et al., 2023) and GPT3.5 in the ICL setting (Ouyang et al., 2022). We fully fine-tune the DistillBert. For LLAMA2, we apply the Lora method for efficient fine-tuning (Hu et al., 2021). Details of the model implementations are included in Appendix A.
Dataset We select four sentiment classification tasks to evaluate the model robustness: Yelp (Zhang et al., 2015), IMDB (Maas et al., 2011), Amazon Shoe (He and McAuley, 2016), and CeBaB (Abraham et al., 2022). Amazon shoe and CeBaB datasets with 5 classes, 0 indicating the most negative and 4 indicating the most positive, are reviews of shoes in Amazon and OpenTable. IMDB and Yelp are binary classification datasets (0 indicating negative and 1 indicating positive), with reviews from the IMDB and Yelp platforms.
Additionally, we include a binary question answering (QA) dataset BoolQ (Clark et al., 2019), which asks Yes/No questions. It takes a paired question and passage as the input to LMs and outputs 1 (Yes) or 0 (No). In the following part, we define the positive class as datapoints with Label 3 and 4 for the 5-way classification tasks and those with Label 1 for the binary classification task. We define the remaining datapoints as the negative class.
Concept For CeBaB, we adopt human-annotated concept labels. For Amazon Shoe, IMDB, Yelp, and BoolQ where there are no concept annotations, we first use ChatGPT to query the concepts embedded in each sentence and count the number of occurrences for each concept following (Fang and Zhang, 2022) to generate concept-level explanations. We then extract the most frequent concepts and identify the concepts whose existence should not influence the sentiments of the text or the Yes/No answer to the question (2 concepts for BoolQ due to more diverse topics and 3 concepts for other datasets). Finally, we use ChatGPT to annotate whether each text input contains the selected concept.
To examine the quality of the annotation, we experiment on the human-annotated “service” concept from the CeBaB dataset and ask the Chat-
GPT to label the concept again. We find that ChatGPT can achieve an accuracy of 90.4% to the gold standard concept labels, comparable to an average agreement of 92.9% for five human annotators given by CeBaB, indicating the reliability of LLM annotations. Table 1, presents dataset statistics and the labeled concept lists for the five datasets.

heading: 3.1 Biased Dataset Construction
text: We first visualize the label distribution for the input texts with the concept c for each sentiment classification in Figure 3. We observe that for the original datasets, the concept-label distributions are balanced for most concepts, but not as balanced for concepts such as “food” in Yelp dataset, “music” in IMDB, and “style” in Amazon Shoe. In 10/12 cases, positive labels comprise large proportions of the corpus with certain concepts. To further demonstrate the impact of concept-level spurious correlation caused by imbalanced concept-label distribution, we construct a biased dataset Dcbiased which, for each concept c, only includes the majority class (positive or negative). Specifically, we keep negative class for “size” in Amazon Shoe and “service” in Yelp. For other concepts in the sentiment datasets, we keep positive class. For BoolQ, we keep negative class with “country” and positive class with “history”.

heading: 3.2 Embedding Analysis of Associated Tokens
text: As shown in Figure 2, we hypothesize that expressions of a concept have similar semantic embeddings, leading to shortcut learning. To verify the hypothesis and further motivate the measurement of spurious correlations, we extract the embeddings of the associated tokens with each concept in the Amazon shoe dataset. We observe whether the embeddings of tokens associated with the same concept are similar using clustering.
We apply the point-wise mutual information (PMI) between the token and the concept to measure the association. For a dataset with a concept c, we calculate the PMI of each token t to concept c, which is PMI(t, c) = log p(t,c)p(t)p(c) , where p(t), p(c) and p(t, c) refer to the probability of the text containing t, c and both together. The higher value of PMI suggests a stronger association between t and c. We present tokens with the top 10 PMI values for each concept in Table 2.
From the associated tokens in Table 2, we observe tokens with various semantics associated with one concept, such as “small,” “sizing” and “9m” to express the “size” concept. We use the tokens in Table 2 to perform the clustering. We exclude tokens with special character, such as “c/d,” “81/2” and “(55)” that affect the interpretation of the results. We feed the tokens into the DistilBERT fine-tuned on the Amazon Shoe and retrieve the corresponding embedding from the model last layer. If the token is tokenized into multiple sub-words, we follow the previous work and calculate the average as the token embedding (Wolfe and Caliskan, 2021). We calculate the cosine distance between their token embeddings and apply hierarchical clustering (Bar-Joseph et al., 2001).
From Figure 4, we can identify four small clusters, each representing a concept. We observe that the LMs will produce similar internal representations for tokens associated with the same concept label. If the label under a concept is imbalanced, the models may learn the undesired shortcut be-
tween similar embeddings and a label. This observation motivates the measurement of spurious correlation at the concept level.

heading: 4 Results of Spurious Correlation Measurement

heading: 4.1 Spurious Correlations in Fine-tuning
text: To evaluate the robustness to the concept shortcut in the fine-tuning setting, we fine-tune the models on the original dataset Dori and the biased dataset Dcbiased, respectively, and measure the concept bias. For each concept, we report the metric Bias@C to quantify the strength of spurious correlations and the robust accuracy for texts with and without concept, i.e., Acc@C and Acc@NoC, as the utility performance. For Bias@C, closer to 0 indicates weaker spurious correlations for concept C, and for robust accuracy, a higher value suggests better performance. We present the results for DistilBERT on sentiment classification in Table 3 and BoolQ dataset in Table 7 Appendix B. Fine-tuned LMs present a clear concept bias when trained on both original and biased data. Table 3 shows that when the models are trained on Dori, they utilize spurious correlations in the training data to make inferences. For example, for “style” in the Amazon Shoe and “music” in IMDB, the Bias@C values in Dori are large due to highly imbalanced label distribution. Since these datasets are well curated and widely adopted, the fact that we are able to identify several highly biased concepts by only investigating the top 3 fre-
quent concepts demonstrates the significance of spurious correlation.
Comparing the results between Dori and Dcbiased, we find that the absolute values of Bias@C are significantly higher when trained on Dcbiased in almost every concept, and the change direction of Bias@C is the same as the trend in the label distribution. For example, the value of Bias@C becomes negative for “service” in Yelp dataset, since we only keep negative reviews with the “service” in Dcbiased. These observations indicate that a greater bias in the fine-tuning dataset causes the model to rely more on spurious correlations to make predictions.
Regarding utility performance (Acc@C and Acc@NoC), we observe that the models trained on Dcbiased have a dramatic performance drop on the texts with the concept in most cases, and the average Acc@C decreases from 79.38% to 74.31%. This pattern suggests that larger spurious correlations affect the utility performance of fine-tuned models. Meanwhile, the average Acc@NoC drops from 78.08% to 76.56%. Its performance drop is not as large as the one of Acc@C, indicating that texts without the concept suffer less from the concept bias in the datasets.
Moreover, we find that for some concepts, the fine-tuned LMs suffer from severe spurious correlation, but the effect of the bias is not fully reflected in the difference between Acc@C and Acc@NoC. For example, the “music” concept in the IMDB dataset has Bias@C = 12.07%, but the difference
between Acc@C and Acc@NoC is less than 3%. This is because if the model is biased towards one label due to the spurious correlation, the accuracy improvement towards the biased label can often offset the performance drop of the other side.
We also verify that the concept bias is not simply due to the shortcut on a few words by masking out the associated tokens, and details are shown in Section 5.2. We show fine-tuning results of LLAMA2 models on Dori and Dcbiased in Table 7 and 8 (Appendix B). Similar patterns suggest the generalizability of our findings on models of different sizes.

heading: 4.2 Spurious Correlations in ICL
text: As LMs exhibit clear evidence of utilizing the concept shortcuts in the fine-tuning data, we also want to ask whether LMs use the shortcuts in the exemplars of the prompts when performing ICL. As discussed in Section 2.4, for each concept c in five datasets, we construct a prompt with eight exemplars. Following a similar setup for fine-tuning, we only include the majority class (positive or negative) for exemplars with concept c. Specifically, for “size” in Amazon Shoe and “service” in Yelp, four exemplars with concept c have negative labels and the other four without concept c have positive labels. The labels are flipped for the rest of the concepts. For the balanced prompt Pbalanced, the label is evenly distributed for the exemplars with or without the concept. With the bias in label distribution for both texts with and without the concept,
we expect the LM to use two types of shortcuts: a) from texts with the concept c to one sentiment and b) from texts without the concept to the other sentiment. We present the utility and bias results of ICL for sentiment classification dataset in Table 4 and BoolQ in Table 7 (Appendix B).
Biased prompts enlarge the concept bias in ICL inference From Table 4, we observe a similar pattern for Bias@C as in the fine-tuning part. When the prompt changes from Pbalanced to Pbiased, for “service” in Yelp and “size” in Amazon Shoe, where the exemplars with concept c are negative, the values of Bias@C flip from positive to negative, and for most other concepts, where conceptual exemplars are all positive, the value of Bias@C increases. Furthermore, in most cases, the absolute values of Bias@C when using Pbiased are higher. These observations indicate that the LMs are affected by concept shortcuts within the prompt of ICL.
For utility performance, when changing from Pbalanced to Pbiased, the average Acc@C and Acc@NoC decrease from 76.80% to 76.42% and from 76.37% to 75.58%, respectively, which means that spurious correlations harm utility performance regardless of the presence of concepts. For both Bias@C and accuracy, the relative change in the ICL scenario is less than the fine-tuning setting. We conjecture that a few exemplars in prompts make it hard to form a strong shortcut inside the LMs between conceptual contents and a specific label.

heading: 5 Mitigate Spurious Correlations

heading: 5.1 Mitigation via Rebalancing
text: We consider two lines of existing data-centric work to mitigate spurious correlations: remove spurious components and rebalance the training dataset (McCoy et al., 2019; Wang et al., 2022). Since it is challenging to identify the conceptual contents in each sentence, we apply dataset rebalancing methods to mitigate the bias at the concept level.
We first downsample the dataset to achieve a balanced label distribution with respect to concept c, denoted as Dcdown−bal. The shortcoming of the method is that, for a highly biased dataset, it filters out a large proportion of examples with the majority labels, leading to a sacrifice of the utility performance. To address this, we propose an upsampling method using ChatGPT to generate counterfactual examples with minority labels. Some concurrent work also demonstrates the effectiveness of synthetic data in mitigating bias (Evuru et al., 2024). The resulting dataset is denoted as Dcup−bal.
Suppose that we need {a0, · · · , an} number of examples for labels {0, · · · , n} to make a balanced subset for texts with concept c. We first sample a0 to an numbers of examples from texts with labels 0 to n but without concept c. Then we ask ChatGPT, Ma, to inject the concept c into the selected texts while maintaining the sentiment or the answer to questions. Given the input text x′ without concept c, the injection prompt Pi with the instruction, and the exemplars hc with concept c, the concept injection process is xc = Ma(Pi∥hc∥x′), where xc is the generated counterfactual for concept c and
input x′. We iteratively generate the counterfactual input xc and add it into the dataset Dori to form a balanced dataset Dcup−bal with upsampling. To demonstrate the effectiveness of concept injection, we conduct a case study on a review in the Yelp dataset. As suggested in Table 5, we inject the “food” concept into a review without this concept and observe that ChatGPT effectively injects the food concept, keeps other content unchanged, and maintains the sentiment of the review.
We also propose a baseline method that masks out words highly associated with the concept. This method is used to verify whether balancing distributions of a few tokens removes conceptual shortcuts. We replace words with the top 10 PMI for each concept (word examples are in Table 2) to the [MASK] token and name the masked dataset as Dcmask.

heading: 5.2 Results of Mitigation Methods
text: To evaluate the effectiveness of proposed methods, we select concepts with Bias@C greater than 1 in Table 3 and fine-tune on three de-biased datasets Dcdown−bal, Dcup−bal, and Dcmask. We report results for DistilBERT in Table 6 and 7 in Appendix B. Upsampling method reduces the bias and increases utility performance From Table 6, we observe that data rebalancing methods are effective in mitigating spurious correlations. For downsampling (Dcdown−bal), it mitigates the mean absolute values of Bias@C from 4.90% to 3.43%, compared to trained on Dori. However, for utility performance, the downsampling obtains less accuracy in 4/8 cases for Acc@C and 5/8 cases for Acc@NoC, indicating that loss of data harms utility. For the upsampling method (Dcup−bal), the mean absolute values of Bias@C are effectively reduced from 4.90% to 2.74%. Furthermore, the average accuracy of Acc@C increases from 79. 24% to 80. 38%, and Acc@NoC is comparable. This observation suggests that adding counterfactual texts to rebalance the data can reduce spurious correlations in the concept level, and more data involved in the fine-tuning can boost the models’ utility performance.
Masking out associated tokens (Dcmask) can reduce spurious correlations in most cases, but cannot fully eliminate bias. This observation suggests that due to the various concept expressions, the learned concept shortcut in the model is not equivalent to the shortcut on a few tokens. The utility performance of Acc@C is also lower than that of the proposed upsampling method in 6/8 comparisons.
In summary, among the three mitigation methods, adding the LLM-generated counterfactual inputs achieves the best performance in both the bias mitigation and utility aspects. The same analysis on LLAMA2 models (Table 7 and 9 in Appendix B) reveals similar patterns, which shows the generalizability of our methods.

heading: 6 Related Work
text: Robustness and Bias Current work on studying spurious correlations for LMs can be split into two categories: utilize the shortcuts during training or ICL. For shortcut learning in training, a series of works explores how models take shortcuts in the data for the causal or non-causal perspective (Tu et al., 2020; Sagawa et al., 2020; Geirhos et al., 2020; Ribeiro et al., 2020; Kaushik et al., 2019; Liu et al., 2024; Friedman et al., 2022) and which aspects of shortcuts will be taken for the predictions in different NLP tasks (McCoy et al., 2019; Jia and Liang, 2017; Lai et al., 2021; Zhao et al., 2018; Niu et al., 2020; Poliak et al., 2018), leading to low generalization in the out-of-distribution data or in the designed adversarial data.
Due to the increasing development of LLM on ICL, researchers find that the design of the prompt significantly decides the LLM predictions (Brown et al., 2020; Gao et al., 2020; Liu et al., 2023b; Zhou et al., 2023; Schick and Schütze, 2020). Another line of work finds that LLMs are sensitive to a certain aspect of prompts and not robust when injecting adversarial triggers into prompt (Lu et al., 2021; Zhao et al., 2021; Tang et al., 2023; Si et al., 2023; Zheng et al., 2023). Tang et al. (2023) shows that LLMs use multiple types of shortcuts in the prompts, from letters to words to text style, and Si et al. (2023) find that LLMs exhibit clear feature biases under the unspecified prompts. Previous work also develops multiple methods to identify the topic or concept of text input (Li et al., 2024a; Abraham et al., 2022; Blei et al., 2003). However, our paper is the first to focus on assessing whether the models use shortcuts at the general concept level.
Spurious Correlation Mitigation An increasing number of methods have attempted to mitigate spurious correlations in models caused by bias in the dataset (Chew et al., 2023; Clark et al., 2020; Le Bras et al., 2020; Zhou and Bansal, 2020; Liu et al., 2021, 2023c; Zhu et al., 2023), by data augmentation (Jin et al., 2020; Alzantot et al., 2018; Wang et al., 2022; Minderer et al., 2020), data rebalancing (McCoy et al., 2019; Sharma et al., 2018; Zellers et al., 2019), multi-task learning (Tu et al., 2020), and model ensembling or adding regularization (Utama et al., 2020; He et al., 2019; Zhao et al., 2022; Liu et al., 2023d). To mitigate spurious correlations in a concept, we propose another data rebalancing method, which uses LLM to generate counterfactual sentences by injecting the concept and saves the human resource to compose them.

heading: 7 Conclusions
text: In this paper, we explore the spurious correlation at the general concept level in both fine-tuning and ICL settings. We find that LMs utilize the concept shortcut in training data (or in demonstrations) when inference on unseen data, and more biased training data (or prompts) lead to more biased predictions. To mitigate the learned shortcut, we propose a rebalancing method by adding counterfactual examples generated from ChatGPT to the training data, shown to be effective through extensive experiments. Our work indicates that LMs form strong spurious correlations on general concepts, encouraging future work to pay attention to unintended shortcut learning.

heading: 8 Limitations
text: Due to the limitation of the budget and the computation resource, we only fine-tuned the LLaMa2 7B model with the Lora method and used GPT3.5 for concept annotation. It could be interesting to fully
fine-tune the LMs with a larger size. Moreover, in Section 3, we find that the ChatGPT annotation of the concept label still achieves slightly lower accuracy than human annotators. We can use a more advanced model, such as GPT4 (OpenAI, 2023), for annotation to increase the performance.
Our work focuses on five classification tasks. Three of them are binary classification tasks, and two are multiclass classification tasks. We apply the difference in accuracy for different groups (positive and negative) to measure bias at the concept level. Moreover, future work could extend our framework and generalize the measurement of concept bias to more complex tasks, such as the evaluation of LLM on QA tasks (Li et al., 2024b) or even on tasks with the vision language model (Wang et al., 2024b; Liu et al., 2023a; Wang et al., 2024a; Yue et al., 2023).
For in-context learning, we observe that the concept bias in the demonstrations leads to larger spurious correlations. However, we also detect that the balanced prompts cannot fully eliminate the bias, and we do not provide a method to mitigate this inner spurious correlation in LMs. We leave that direction to future work.

heading: Acknowledgments
text: Zhou and Huang are supported by DARPA Transfer from Imprecise and Abstract Models to Autonomous Technologies (TIAMAT) 80321, National Science Foundation NSF-IIS-2147276 FAI, DOD-ONR-Office of Naval Research under award number N00014-22-1-2335, DOD-AFOSRAir Force Office of Scientific Research under award number FA9550-23-1-0048, DOD-DARPADefense Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD) HR00112020007, Adobe, Capital One and JP Morgan faculty fellowships.

heading: B Prompt Details and Supplementary Results
text: In Table 7, we perform the same analysis on the BoolQ question and answering dataset for all experiments (ICL and fine-tuning) in Section 4. In Table 8 and Table 9, we repeat the experiments for fine-tuning LLAMA2 7B models for Section 4.1 and 5.2. In Table 10, 11, and 12, we present the details of the prompts in the annotation of the concept, the ICL experiments, and the countergactual sentence generation.
2https://pytorch.org/ 3https://huggingface.co/ 4https://ai.meta.com/llama/

heading: Future Work
text: Moreover, future work could extend our framework and generalize the measurement of concept bias to more complex tasks, such as the evaluation of LLM on QA tasks (Li et al. , 2024b) or even on tasks with the vision language model (Wang et al. , 2024b; Liu et al. , 2023a; Wang et al. , 2024a; Yue et al. , 2023). For in-context learning, we observe that the concept bias in the demonstrations leads to larger spurious correlations. However, we also detect that the balanced prompts cannot fully eliminate the bias, and we do not provide a method to mitigate this inner spurious correlation in LMs. We leave that direction to future work