Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMs
Authors: Pengda Wang
Zilin Xiao
Hanjie Chen
Frederick L. Oswald
Venue: COLM (2024)
Keywords: Psychology, Representativeness Heuristic, Language Models
TL;DR: Dissecting Representativeness Heuristic in Language Models

Abstract: Although large language models (LLMs) have demonstrated remarkable proficiency in modeling text and generating human-like text, they may exhibit biases acquired from training data in doing so. Specifically, LLMs may be susceptible to a common cognitive trap in human decision-making called the representativeness heuristic. This is a concept in psychology that refers to judging the likelihood of an event based on how closely it resembles a well-known prototype or typical example, versus considering broader facts or statistical evidence. This research investigates the impact of the representativeness heuristic on LLM reasoning. We created ReHeAT (Representativeness Heuristic AI Testing), a dataset containing a series of problems spanning six common types of representativeness heuristics. Experiments reveal that four LLMs applied to ReHeAT all exhibited representativeness heuristic biases. We further identify that the model's reasoning steps are often incorrectly based on a stereotype rather than on the problem's description. Interestingly, the performance improves when adding a hint in the prompt to remind the model to use its knowledge. This suggests the uniqueness of the representativeness heuristic compared to traditional biases. It can occur even when LLMs possess the correct knowledge while falling into a cognitive trap. This highlights the importance of future research focusing on the representativeness heuristic in model reasoning and decision-making and on developing solutions to address it.

Published as a conference paper at COLM 2024
Will the Real Linda Please Stand Up...to Large Language Mod- els? Examining the Representativeness Heuristic in LLMs
Pengda Wang', Zilin Xiao", Hanjie Chen, Frederick L. Oswald Rice University
[pw32,ziling hc86, fo3 (oo3)rriceeem
Abstract
Although large language models (LLMs) have demonstrated remarkable proficiency in modeling text and generating human-like text, they may exhibit biases acquired from training data in doing so. Specifically, LLMs may be susceptible to common cognitive trap in human decision-making called the representativeness heuristic. This a concept in psychology that refers to judging the likelihood of an event based on how closely it re- sembles a well-known prototype or typical example, versus considering broader facts statistical evidence. This research investigates the im- pact of the representativeness heuristic on LLM reasoning. We created REHEAT (Representativeness Heuristic AT Testing), dataset containing a series of problems spanning six common types of representativenes heuris- tics. Experiments reveal that four LLMs applied to REHEAT all exhibited representativeness heuristic biases. We further identify that the model's s reasoning steps are often incorrectly based on a stereotype rather than on the problem's 's description. Interestingly, the performance improves when adding hint in the prompt to remind the model to use its knowledge. This suggests the uniqueness of the representativeness heuristic compared to traditional biases. It can occur even when LLMs possess the correct knowl- edge while falling into cognitive trap. This highlights the importance of future research focusing on the representativeness heuristic in model reasoning and decision-making and developing solutions to address it.
1 Introduction
"Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As student, she was deeply concerned with issues of discrimination and social justice and also participated in anti-nuclear demonstrations."
We have seven statements below: [1] Linda is a teacher in elementary school. [2] Linda works in a bookstore and takes Yoga classes. [3] Linda a psychiatric social worker. [4] Linda is member of the League of Women Voters. [5] Linda is bank teller. [6] Linda is an insurance salesperson. [7] Linda bank teller and is active the feminist movement.
Question: Rank the seven statements associated with each description by the degree to which Linda resembles the typical member of that class.
Tversky & Kahneman (1983)
This is one experiment known as the "Linda problem," devised by Tversky and Kahne- man; results demonstrate how respondents are influenced by specific descriptions, such as being deeply concerned with issues discr iminatioo and social justice and also
participating in inti-nuclear demonstratio ietssstrations to rank [7] higher than [5]. However, [7] combines [5] and an additional event. This means that, from a statistical perspective, [5] is more likely to occur than [7] because it is more general and less restrictive.
"qqual contribution

---

Published as a conference paper at COLM 2024
?
Statistic Prololype Question (SPQ)
You have two independent events A and B.
P(B)
P(B) or P(ANB). Which greater?
Representativeness Heuristic Question (RHQ)
[$] Linda is bank teller [7] Linda is bank teller and is active in the feminist movement.
Prompt
[7]
x
Which is greater?
Prompt Hint!
[S]
Figure 1: Illustration of the representativeness heuristic problem. The model possesses the knowledge to answer the statistical prototype question, yet fails to use it to solve the representativeness heuristic question. Providing an appropriate hint in the prompt can guide the model in making a correct prediction.
Kahneman & Tversky (1973) introduced this phenomenon as the "representativeness heuris- tic," which involves estimating the likelihood of an event by comparing it to an existing prototype in our minds. offers a convenient shortcut in decision-making by aligning with intuitive thinking, leading people to rely on it frequently. Generally, this heuristic is quite beneficial, as simplifies complex judgments. However, it is important to recognize that it can also result in significant decision-making errors, given that people are prone to assessing the likelihood of an object's s category membership based on superficial similarities, while neglecting actual statistical evidence. For example, when they lack the benefit of individuating information about a person, decision- decision-makers might categorize individuals based their physical appearance, observation of limited behaviors, or superficial back- ground descriptions, leading to skewed perceptions and decisions. This type of stereotyping phenomenon is durable, being widespread across cultures and over time (Spencer et al., 2016).
Because LLMs are trained on real-world data and instructed emulate human behavior, they may capture the representativeness heuristic. Previous work has mainly focused on the biases within training data (Bender et al., 2021; Bolukbasi et al., 2016; Garg et al., 2018; Sheng et al., 2019; Zhao et al., 2019). These biases often stem from data distributions that do not reflect proportions that would drive decision-making that is unbiased, given a researcher's operational definition of bias. In contrast, the representativeness heuristic represents type cognitive bias that has yet to be thoroughly investigated in LLMs. in potentially leading models to make mistakes, even when they possess the knowledge necessary to solve the problem. As illustrated in Figure 1, the model is able to answer the Statistical Prototype Question (SPQ), yet it tends to fail to answer the Representativeness Heuristic Question (RHQ). The SPQ and RHQ are intrinsically equivalent, where SPQ is expressed statistically, and RHQ expresses the same statistical logic verbally in scenario. This indicates that LLMs can engage in erroneous cognitive reasoning, even with accurate knowledge of statistical probabilities. Interestingly, providing an appropriate hint can prompt an LLM to use its knowledge to make correct predictions more frequently. This indicates that the representativenns eness heuristic can block the model from following a correct reasoning path; instead, it relies on the data that captures the human tendency to use cognitive shortcuts to make decisions.
is unique
To investigate the representativeness heuristic in LLMs, we construct a dataset REHEAT IRepresentativeness Heuristic AI Testing), which contains 202 RHQs that span six types of representativeness heuristics: Base Rate Fallacy, Conjunction Fallacy, Disjunction Fallacy, Insensitivity to Sample Size, Misconceptions of Chance, and Regression Fallacy. The ques-
Code and dataset will be available tittps://githie wtw.thuww.com////iiiiiiiiaaaa//a httt::://gthhhu../////iillliiiit i sticReHET..
2

---

Published as a conference paper at COLM 2024
tions we designed are adapted from those used in prior investigations into heuristics within the field of psychology (Bar-Hillel & Neter, 1993; Kahneman & Tversky, 1973; Kahneman et al., 1982; Tversky & Kahneman, 1974; 1983). To the best of our knowledge, our dataset is the first to offer extensive and comprehensive coverage of RHQs, enabling exploration of LLMs' capabilities in countering the representative heuristic cognitive bias.
We evaluate four LLMs: GPT-3.5 (Ouyang et al., 2022), GPT-4 (OpenAI, 2023), PaLM 2 (Anil et al., 2023), and LLaMA (Touvron et al., 2023), on REHEAT using different prompts. Our findings indicate that these LLMs exhibit behaviors that closely mirror human heuristic behavior. Additionally, advanced prompting techniques such as chain of thought (CoT) (Wei et al., 2022), in-context learning (Brown et al., 2020), and self-consistency (Huang et al., 2022) prompting, offer marginal improvements. Nevertheless, when explicitly prompted to recall its knowledge, the model shows an improvement in performance. This underscores the significance of future research to address the representativeness heuristic, guiding L.Mss toward correct reasoning and decision-making.
Related Work
Social biases in natural language processing (NLP) systems and related data have been studied with respect to their fairness, inclusivity, and accuracy (Hutchinson et al., 2020; Maass, 1999; Zhao et al., 2018). For example, Bolukbasi et al. (2016); Garg et al. (2018) are among the pioneers in demonstrating gender-related associations in word embeddings that might reflect and perpetuate stereotypes. Caliskan et al. (2017) concludes standard machine learning methods for NLP could acquire societal biases from textual data. Some research has expanded our understanding of where NLP systems might acquire subgroup associations within data, and potential bias, including those from data collection (Bender & Friedman, 2018), annotation processes (Gebru et al., 2018), and model architecture choices (Zhao et al., 2017). To date, numerous efforts have been made to mitigate social biases in systems through a variety of methods, including data augmentation (Lu et al., 2018), changes in model architecture (Liang et al., 2020), and training objectives (Liu et al., 2021; Romanov et 2019).
In similar vein, although recent advancements in LLMs are exciting, researchers are concerned about whether LLMs inherit social biases from the trillions of tokens they have been trained on. Weidinger et al. (2022) provides comprehensive taxonomy of social risks within LLMs. Although the research community has documented numerous social biases in LLMs (Ferrara, 2023; Mei et al., 2023), few LLM researchers have examined these biases from the perspective of the psychology of human decision-makinn we study the bias issue LLMs from the new angle of the aforementioned representative heuristic, a concept originating in psychology (Kahneman & Tversky, 1973; Kahneman et al., 1982; Tversky & Kahneman, 1974; 1983).
Thus, in the present work,
Contrary to the very recent work of Suri et al. (2024), which explores the decision-making heuristics of GPT-3.5, our current research emphasizes a more in-depth and comprehensive exploration of the application of the representativenes heuristic within LLMss For example, we have compiled dataset encompassing a much larger number of questions (a total of 202 compared to 9) and we have benchmarked performance across wider range of LLMs based on diverse prompting strategies.
3 Representativeness Heuristic
Drawing from prior research, we organize our research around framework that categorizes the representativenes. heuristic into six types (Kahneman & Tversky, 1973; Kahneman et al., 1982; Tversky & Kahneman, 1974; 1983). These categories vary in their fundamental logical approach and their impact on decision-making processes.
Base Rate Fallacy occurs when individuals overlook or insufficiently account for the pop- ulation base rate of an event (i.e., its overall prevalence or frequency) in favor of specific instances or recent information. Figure 2a presents example where P(B) represents the
3

---

Published as a conference paper at COLM 2024
Aur B.
A.
People may think However, IfA is subset of B, this conclasion
PAND PUD
(a) Base Rate Fallacy
(b) Conjunction Fallacy
(c) Disjunction Fallacy
sample
Smsll sample
Same probability!
(d) Insensitivity to Sample Size
(e) Misconceptions of Chance
(f) Regression Fallacy
Figure 2: Illustrations on six types of representativeness heuristic
proportion of individuals with a symptom (all blue points within the large circle), P(D) denotes the rate of illness (all points within the small circle), and P(B]D) indicates the pro- portion of the sick who have the symptom (all blue points within the small circle). P(D]B) represents the illness rate among those with the symptom. Most people would assume that because P( D) is high, P(D[B) B) would also be high. Yet, according to Bayes' theorem,
P(D[B) = P(BID) P(D)
P(B)
meaning P(D[B) is greatly influenced by the base rates of P(B) and P(D), showing the importance of considering general prevalence in evaluating specific probabilities. An example question can be seen in Table 18 of the Appendix.
Conjunction Fallacy occurs when people mistakenly believe that the chance of two events happening together greater than the chance of either event happening alone. See Figure 2b, consider the example at the beginning of the article. P(A) represents the probability that the person is bank teller (all points within the small circle, less relevant to the description). P(B) is the probability that the person is active in the feminist movement (all points within the large circle, more relevant to the description). P(A n B) the probability that the person is both bank teller and active in the feminist movement (all purple points). P(A B) will always be no larger than ((A) and P(B), no matter which one is closer to the description. An example question can be seen in Table 19 of the Appendix.
Disjunction Fallacy occurs when people incorrectly judge the probability of a broader category to be smaller than one of its specific components. In Figure 2c, we can imagine the small circle representing ice cream and the large circle representing frozen food. Since ice cream is subset of frozen food, the probability of frozen food is higher than ice cream. However, when people talk about summer refreshments, they often think of ice cream rather than frozen food. This choice illustrates common tendency to differentiate specific items from their general classifications based on contextual associations. An example question can be seen in Table 20 of the Appendix.
Insensitivity to Sample Size occurs when people underestimate how important sample size is in data evaluation, potentially leading to incorrect conclusions. Figure 2d presents
4

---

Published as a conference paper at COLM 2024
example where small group (small circle) and large group (large circle) both have 50% blue dots and 50% red dots (2:2 and 12:12). we add one dot to each sample, then the ratio of blue to red becomes 40% and 60% (2:3) in the small group, versus 48% and 52% (12:13) in the large group. It should be recognized that smaller groups are more prone to skewed outcomes because even small changes have larger impact on the overall dynamics of a small group. An example question can be seen in Table 21 of the Appendix.
Misconceptions of Chance involve misunderstanding how randomness works, especially thinking that past outcomes will affect future outcomes in cases where the outcomes are, in fact, independent. For example, Figure 2e presents dice-rolling example. People wrongly believe that if a specific outcome has occurred frequently, it is less likely to happen again shortly, vice versa if it has occurred rarely. However, in truly random events, such as rolling a fair die, the probability any given outcome (1-6) remains constant at 1 unaffected by the sequence of previous results, An example question can be seen in Table 22 of the Appendix.
Regression Fallacy occurs when individuals overlook the natural tendency for extreme situations to move back towards the average (regress to the mean) and instead, erroneously assign this regression to particular cause, Figure 2f. For instance, if an athlete shows lackluster performance following perfect game, might be incorrectly ascribed to external factors, neglecting the likelihood of natural variance in performance. An example question can be seen in Table 23 of the Appendix.
4 Dataset and Experimental Setup
We begin by constructing a dataset intended to reflect a diverse array of questions and scenarios (64.1). Following this, we introduce the models and prompting strategies ($4.2), along with the evaluation methods (54.3).
4.1 Data Collection
Our main resource for creating test questions on representativeness heuristic comes from academic research by Kahneman and Tversky (Kahneman & Tversky, 1973; Kahneman et al., 1982; Tversky & Kahneman, 1974; 1983). Their work introduced various question types and insights into the design of cognitive heuristic measures. Building on their foundation, we design questions that extensively explore the representativeness heuristic in LLMs. Specifically, our test set contains 49 questions drawn directly from previous research and 153 new questions that have been carefully adapted. The reason for creating these new questions is provide diverse contexts and achieve stronger external validity. In creating our adapted items that change the situation and context of the original items, we paid special attention to retaining the essence of the representativeness heuristic being tested.
The total 202 examples are all in English. As shown in Table 1, each example is composed of the following characteristics:
1. Query: Provide the question's 's description and requirements.
2. Type: Indicate the question's category, such Disjunction Fallacy.
3. Feature: Highlight unique aspects, such as choose one, or the question's source, such
as: original problem in Bar-Hillel Neter (1993).
4. Ground Truth: Represents the standard answer.
5. Human Response: Document the outcomes of human responses from academic research when available, or denoting N/A otherwise.
4.2 Models and Prompting Strategies
We investigate four LLMs, encompassing both closed-source and open-source models: GPT- 3.5 (gpt-3.5-turbo-e 5-turbo-061 0613)) (Ouyang et al., 2022), GPT-4 (gpt-4-0613) (OpenAI, 2023), PaLM

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
Danielle is sensitive and introspective. In high school, she wrote poetry secretly. Did her military service as a teacher. Though beautiful, she has little social life since she prefers to spend her time reading quietly at home rather than partying. What is she most likely to study? Choose answer that follows: [1] Literature, [2] Humanities, [3] Physics, [4] Natural science.
Query
Type
Disjunction Fallacy
Feature
Choose one, original problem in Bar-Hillel & Neter (1993)
Ground Truth
[2]
Humanities
Human Behavior
Disjunction Fallacy Rate 50%
56%
Table 1: Exemplar in REHEAT
(chat-bison-001) (Anil et al., 2023), and LLaMA (llama-2-70b-chal (Touvron et al., 2023). We apply four different prompting strategies to each of them to generate answers.
Standard: We ask the model to directly answer the query without explicit reasoning instruc- tions with greedy decoding method.
Zero-shot Chain-of-Thoughte (CoT): We first ask the model to generate its reasoning with instruction?, then direct it to answer the query with the context of reasoning steps This two-step CoT strategy is developed based on CoT (Wei et al., 2022) and its zero-shot variation (Kojima al., 2022).
We prompt the model to generate ten answers with a temperature sampling parameter of T 0.7, using majority voting finalize the model decision via diverse reasoning paths. This prompting strategy is also known as self-consistency prompting (Wang et al., 2023).
Self-Consistency:
Few-shot In-Context Learning (ICL): The model will be prompted with few selected examples from the same category of representativeness heuristic that the model can learn a task from demonstrations, Samples used as exemplars will not participate in accuracy calculations. This ability to learn from context known in-context learning (Brown al., 2020).
4.3 Evaluation Methods
Automatic evaluation: All query questions REHEAT are either in multiple-choice or ranking format. We adopt precise prompt templates in Table 14 of the Appendix to instruct models to generate responses. For a multiple-choice question, a response is deemed correct if and only if contains the ground-truth option. For ranking question, the exact match between relative permutations of response options and ground-trui options qualifies for true positive model prediction.
In addition, one of the present authors with expertise in psychology conducted the human evaluation to assess the output reasoning steps from zero-shot CoT prompting. We report the proportions of the four possible LLM outcomes: both reasoning and prediction correct; the reasoning is correct, but the prediction is incorrect; the
Human evaluation:
2Step 1: Let's think step by step, but don't give the answer directly.
SStep 2: Therefore, the answer is.

---

Published as a conference paper at COLM 2024
prediction is correct, but the reasoning is incorrect; both reasoning and prediction incorrect.
5 Experimental Results
In addition to evaluating the models' performance on the REHEAT dataset and analyzing their reasoning abilities ($5.1; (5.2), we further investigated the potential performance boost that could be achieved by providing hints to the models ($5.4), as well as how situational similarity influences model performance ($5.3).
5.1 LLMs Exhibit the Representativeness Heuristic
We report the model's performance on the REHEAT dataset with different prompting strate- gies, shown in Table 2. Although GPT-3.5 shows the strongest performance in standard prompting, which we will elaborate on later in $5.2, the advanced prompting methods of zero-shot CoT, self-consistency. and one-shot ICL have negative impact on its performance. GPT-4 and LLaMA-2-70B benefit from these prompting strategies with noticeable growth in prediction accuracy, but it is still not substantial. LLaMA2-70B demonstrates the most significant improvement (+14.5%) with one-shot ICL prompting, possibly due to the stan- dard prompting on LLaMA2-70B results in response formatting issues. PaLM 2 is the least effective LLM, with its performance lagging behind other models using standard prompts, despite applying prompt strategies.
How Much Can In-Context Learning Help? As shown in Table 2, one-shot ICL performs the best among various prompting strategies. This has piqued our curiosity: Could providing more ex- amples further improve performance?
50
GPT-35
45
40
.35
Accuracy
30
To explore whether LLMs can acquire knowledge related to the representative- ness heuristic through ICL, we report the k-shot performance of four types of 5, both
25
20
15
LLMs in Figure 3. From GPT-4 LLaMA-2 exhibit noticeable improvement accuracy from adding an example in context. However, such an uptrend becomes saturated and be- gins to fluctuate with further increases in the number of shots. In contrast, GPT- 3.5 and PaLM do not display clear pattern of improvement with the addition of more in-context examples, indicating a weak or non-existent correlation between the number of exemplars and accuracy.
10
Few-shot Num.
Figure 3: In-context learning accuracy of four se- lected LLMs on REHEAT with few-shot demon- strations.
Error Type Analysis We also check each type of representativeness heuristic (Table 7 10 of the Appendix). We have observed that in most cases, most models perform relatively poorly when dealing with questions of Conjunction Fallacy and Disjunction Fallacy. Compared to other types of questions, such as Base Rate Fallacy, the challenge with Conjunction Fallacy and Disjunction Fallacy lies in the fact that the statistical reasoning required is embedded within the connotations and combinations of texts (e.g., South Africa is a subset of Africa) rather than being directly indicated by expressions like large sample size far more than.
Do LLMs Possess Statistical Knowledge? We examine LLMs' knowledge of statistical principles in answering SPQs. The four models possess statistical knowledge, demonstrating a comprehensive understanding across all categories of statistical principles (Table of the Appendix; SPQs example in Table 15 of the Appendix). This significantly differs from our
7

---

Published as a conference paper at COLM 2024
Strategy
Model
Standard
Zero-shot CoT
Self-Consistency One-Shot ICL
GPT-3.5
34.7
30.7 (4.00
35.6 (+0.9)
32.1 (-2.6)
GPT-4
31.2
35.1 (+3.9)
34.2 (+3.0)
36.4 (+5.2)
LLaMA2-70B
27.2
29.2 (+2.0)
29.2 (+2.0)
41.7 (+14.5)
2
14.8
12.4 (-2.4)
20.8 (+6.0)
19.7 (+4.9)
Table 2: Model prediction accuracy (%) on REHEAT. The relative performance of different prompt strategies with respect to standard prompting is indicated with script-size fonts.
Model
Type
Human
GPTT-3.5
GPT-4
LLaMA2-70B
PaLM
Base Rate Fallacy
27.7
31.9 (+4.2)
34.0 (+6.3)
23.4 (4.3)
19.1 (-8.6)
Conjunction Fallacy
24.4
31.1 (+6.7)
20.0 (-44)
20.0 (-4.)
2.2 (-22.2)
Disjunction Fallacy
12.5
22.9 (+10.4)
6.3 (-6.2
10.4 (-2.1)
2.3 (-10.2)
Insensitivity to Sample Size
33.3
45.1 (+11.8
54.9 (+21.6)
43.1 (+9.8)
22.0 (-11.3)
Misconceptions of Chance
50.0
50.0 (-)
50.0 (=)
75.0 (+25.0)
50.0 (=)
Regression Fallacy
85.7
71.4 (-14.3)
71.4 (-14.))
71.4 (-14.3)
71.4 (-14.3)
Average
27.2
34.7 (+7.5)
31.2 (+4.0)
27.2 (=)
14.8 (-12.4)
Table 3: Human performance REHEAT, A comparison with standard results from LMs.
observations on the accuracy of the four models' performance on RHQs, indicating that the models indeed made errors related to the representativeness heuristic.
We also present human performance on REHEAT in Table 3. Through the Prolific platform, we recruited 153 participants (66 males, 86 females, and other) to answer questions. The participants' ages ranged from 19 77 years (M 37.2, SD 12.6). The racial distribution of the participants was follows: 70.6% White, 9.8% African American, 7,8% Asian, and 11.8% other. Each participant was asked to respond to 3-4 different representative heuristic questions, providing three responses for each REHEAT question. We used majority voting to finalize the human decision. It can be observed that the human results closely matched the LLaMA2-70B result, while the GPT series performed slightly better than the human results. This aligns with the findings of Bar-Hillel & Neter (1993), Kahneman & Tversky (1973), Kahneman et al. (1982), Tversky & Kahneman (1974), and Tversky & Kahneman (1983), who reported that human accuracy across various tasks ranges from 10% 50%.
Human Performance on REHEAT
Overall, LLMs have demonstrated a representativeness heuristic bias similar to humans. Example questions and model answers are in Table 18 -2 of the Appendix.
5.2 Discrepancies in Model Reasoning: Beyond Predictions
While testing the REHEAT dataset, we also conducted human evaluations on the reasoning steps produced by the GPT-4 and LLaMA2-70B models under CoT prompting, As before, we form four possible combinations of outcomes, as shown in Figure 4 and Table 11 of the Appendix.
8

---

Published as a conference paper at COLM 2024
GPT-4
LLaMA2-708
100
80
Percentage
20
0
Base Rate Fallacy
Conjunction Fallacy
Disjunction Fallacy
Insensilivity to Sample Size
Misconcepitons of Chance
Regression Fallacy
Base Rate Fallacy
Conjunction Fallacy
Falllcy Disjunction Fallacy
Insensitivity to Sample Size
Misconceptions of Chance
Regression Fallacy
Condition
Figure 4: The combination of reasoning and outcome correctness for GPT-4 and LLaMA2- 70B on the REHEAT via CoT prompts. Detailed Table 11 in Appendix.
Generally, GPT-4 outperformed LLaMA2-70B in all types of RHQs. However, this improve- ment is accompanied by a side effect: an increased proportion of instances where GPT-4's reasoning process is accurate, but the outcome is incorrect. This primarily occurs because when faced with ambiguous questions, GPT-4 often preferred to express the need for addi- tional information before making decision. This is also why GPT-4 accuracy slightly lower than GPT-3.5 under some prompts. Although we mark such cautious answers as incorrect outcomes berouue they did not adhere to the directive of providing a definitive answer-caution answers may remain beneficial in practical settings. Conversely, LLaMA2- 70B more frequently produces correct outcomes from incorrect reasoning processes; the reasons are often incorrectly based on stereotype rather than the problem's 's description.
We have conducted case studies on some interesting reasoning, we include the case study in Appendix A. In summary, our findings indicate that LLMs are influenced by semantic content in their reasoning processes, aligning with results from previous research (Acerbi Stubbersfield, 2023; Dasgupta et al., 2022). Specifically, when the semantic content of tasks was clear and did not contain sensitive content (e.g. gender, race), the models' responses were better.
5.3 Improving Performance by Hinting LLMs to Use Their Knowledge
How can the model's cognitive process be put back on track? We test whether LLMs show enhanced performance with prompts that hint for them to use their existing knowledge. We tested two types of hints: one general and the other based on more detailed cues for each representativeness heuristic type (Table 17 the Appendix). These prompts aimed to hint at the model to recall the knowledge it possessed. Results presented in Table 4. We also provide detailed results for each type of representativeness heuristic in Table 12 and Table 13 of the Appendix. We found that both types of hints provided noticeable improvement for most models, with the specific type of hints providing a more significant boost in performance compared to the general prompts.
5.4 The Impact of Situational Similarity on Model Performance
After observing the performance differences between the model statistical SPQs and the contextual RHQ counterparts found in REHEAT, we delve further into the middle of these two questions Intermediate Questions (IQs) (Table 16 of the Appendix). This type of question integrates specific situational contexts and statistical data, making it necessary to consider both concrete data and the potential impact of the situation when making decisions.

---

Published as a conference paper at COLM 2024
Model
Type
GPT-4
LLaMA2-70B
GPT-3.5
PaLM
Standard
31.2
27.2
34.7
14.8
General
36.6(+5.4)
27.7(+0.))
36.1(+14)
15.6(+0.)
Specific
46.5(+15.3)
31.1(+399)
44.1(+9.4)
26.6(+11.8)
Table 4: Performance of four models after incorporating general and specific prompts that remind the model of its existing knowledge. See detailed results for each type of representa- tiveness heuristic in Table 12 and Table 13 of the Appendix. The relative performance of different prompt strategies with respect to standard prompting is indicated with script-size fonts.
For example, for Linda's problem, we assign a probability each independent event to offer the model with more explicit statistical information. Nevertheless, the model has to infer the information based on the similarity manifested in the meaning of the statements in RHQs. The performance of the models on IQs is reported in Table 6 of the Appendix. Compared with SPQs' results (Table 5 of the Appendix), more errors were made for iQs. This further indicates that introducing scenarios interferes with the model's statistical decision- making process. This also explains why LLMs exhibit the representativeness heuristic more frequently when responding to RHQs.
Conclusion
We introduce a novel dataset (REHEAT) for the representativeness heuristic to assess whether models, like humans, make representativenes heuristic errors in decision-making- specifically, by overlooking known information and relying on similarity for judgment. Our research reveals that even the most advanced LLMSs tend to repeat humans' heuristic mistakes when addressing issues related to the representativenese heuristic, highlighting the necessity for deep understanding of these cognitive biases in data that then propagate into model decision-making processes. Furthermore, we explored how models perform differently when faced with various types of questions. For example, questions regarding the Conjunction Fallacy and Disjunction Fallacy present significant challenge. This is due to the model's s difficulty discerning the latent probabilistic relationships embedded within the text. We also found that hints designed to stimulate the model to recall its existing knowledge can, to some extent, enhance its performance. More specific and detailed prompts tend to lead the model to demonstrate better performance. However, although this method is effective, the model's potential is far from fully tapped, and there remains significant room for research exploration and improvement.
Ethics Statement
The dataset involved in this work does not contain any sensitive information, such as person- ally identifiable information. It should be noted that although the output generated by the model could potentially contain harmful or biased information, based on our experimental observations, such situations have not occurred. We commit to continuously improving technology efficiency and ensuring our research promotes a fairer, safer, and more inclusive society.
References
Alberto Acerbi and Joseph M Stubbersfield. Large language models show human-like content biases in transmission chain experiments. Proceedings of the National Academy of Sciences, (31144)))231377012 2023.
10

---

Published as a conference paper at COLM 2024
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arv preprint arXiv:2305.10403 2023.
Maya Bar-Hillel and Efrat Neter. How alike is it versus how likely is it: A disjunction fallacy in probability judgments. Journal of Personality and Social Psychology, 65(6):1119, 1993.
Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604, 2018. doi: 10.1162 la00___00011.. URL https:
//aclanthology.com org/ org/Q18 /018-111111 1041.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623, 2021.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer woman is to homemaker? debiasing word embed- dings. Advances in neural information processing systems, 29, 2016.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: http:///////oooociggg....o neurippsss..nriis....... cc/ cc/paper/2020/ hast:/1557codbbrrdd444441114 1atpoeaaaaa2444a--aaaaaaa-0 42f6444-bssracc html.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science 356:183-186, 2017. doi: 10.1126/ science.aal2230 URL https: //www. science.......... org/ httg////ppaaageggg pdf /10.
aa14230.
Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James McClelland, and Felix Hill. Language models show human-like content effects on reasoning. ar Xrviv preprint ar arXin:2007 00701 2022.
Emilio Ferrara. Should chatgpt be biased? challenges and risks of bias in large language models. First Monday, 2023. doi: 10.5210/fm.v28in 2m1.11811..3334.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, (55116):33335E364 2018
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 2018. doi: 10.1145/3458723.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint ar 2rXi::2200.16610, 2022.
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. Social biases in NLP models as barriers for persons with disabilities. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5491-5501, Online, July Senguiistiss,ppaa
2020. Association for Computational Linguistics. doi: 10.18653 / vv1 2020 aal-aaain-4444 URL attps://aclanthane
/ 2020 ael-main.4877
11

---

Published as a conference paper at COLM 2024
Daniel Kahneman and Amos Tversky. On the psychology of prediction. Psychological review, 80(4):237, 1973.
Daniel Kahneman, Paul Slovic, and Amos Tversky. Judgment under uncertainty: Heuristics and biases. Cambridge university press, 1982.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-223 -22113, 2022.
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. Towards debiasing sentence representations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5502-5515, Online, July 2020. URL
Association for Computational Linguistics. doi:
https://aclanthollee
oroooog///// 2020. acl -mcin.iin.48 488.
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. Miti- gating political bias in language models through reinforced calibration. AAAI Conference on Artificial Intelligence, 2021. doi: 10.1609/ htt///aii..3//// aaai.v35517.177444
Kaiji Lu, Piotr (Peter) Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. Logic, Language, and Security, 2018. doi: 10.1100/9978---00000 0000//999----00777---2
Anne Maass. Linguistic intergroup bias: Stereotype perpetuation through language. Advances in Experimental Social Psychology, 31:79-121, 1999. URL https://api.
semanticscholar httpttiiccccoaa..or/oroooree /rgp/orpp//:: 42809360.
Katelyn X. Mei, Sonia Fereidooni, and Aylin Caliskan. Bias against 93 stigmatized groups in
masked language models and downstream sentiment classification tasks. arv preprint arXiv: 2306.05550, 2023.
OpenAI. GPT-4 technical report. CoRR, abs abs//303.08774 2023. doi: 10.48550 ARXT9//3XXX 08774. URL https://doi..rr//// 4a855//rr.../........... 3.330.0..77....
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Alexey Romanov, Maria Dr-Artaa Hanna M. Wallach, J. Chayes, C. Borgs, A. Choulde- chova, Geyik, K. Kenthapadi, Anna Rumshisky, and A. Kalai. What's in a name? reducing bias in bios without access to protected attributes. North American Chapter of the Association for Computational Linguistics, 2019, doi: 10.18653/v1/N19111
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked babysitter: On biases in language generation. arXiv preprint 1XXv::909.00326,, 2019.
Steven J Spencer, Christine Logel, and Paul G Davies. Stereotype threat. Annual review of psychology, 07:415-4377 2016.
Gaurav Suri, Lily R Slater, Ali Ziaee, and Morgan Nguyen. Do large language models show decision heuristics similar to humans? a case study using gpt-3.5, Journal of Experimental Psychology: General, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092 2023.
Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics thinking under uncertainty. science, 185 1414 1974.
12

---

Published as a conference paper at COLM 2024
Amos Tversky and Daniel Kahneman. Extensional versus intuitive reasoning: The conjunc- tion fallacy in probability judgment. Psychological review, 90(4):293, 1983.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openrevieww net wet/pdf?id=11 1LL
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Adoances in neural information processing systemss, 35:24824-24831 2022.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John
Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and lason Gabriel. Taxonomy of risks posed by language models. In FAccT 22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 24, 2022, pp. 214-229. ACM, 2022. doi: //doi. ahoii.org/1... 1145/3531 14611146.222. 05513/351116..33388...
URL https:
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Kei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. Conference On Empirical Methods In Natural Language Processing, 2017. doi: 10. (08853/v1/D17///11/
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. NAACL, 2018.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. Gender bias in contextualized word embeddings. Xie preprint wxXim:194400331 2019.
13

---

For example, in Table 25 and Table 26, gender-related content is mentioned, the prevailin ing stereotype is that boys/girls are more likely to.... However, Table 25 does not contradict the common assumption, boys more likely to be involved in sports, Published as a conference paper at COLM 2024
A
Case Study
Here, we want to show some cases where the reasoning may be wrong, but the conclusion surprisingly correct. These examples are very interesting and reflect some unique and exciting behavior patterns of the model when processing information.
A.1 Counter-intuitive Question
Counter-intuitive question refers to a question that challenges common sense, expectations, or widely held beliefs. The following is an example of counter-intuitive question; see Table 24:
"In city, most residents use bicycles to get around, while only small percentage of people choose cars as their daily transportation. It is often assumed that car users are more concerned about the environment and healthy lifestyles, while bicycle users may be more concerned about confort and convenience. If we learn that a particular resident is very concerned about the environment and health, is that person more likely to be an automobile user or a bicycle user? [1] automobile user, [2] bicycle user.
Generally speaking, in human perception, car users are more concerned about comfort and convenience, and bicycle users are more concerned about the environment and healthy lifestyles; the opposite statement is given in the question. This is a question of the Base Rate Fallacy type. Based on the description given in the question, most residents use bicycles to get around, while only small percentage of people choose cars as their daily transportation. We know that most residents should use bicycles for daily commuting. The question asked If we learn that a particula resident is very concerned about the environment and health, is that person more likely to be an automobile user or a bicycle user? The answer based on base rate should be [2] bicycle user, and the answer based similarity should be [1] automobile user.
Most of the models' answers are [2] bicycle user, which is correct. However, there some interesting problems in reasoning. The model will ignore the counter-intuitive assumption given in the question and instead base it on the common-senss assumption that bicycle users are more concerned about the environment and healthy lifestyle (see the teal bold line in Table 24). Even under prompting like COT, which can greatly improve reasoning accuracy, most models only mention the counter-intuitive assumption given in the question, and some do not mention it at all. They often consider this assumption incorrect and complete the reasoning based on common-sense assumptions. We hypothesize that this is because the training data is rich in examples and discourse that emphasizes cycling as an environmentally friendly and healthy mode of transportation. This common-sense assumption becomes the anchor for the model to answer questions, even when faced with specific problem that sets conditions that contradict it.
This implies that, despite the significant improvements in LLMs' modeling and application capabilities, completely freeing them from the constraints common-sense assumptions remains a challenge. Therefore, when designing and using these models, we need to consider how to balance the model's modeling of specific contexts and the application of general common sense.
A.2 Sensitive Content Related Question (e.g. gender, race)
Sensitive content related questions, such as those about gender and race, when combined with the previously mentioned counter-intuitive questions, have led some intriguing phenomena. Here, we will provide three examples (due to the length of the question, see Table 25, Table 26, and Table 27).
while girls are likely to be more involved in the arts and theater, whereas Table
26 is counter-intuitive assumption, girls are more likely to be involved in sports,
14

---

Published as a conference paper at COLM 2024
while boys are likely to be more involved in the arts and theater. Table 27, the other hand, does not mention gender-related content but has counter-intuitive information.
All three examples are Base Rate Fallacy type, so according to the base rate descriptions of the three questions, the answers to the three are, [1] sports program, [2] arts and drama club, and [2] arts and drama club. For Table 25 and Table 26, mentioning gender- related stereotypes makes models more vigilant, improving the accuracy of reasoning and conclusions. Especially for Table 26, unlike Table 27, the models do not default to common-sense assumptions when sensitive content is present.
We speculate that this phenomenon may originate from the model specifically programmed to recognize information related to sensitive content, thereby exhibiting exceptionally high sensitivity and vigilance when processing such information. This leads to increased correct- ness in reasoning and conclusions when the model deals with questions related to sensitive content. Since the model's attention to sensitive content can improve its reasoning and conclusion correctness when answering representative heuristics questions, this approach may be an effective way to enhance the model's 's accuracy in dealing with such questions.
B Limitation
This work is an initial attempt to introduce the representativeness heuristic into NLP and computational social science research, providing a new perspective for understanding the LLM behavior on the representativeness heuristic question. Nonetheless, the study has several significant limitations. Firstly, the exploration of representativeness heuristic in its early stages, facing multiple challenges in practical applications. These include accurately quantifying and identifying representativeness heuristic behaviors and effectively utilizing these findings in different NLI' tasks. Secondly, the datasets used in the existing research are relatively limited in size, and the human heuristic phenomenon encompasses not only the representativennss heuristic but also others, such as the anchoring heuristic and availability heuristic. We plan to add more questions designed around the representativeness heuristic in the future and gradually build datasets for other heuristics. Additionally, the evaluation of reasoning can be further refined, including understanding conceptual errors (such as misunderstanding Bayes' theorem), computational errors, and more. This paper evaluates the reasoning from a broad perspective. By adopting more specific viewpoint, we might discover new insights into the behavior of LLMs. Another important limitation is that the current research primarily focuses on English datasets and models, limiting its global applicability and impact. The diversity of languages and cultures requires the verification and expansion of these findings across different linguistic and cultural backgrounds.
C Additional Tables
Model
Type
GPT-3.5
GPT-4 LLaMA2-70B PaLM 2
Base Rate Fallacy
Conjunction Fallacy
Disjunction Fallacy
Insensitivity to Sample Size
Misconceptions of Chance
Regression Fallacy
Table Model performance on six Statistical Prototype Questions resulting from five iterations of self-consistency.
15

---

Published as a conference paper at COLM 2024
Model
Type
GPT-3.5 GPI-4
LLaMA2-70B
PaLM
Base Rate Fallacy
X
X
Conjunction Fallacy
X
X
X
X
X
EXPL
Disjunction Fallacy
X
Insensitivity to Sample Size
V
Misconceptions of Chance
X
X
V
Regression Fallacy
V
Table 6: Model performance on Intermediate Questions resulting from five iterations of self-consistency.
Strategy
Type
Standard
Zero-shot CoT
Self-Consistency
One-Shot
Base Rate Fallacy
31.9
38.3 (+6.4)
36.2 (+4.3)
26.1 (4.8)
Conjunction Fallacy
31.1
33.3 (+2.2
33.3 (+2.2)
15.9 (-152)
Disjunction Fallacy
22.9
18.8 (4.1)
22.9 (+)
40.4 (+17.5)
Insensitivity to Sample Size
45.1
25.5 (-19.6)
43.1 (2.0)
44.0 (1.1)
Misconceptions of Chance
50.0
25.0 (:25.0)
50.0 (-)
N/A (N/A)
Regression Fallacy
71.4
85.7 (+14.3)
71)
N/A (N/A)
Table GPT-3.5 performance on REHFAT. The Ont- was evaluated on the
'Misconceptions of Chance' and 'Regression Fallacy' questions, due to their limited quantity. This limited quantity would also lead to significant fluctuations accuracy changes for these question types. The relative performance of different prompt strategies with respect to standard prompting is indicated with script-size fonts.
Strategy
Type
Standard
Zero-shot CoT
Self-Consistency
One-Shot
Base Rate Fallacy
34.0
27.7 (4.3)
36.2 (+2.2)
37.0 (+3.0)
Conjunction Fallacy
20.0
26.7 (+6.7)
24.4 (H4.4)
27.3 (+7.3
Disjunction Fallacy
6.3
10.4 (+4.1)
10.4 (+4.1)
4.3 (-2.0
Insensitivity to Sample Size
54.9
64.7 (49.8)
58.8 (+3.9)
74.0 (+19.1)
Misconceptions of Chance
50.0
50.0 (=)
50.0 (-)
N/A (N/A)
Regression Fallacy
71.4
85.7 (414.3)
57.1 (-14.3)
N/A (N/A)
Table 8: GPT-4 performance on REHEAT. The One-Shot approach was not evaluated on the Misconceptions of Chance' and 'Regression Fallacy' questions, due to their limited quantity. This limited quantity would also lead to significant fluctuations in accuracy changes for these question types. The relative performance of different prompt strategies with respect to standard prompting is indicated with script-size fonts.
16

---

Published as a conference paper at COLM 2024
Strategy
Type
Standard
Zero-shot CoT
Self-Consistency
One-Shot
Base Rate Fallacy
23.4
31.9 (48.5)
29.8 (+6.4)
21.7 (-1.7)
Conjunction Fallacy
20.0
31.1 (+11.1)
24.4 (H4.4)
27.3 (+7.3)
Disjunction Fallacy
10.4
12.5 (+2.1)
10.4 (=)
63.8 (453.4)
Insensitivity to Sample Size
43.1
35.3 (778)
43.1
52.0 (+89)
Misconceptions of Chance
75.0
50.0 (-2..0
50.0 (-25.0)
N/A (N/A)
Regression Fallacy
71.4
57.1 (-14.)
(-)
N/A (N/A)
Table 9: I.L.MA2-7BB performance on RвHEAT. The One-Shot approach was not evaluated on the Misccnccptiios of Chance' and Regression Fallacy' questions, due their limited
quantity. This limited quantity would also lead to significant fluctuations in accuracy changes for these question types. The relative performance of different prompt strategies with respect to standard prompting is indicated with script-size fonts.
Strategy
Type
Standard
Zero-shot CoT
Self-Consistency
One-Shot
Base Rate Fallacy
19.1
17.0 (221)
21.3 (+2.2)
23.9 (+48)
Conjunction Fallacy
2.2
2.2 (-)
2.2 (4)
7.3 (45.1)
Disjunction Fallacy
2.3
4.2 (41.9)
4.2 (+1.9)
14.6 (+12.3)
Insensitivity to Sample Size
22.0
19.6 (2.4)
39.2 (+17.2)
30.0 (+8.0)
Misconceptions of Chance
50.0
50.0 (=)
75.0 (+25.0)
N/A A (N/A)
Regression Fallacy
71.4
28.6 (-42.)
85.7 (+14.3)
N/A (N/A)
Table 10: PaLM 2 performance on REHEAT. The One-Shot approach not evaluated on the "Misconcetiioss of Chance' and 'Regression Fallacy' questions, due their limited
quantity. The limited quantity would also lead to significant fluctuations in accuracy changes for these question types. The relative performance of different prompt strategies with respect to standard prompting is indicated with script-size fonts.
GPT-4
LLaMA2-70B
Type
TT
TF
FT
FF
TT
TF
FT
FF
Base Rate Fallacy
25.6
14.9
2.1
57.4
19.1
4.3
12.8
63.8
Conjunction Fallacy
26.7
0.0
0.0
73.3
17.8
0.0
13.3
68.9
Disjunction Fallacy
8.3
0.0
2.1
89.6
10.4
0.0
2.1
87.5
Insensitivity to Sample Size
64.7
3.9
0.0
31.4
29.4
0.0
5.9
64.7
Misconceptions of Chance
50.0
0.0
0.0
50.0
50.0
0.0
0.0
50.0
Regression Fallacy
85.7
0.0
0.0
14.3
57.1
0.0
0.0
42.9
Table 11: The detailed combination of reasoning and outcome correctness for GPT-4 and LLaMA2-70B on the REHEAT via CoT prompts.
17

---

Published as a conference paper at COLM 2024
GPT-4
LLaMA2-70B
Type
Standard General
Specific Standard General
Specific
Base Rate Fallacy
34.0
44.7(+107) 53.2(+19.2)
23.4
23.4(=)
14.9(-8.)
Conjunction Fallacy
20.0
20.0(=) 40.0(+20.0)
20.0
24.4(+-.4)
31.1(+11.))
Disjunction Fallacy
6.3
10.4(+4.1)
14.6(+8.3)
10.4
14.6(-4.2)
14.6(=)
Insensitivity to Sample Size
54.9
62.7(+7.8)
70.6(+15.7)
43.1
(7.8(558))
51.0(479)
Misconceptions of Chance
50.0
50.0(=)
50.0(=)
75.0
(0.0(2250)
50.0(=)
Regression Fallacy
71.4
71.4(-)
85.7(+14.3)
71.4
85.7(+14.3)
200.0(+28.)
Table 12: Performance of GPT-4 and LLaMA2-70B after incorporating general and specific prompts that remind the model of its existing knowledge. The limited number of questions on 'Misconceptions of Chance' and the 'Regression Fallacy' could lead to significant fluctua- tions in accuracy changes for these question types. The relative performance of different prompt strategies with respect to standard prompting is indicated with script-size fonts.
GPT-3.5
PaLM 2
Type
Standard
General
Specific Standard General
Specific
Base Rate Fallacy
31.9
34.0(+2.))
42.6(+10.7)
19.1
17.0(2.1)
25.5(+6.4)
Conjunction Fallacy
31.1
24.4(-6.7)
35.6(+4.5)
2.2
2.2(=)
4.4(+2.2)
Disjunction Fallacy
22.9
(7.5(+14.6)
(7.5(+14.6)
2.3
4.3(+2.0)
6.7(44.4)
Insensitivity to Sample Size
45.1
37.3(-78)
51.0(+5.9)
22.0
22.0(=)
52.9(+3099)
Misconceptions of Chance
50.0
50.0())
50.0())
50.0
75.0(+25.0)
50.0(225.0
Regression Fallacy
71.4
100.0(+28.6
100.0(+28.)
71.4
85.7(+143)
100.0
Table 13: Performance of GPT-3.5 and PaLM 2 after incorporating general and specific prompts that remind the model of its existing knowledge. The limited number of questions on 'Misconceptions of Chance' and the 'Regression Fallacy' could lead to significant fluctua- tions in accuracy changes for these question types. The relative performance of different prompt strategies with respect to standard prompting is indicated with script-size fonts.
18

---

Published as a conference paper at COLM 2024
TYPE
PROMPT
For multiple-choice questions, choose the that you think is best, even you think more than one option will work, choose the one you think is best. The answer should be accompanied by the number gave for each option. For example, when the options are [1] choicel, [2] choice2, [3] choice3, if [1] choicel is the correct answer, the answer format is [1] choicel.
Multi-Choice
For ranking questions, provide ranking, and make distinctions even if you think multiple options may have the same position in the ranking. When displaying the answer, you must provide the number assigned to each option in the original question. For example, when the options are [1] choicel, [2] choice2, [3] choice3. If you think [2] ranks before [1], and [1] ranks before [3], the answer format is [2] choice2 > [1] choicel [3] choice3.
Ranking
Table 14: System messages for different types of query questions.
19

---

Published as a conference paper at COLM 2024
TYPE
QUESTIONS
The probability of an event occurring in a general population 1 in 10,000 000 (P(A) ToXOr ) A test designed to detect this event has: A true positive rate (sensitivity) of 99% (P(B) A) = 0.99), meaning it will correctly identify the event 99% of the time when it occurs. A true negative rate (specificity) of 99% (P(-B -A) = 0.99), meaning it will correctly identify the non-occurrence of the event 99% of the time when the event does not occur. An individual randomly selected, and the test indicates that the event is present (positive result). The question is to calculate P)A[B), the probability that the event truly occurs given that the test indicates it does. Is P)A [1] Greater than 50, [2] Less than 50%?
Base Rate Fallacy
Given two independent events and B, where the probability of A is 0.8 and the probability of B is 0.2, compare which is greater: the probability of event B or the probability of events A and B occurring together.
Conjunction Fallacy
Given two main categories A and B with probabilities P(A)=0.7 and P(B)=0.2, and two subsets (of A) and b (of B), P(a) P(B), rank the probabilities P(A), P(B), P(a), and P(b) based on the given and inferred hierarchical structure of probabilities.
Disjunction Fallacy
Given two populations, A and B, where A generates an average of 45 events per period and B generates an average of 15 events per period, and each event has a binary outcome with a base probability of 0.5, over long-ter term observation year), which population is expected to Insensitivity to Sample Size show greater variability in the proportion of periods during which the frequency of one specific outcome exceeds 60% of the events? Choose the most probable answer: [1] Population A (the larger population). [2] Population B (the smaller population). [3] The variability in both populations is about the same (that is, within 5% of each other).
Consider sequence of 10 independent events, each with two possible outcomes: or B. The outcomes are equally likely, with the probability of each outcome being 0.5. Given that the first events have all resulted in outcome A, what is the most likely final distribution of outcomes A and B after all ten events have occurred? Choose the most probable answer from the following: [1] 5 As and 5 Bs, [2] 10 As and 0 Bs, [3] The probability for both distributions the same.
Misconceptions of Chance
Civen a sequence of performance measurements with an average value of X, and observing that recent measurements significantly exceed X, followed by sudden drop to value of Y below X, which of the following is the most statistically probable explanation for the sudden drop, assuming
external factors known to have influenced this change? [1] A specific external factor negatively impacted this particular measurement. [2] A random fluctuation within the expected variability of performance measurements. [3] A systematic change occurred but is not immediately apparent from the data provided. [4] The measurement at Y an outlier caused by an unaccounted-for error in measurement.
Regression Fallacy
Table 15: Statistical Prototype Questions.
20

---

Published as a conference paper at COLM 2024
TYPE
QUESTIONS
A disease has an incidence rate of in 10,000 in the population. A test for the disease is 99% accurate, meaning it has a 99% chance of correctly identifying the disease if it is present (true positive rate) and 99% chance of correctly identifying the absence of the disease if it is not present (true negative rate). An individual tests positive for the disease. Question: What is the probability that the individual actually has the disease? Is it [1] greater than 50 percent or [2] less than 50 percent?
Base Rate Fallacy
Bill is 34 years old. He is intelligent, but unimaginative, compulsive, and generally lifeless. In school, he was strong in mathematics but weak in social studies and humanities. The probability of Bill being physician 0.3; the probability of Bill being an architect is 0.19; the probability of Bill playing poker as a hobby is 0.2; the probability of Bill playing jazz as a hobby is 0.4; the probability of Bill surfing as a hobby is 0.36; the probability of Bill is reporter is 0.25; the probability of Bill is accountant is 0.8; the probability of Bill climbs mountains for hobby is 0.35. We have seven statements below: [1] Bill is a physician who plays poker as a hobby. [2] Bill is an architect. [3] Bill plays jazz as a hobby. [4] Bill surfs as a hobby. [5] Bill is a reporter. [6] Bill is an accountant who plays jazz hobby. Ranked the seven statements' probability from high low.
Conjunction Fallacy
hobby. [7] Bill climbs mountains for
Danielle is sensitive and introspective. In high school, she wrote poetry secretly. Did her military service as a teacher. Though beautiful, she has little social life, since she prefers to spend her time reading quietly at home rather than partying. The probability of her studying Human- ities is 0.7, and the probability of her studying Natural science is 0.2; Literature is the Humanities subset, and Physics is the Natural science subset. Rank the probability of the below options. [1] Literature, [2] Humanities, [3] Physics, [4] Natural science.
Disjunction Fallacy
Approximately 45 babies are born in the large hospital while 15 babies are born in the small hospital. Half (50%) of all babies born in general are boys. However, the percentage changes from day to another. For 1-year period, each hospital recorded the days on which >60% of the babies born were boys. The question posed is: Which hospital do you think the ratio varies more from day to day? Just choose the most probable answer below: [1] The larger hospital. [2] The smaller hospital. [3] About the same (that is, within 5% of each other).
Insensitivity to Sample Size
Two equally matched soccer teams are currently playing 10 games, and one team has won the first 5 games. For each game, each team's 's win rate 50%. Which is the most likely final score of the game? Choose the most probable answer. [1] 5:5, [2] 10:0, [3] is the same probability.
Misconceptions of Chance
basketball player has an average score of points per game over season. The player scores significantly above this average in the last several games, reaching new personal bests. However, in the next game, the player's score drops well below their season average to Y points. Without additional specific information about training, injuries, or team dynamics, which of the following is the most statistically probable explanation for the sudden drop in performance? [1] The player was poorly trained for this particular game. [2] The player sustained an injury that affected their performance. [3] There was discord within the team that impacted the player's performance. [4] No specific reason; it's a normal fluctuation in performance.
Regression Fallacy
Table 16: Intermediate Questions
21

---

Published as a conference paper at COLM 2024
TYPE
HINTS
Please answer this question based on your statistical probability knowledge.
General
Please answer this question based on your statistical probability knowledge, the conditional probability of an event needs to take into account the overall base rate (or prior probability) of that event.
Base Rate Fallacy
Please answer this question based on your statistical probability knowledge, the probability of two specific conditions occurring simultaneously is not higher than that of one specific condition.
Conjunction Fallacy
Please answer this question based on your statistical probability knowledge, the probability of subset of events cannot be higher than the probability of the entire event.
Disjunction Fallacy
Please answer this question based on your statistical probability knowledge, changes in small samples tend to have more pronounced effect on the overall statis- tics than changes in large samples.
Insensitivity to Sample Size
Please answer this question based on your statistical probability knowledge, the probabilities of each inde- pendent event are not related to each other.
Misconceptions of Chance
Please answer this question based on your statistical probability knowledge, a natural relationship between two variables does not imply causation.
Regression Fallacy
Table 17: Hints we use to remind LLMs to use their existing knowledge.
FIELD
DESCRIPTION
A doctor performs test that is 99% accurate, and you test pos- itive for the disease. However, the incidence of the disease is 1/10,000. How do you think you will get the disease? Choose between the following two probabilities: [1] greater than 50 per- cent, [2] less than 50 percent.
Query
Type
Base Rate Fallacy
Ground Truth
[2] less than 50 percent
Model Answer
[2] less than 50 percent.
Table 18: Example of Base Rate Fallacy in REHEAT answer by GPT-3355, Standard prompt
22

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
Bill is 34 years old. He is intelligent, but unimaginative, com- pulsive, and generally lifeless. In school, he was strong math- ematics but weak in social studies and humanities. We have eight statements below: [1] Bill a physician who plays poker hobby. [2] Bill is an architect. [3] Bill is an accountant. [4] Bill plays jazz as a hobby. [5] Bill surfs as a hobby. [6] Bill is a reporter. [7] Bill is an accountant who plays jazz
Query
hobby. [8]
Bill climbs mountains for a hobby. Ranked the eight statements associated with each description by the degree to which Bill resembles the typical member of that class.
Type
Conjunction Fallacy
[3] Bill is an accountant. > [4] Bill plays jazz as a hobby. > [7] Bill is an accountant who plays jazz as hobby.
Ground Truth
Sure, let's break down each statement and consider how well it matches with Bill's description.
1. Bill is physician who plays poker hobby: Being physi- cian requires high intelligence, which matches Bill's description. However, it also often requires a level of creativity and adaptabil- ity, which might not suit Bill's unimaginative and compulsive nature. Poker requires a good understanding of probability and strategy, which could align with Bill's strength in mathematics.
2. Bill is an architect: This profession often requires high creativ- ity and imagination, which doesn't 't align with Bill's description. 3. Bill is an accountant: This profession often requires a high level of detail orientation, compulsiveness, and a strong un- derstanding of mathematics, all of which align well with Bill's description.
4. Bill plays jazz as a hobby: Jazz often requires high creativity and improvisation, which doesn't 't align with Bill's unimagina- tive nature.
Model Reasoning
5. Bill surfs hobby: Surfing is very physical and adventur- ous hobby, which doesn't seem to align with Bill's description of being generally lifeless.
6. Bill is a reporter: This profession often requires high social skills and a strong understanding of humanities, which doesn't t align with Bill's s academic strengths.
7. Bill is an accountant who plays jazz as a hobby: As mentioned before, being an accountant aligns well with Bill's description, but playing jazz does not.
8. Bill climbs mountains for hobby: Mountain climbing is
very physical and adventurous hobby, which doesn't seem to align with Bill's description of being generally lifeless. Now, you can rank these statements based on how well they align with Bill's description.
[3] Bill is an accountant [1] Bill is a physician who plays poker a hobby > [7] Bill is an accountant who plays jazz as a hobby > [2] Bill is an architect > [6] Bill is reporter > [4] Bill plays jazz
Model Answer
as hobby [5] Bill surfs as a hobby > [8] Bill climbs mountains
for hobby.
Table 19: Example of Conjunction Fallacy in REHEAT answer by GPT-4, CoT prompt
23

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
It is quite possible for something to be included in more than one option listed. For example, suppose the list is of foodstuffs, and one option is "frozen foods," " while another is "desserts". In this case, do not interpret "frozen foods" as "frozen foods excluding deserts," nor interpret " "desserts" are not frozen" -icee cream" qualifies as both "frozen food" and "dessert." Danielle is sensitive and introspective. In high school, she wrote poetry secretly. Did her military service as teacher. Though beautiful, she has little social life, since she prefers to spend her time reading quietly at home rather than partying. What is she most likely to study? Choose one answer that fol- lows: [1] Literature, [2] Humanities, [3] Physics, [4] Natural science.
"desserts which
Query
Type
Disjunction Fallacy
Ground Truth
[2] Humanities
Based on the information provided, Danielle most likely to study [1] Literature or [2] Humanities, she has a background writing poetry and teachnng, and prefers to spend her time reading quietly at home. It's less likely that she would study [3] Physics or [4] Natural science, as there is no indication of her having interest or background in these fields. Therefore, the best answer is either [1] Literature or [2] Humanities.
Based the information provided, Danielle is most likely to study [1] Literature. Danielle's s passion for reading and her back- ground in writing poetry suggests that she has strong interest in literature and the humanities. Additionally, her preference for spending time alone reading rather than socializing suggests that she enjoys immersing herself in written works. While it's possible that she may also be interested in other fields such as hu- manities natural sciences, her interests and personality traits make literature the most likely choice for her to study. Therefore, the answer is [1] Literature.
Model Reasoning
Based on the information provided, Danielle most likely to study [1] Literature. Danielle's passion for reading and her interest in poetry suggests that she has a strong appreciation for literature and writing. Additionally, her introspective nature and preference for spending time alone reading rather than partying also indicate that she may be drawn studying literature. The fact that she taught in military service also suggests that she might have an interest humanities, but given her personal preferences and interests, literature seems like the most fitting choice. Therefore, option [1] Literature is the best answer.
Model Answer
[1] Literature
Table 20: Example of Disjunction Fallacy in REHEAT answer by LLaMA2-70B, Self- Consistency prompt
24

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
Suppose there are two restaurants, and the other a small family-run restaurant. Each restaurant records the number of days in a year that customers complain The large chain restaurant serves an average of 1,000 customers per day, while the small family-run restaurant serves about 50 customers per day. Each restaurant is generally considered to have a complaint rate of about 20% The question is: which restaurant is more likely to have recorded customer complaints up 40% over the course of the year? Just choose the most probable answer below: [1] Large chain restaurant. [2] Small family-run restaurant. [3] Both restaurants are roughly the same (i.e., the difference between the two is within 5%).
large chain restaurant
Query
Type
Insensitivity to Sample Size
Ground Truth
[2] Small family-run restaurant.
The answer is: [1] Large chain restaurant. The large chain restau- rant serves an average of 1,000 customers per day, while the small family-run restaurant serves about 50 customers per day. Each restaurant is generally considered to have complaint rate of about 20%. This means that the large chain restaurant is likely to have 200 complaints per day, while the small family-run restaurant is likely to have 10 complaints per day. To calculate the probability of each restaurant having customer complaints up to 40% over the course of the year, we can use the following formula: Probability = (Number of complaints / Total number of customers) 100% For the large chain restaurant, the probability having customer complaints up to 40% is: Probability = (200 / 1000) 100% = 20% For the small family-run restaurant, the probability of having customer complaints up to 40% is: Proba- bility (10 50) 100% 20% As you see, the probability of each restaurant having customer complaints up to 40% is the same. However, the large chain restaurant is more likely to have a higher number of complaints overall, because serves more customers. Therefore, the large chain restaurant more likely to have recorded customer complaints of up to 40% over the course the year.
Model Reasoning
Model Answer
[1] Large chain restaurant
Table 21: Example of Insensitivity to Sample Size in REHEAT answer by PaLM 2, One-Shot prompt
25

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
Consider a regular six-sided die with four green faces and two red faces. The die will be rolled 20 times and the sequence of greens (G) and reds (R) will be recorded. You are asked to select one sequence, from set of three. Please check the sequence of greens and reds on which you prefer to bet. Choose the most probable answer. [1] RGRRR, [2] GRGRRR, [3] GRRRRR, [4] Same probability.
Query
Type
Misconceptions of Chance
Ground Truth
[1] RGRRR
Sure, let's break this down.
First, let's understand the probability of rolling a green or a red. Since there are green faces and red faces, the probability of rolling a green (G) is 4/6 2/3, and the probability of rolling red (R) is 2/6 1/3.
Now, let's look at each sequence:
1. RGRRR: This sequence has 1 green and 4 reds. The probability
Model Reasoning
of this sequence happening is (2/3)² * (1/3)4.
2. GRGRRR: This sequence has 2 greens and 4 reds. The proba-
bility of this sequence happening is (2/3)2 + (1/3)4.
3. GRRRRR: This sequence has green and 5 reds. The probabil-
ity of this sequence happening is (2/3)² (1/3)5.
We can see that the sequence with the most greens, which is
more likely to be rolled, will be the most probable sequence.
Model Answer
[2] GRGRRR
Table 22: Example of Misconceptions of Chance in REHEAT answer by GPT-3.5, Standard prompt
26

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
A basketball player scores higher than before in several games. In the next game, his score gets low. What might be the reason? Choose the reason from below: [1] poorly trained. [2] injury. [3] Discord within the team. [4] No specific reason, normal fluctuation. Choose the most possible answer.
Query
Type
Regression Fallacy
Ground Truth
[4] No specific reason, normal fluctuation.
First, let's consider each option:
1. Poorly trained: This seems unlikely because the player has been scoring higher than before several games, This indicates that the player is well-trained and capable of performing well. 2. Injury: This could be a possible reason. An injury could affect the player's performance. However, we don't have any information about the player being injured.
Model Reasoning
3. Discord within the team: This could also be a possible reason.
Team dynamics can affect player's 's performance. But again, we don't have any information about any discord within the team. 4. No specific reason, normal fluctuation: This is also possible reason. It's normal for a player's performance fluctuate from game to game.
Now, let's rank these options based on the information we have.
Model Answer
[4] No specific reason, normal fluctuation.
Table 23: Example of Regression Fallacy in REHEAT answer by GPT-4, CoT prompt
27

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
In city, most residents use bicycles to get around, while only small percentage of people choose cars as their daily transporta- tion. It is often assumed that car users are more concerned about the environment and healthy lifestyles, while bicycle users may be more concerned about comfort and convenience. If we learn that a particular resident is very concerned about the environment and health, is that person more likely to be an automobile user or a bicycle user? [1] automobile user, [2] bicycle user.
Query
Type
Base Rate Fallacy
Ground Truth
[2] bicycle user
To determine whether person who is very concerned about the environment and health is more likely to be an automobile user bicycle user, can consider the following factors:
n1. Environmental impact: Bicycles are generally considered to have lower environmental impact compared to cars, as they do not emit greenhouse gases or contribute to air pollution. This suggests that someone who is highly concerned about the environment may be more inclined to use a bicycle.
n2. Health benefits: Cycling form of exercise that promotes physical fitness and contributes to a healthy lifestyle. If some- one prioritizes their health and well-being, they may be more likely to choose bicycle as their mode of transportation. n3, City infrastructure: The availability of bike lanes, cycling infrastructure, and the overall fikeddiienls of the city can also influence the choice between a car and a bicycle. has well-developet cycling infrastructure, it may encourage more people to choose bicycles, regardless of their concerns about the environment and health.
Model Reasoning
the city
Considering these factors, it is reasonable to assume that per- son who is very concerned about the environment and health is more likely to be bicycle user rather than an automobile user. However, it is important note that individual prefer- ences and circumstances can vary, and there may be exceptions this general assumption.
Model Answer
[2] bicycle user
Table 24: Case study example for A.1 in REHEAT, sentences highlighted teal are the ones the case study focuses on.
28

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
In diverse high school, most students participate in the school's sports program, and only few participate in the school's arts and drama clubs. The prevailing stereotype is that boys more likely be involved in sports, while girls are likely to be more involved in the arts and theater. If we that student more likely to be a student in a sports program or a member of an arts and drama club? [1] sports program, [2] arts and drama club.
Query
girl, is
Type
Base Rate Fallacy
Ground Truth
[1] sports program
Sure, let's break this down:
1. The information given states that most students participate in the school's sports program. This means that the majority of the student body, regardless of gender, involved in sports.
2. It is also mentioned that only a few participate in the school's arts and drama clubs. This indicates that the number of students involved in arts and drama is significantly less than those in- volved in sports.
Model Reasoning
3. The stereotype is that boys are more likely to be involved in sports, while girls are likely to be more involved the arts and theater. However, this is stereotype and not fact. It does not necessarily reflect the actual distribution of students in the school.
4. Given these points, when we see girl, we have two options consider: [1] sports program, [2] arts and drama club.
Now, based on the information provided, you can make an informed decision.
Model Answer
[1] sports program.
Table 25: Case study example for A.2 in REHEAT, sentences highlighted teal are the ones the case study focuses on.
29

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
In diverse high school, most students participate in the school's arts and drama clubs, and only few participate in the school's s sports program. The prevailing stereotype is that girls are more likely to be involved in sports, while boys are likely to be more involved in the arts and theater. If we see girl, is that girl more likely to be a student in sports program or member of an arts and drama club? [1] sports program, [2] arts and drama club.
Query
Type
Base Rate Fallacy
Ground Truth
[2] arts and drama club
Sure, let's break down:
1. The information given states that most students in the school participate in the arts and drama clubs. This means that regard- less of gender, the majority of students are involved in these clubs.
2. It's also mentioned that only few students participate in the school's sports program. This indicates that the sports program has less participation overall.
Model Reasoning
3. The stereotype mentioned is that girls are more likely to be involved in sports, while boys are more likely to be involved in the arts and theater. However, this is a stereotype and may not necessarily reflect the actual situation in the school.
4. Given these points, when we see girl in this school, we have consider the overall participation rates in the sports program and the arts and drama clubs.
Now, based on the information provided, you can make an informed decision.
Model Answer
[2] arts and drama club.
Table 26: Case study example for A.2 in REHEAT, sentences highlighted in teal are the ones the case study focuses on.
30

---

Published as a conference paper at COLM 2024
FIELD
DESCRIPTION
In diverse high school, most students participate in the school's arts and drama clubs, and only few participate in the school's sports program. The prevailing stereotype is that students who love reading are more likely to be involved in sports, while students who love sports are likely to be more involved in the arts and theater. If we see student who likes to read books, is that student more likely to be student in a sports program or member of an arts and drama club? [1] sports program, [2] arts and drama club.
Query
Type
Base Rate Fallacy
Ground Truth
[2] arts and drama club
Sure, let's break down:
1. The school is diverse and most students participate in arts and drama clubs, while only a few participate in the sports program.
2. The stereotype is that students who love reading are more likely to be involved in sports, and those who love sports are more likely to be involved in arts and drama.
3. We have a student who likes to read books.
Given these points, stereotype, student who loves reading would be more likely to be involved in sports. However, we know that students who love reading may fit more with arts and drama clubs.
might initially think that, according to the
Model Reasoning
So, even though the stereotype suggests that book-loving student might be more likely to be involved in sports, the fact that the student more likely to be involved in arts and drama clubs than in the sports program.
Now, based on the information provided, you can make an informed decision.
Model Answer
[2] arts and drama club.
Table 27: Case study example for A.2 in REHEAT, sentences highlighted teal are the ones the case study focuses on.
31


Reviews:
[{"review_id": "5d311BsnPV", "reviewer": "Reviewer_3FvA", "summary": "This paper presents an evaluation suite testing language models for the \"representativeness heuristic\" cognitive bias. It reviews six types of this heuristic, with illustrations, detailed descriptions, and examples, and reviews the results of testing LMs for these biases. The paper includes results for:\n* Four LLMs,\n* four prompting approaches,\n* varying numbers of few-shot examples,\n* correctness of the models' reasoning,\n* two methods of 'hinting' the models, and\n* two easier sets of related questions (\"SPQs\" and \"IQs\") testing necessary knowledge for solving the problems.\n\nThese experiments show that LMs exhibit qualitatively similar behavior to humans on such questions.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 3FvA,\n\nThank you for your encouraging, thorough, and positive feedback. Below, we address your comments.\n\n- *What would really improve this paper a lot is spending some time discussing construct validity and external validity. Why are we testing models for these kinds of biases? What do we conclude about them when they get these results?*\n\n**We agree that further discussing construct validity and external validity will improve this paper, and we will add further discussion to the final version.**\nTo address construct validity, psychologists develop measures and apply psychometrics to measure individual differences in personal characteristics (e.g., traits, behaviors) as well as individual differences in intra-individual change (e.g., states, behavioral patterns). In a similar spirit of construct validity, here seek to measure the consistency between models, so that any empirical convergence provides indirect evidence of construct validity. Additionally, in psychology, common methods for improving external validity include improving the diversity of samples and improving the psychological fidelity of the data, so that both are more likely to generalize across situations. In this vein, when we explicitly attempt to provide diverse contexts for each type of representativeness heuristic and more realistic descriptions. **We test the model's performance on these diverse and high-fidelity questions in our dataset to achieve external validity.**\n\n- *The base rate fallacy picture doesn’t seem to be a great example of the fallacy. Would be nice if you could illustrate it there too. Unless I’m missing something?*\n\nThank you for the advice regarding the base rate fallacy figure. We agree that although the current image reflects the Bayesian theorem of the base rate fallacy, it does not demonstrate its counterintuitive aspect. **We will change the figure in the following way:** (a) the blue dots within the large circle, but outside the small circle, will change to red, and (b) the red dots within the large circle, but outside the small circle, will change to blue. Revising in this manner shows how a high proportion of those who are sick have symptoms, but that does not mean that a high proportion of those with symptoms are sick. \n\n- *The caption on the Disjunction Fallacy image is ungrammatical.*\n\n**We will change it to** “A or B. A. People may think if A, then not B. However, if A is a subset of B, this conclusion would be incorrect.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting and well-executed study of LM reasoning. The paper is super well written and clear, focusing clearly on a question and testing it thoroughly. Lots of extra details are given in the appendix. This is probably a good evaluation to add to the long running list of interesting LM evaluations to run on frontier models. Lots of different settings are tried and tested and results are reported super clearly and legibly. I appreciate the text coloring and sizing in the tables. Overall just a nice paper.", "reasons_to_reject": "Definitely not worth rejecting over, but I think what would really improve this paper a lot is spending some time discussing construct validity and external validity. Why are we testing models for these kinds of biases? What do we conclude about them when they get these results?\n\nWith humans, results on these tests may be taken to be indicative of broader reasoning patterns outside of the testing setting. But we already know that can be pretty problematic — psychology tests are run on a select group of people in a very artificial environment, subjects generally try to perform the way they think they are expected to perform, and the results may not be representative of how people act in the real world (See [this paper](https://www2.psych.ubc.ca/~schaller/528Readings/Rozin2001.pdf) for a nice discussion of this in the context of social psych). So there are important caveats to these kinds of results, but we take them seriously because of what we understand about the mental processes behind human decision making.\n\nWe do not have a comparable understanding of LM decision making, into which we do not have introspection, and which is much less consistent and systematic than human decision making. So what does it mean that LMs respond like humans do? Perhaps it just means they were trained on human cognitive bias tests, and any other place where they are presented with the opportunity to do such reasoning, they will behave very differently! Or perhaps these biases will show up in quite a different way when the LM is presented with a more complex or compound problem to solve, and its implicit reasoning won't follow the same patterns. It's really hard to say how these kinds of tests generalize to explain human behavior more broadly, and the case is much, much harder for LMs.\n\nNot to say it's not worth doing and trying, but I think this is an important scientific consideration and a serious limitation of this kind of study, and I think the paper would be much better if it addressed it carefully.", "questions_to_authors": "* More of a comment: the base rate fallacy picture doesn't seem to be a great example of the fallacy. Would be nice if you could illustrate it there too. Unless I'm missing something?\n* The caption on the Disjunction Fallacy image is ungrammatical.", "extra_scores": {}}, {"review_id": "WuXssgAqgS", "reviewer": "Reviewer_f6wA", "summary": "This paper investigates an interesting topic, i.e. whether LLMs show typical representativeness heuristic as in human decision-making process.  The authors construct a test set covering six types of representativeness heuristics within the field of psychology, and evaluate multiple existing LLMs  including GPT-3.5, GPT-4, PaLM 2 and LLaMA 2.  Extensive experiments show that all LLMs demonstrated similar representativeness heuristic bias to humans, and advanced prompt techniques such as COT, self-consistency can only bring marginal improvements.\n\nOverall , I think this is a good paper.  Studying and understanding what kind of shortcoming LLMs have inherited from texts produced by human is important to build intelligent systems that are smarter than human, and I believe this paper takes an important step towards this goal.  The topic also perfectly fits the theme of COLM.\n\nThe paper also shines in its strong experiments, discussing multiple prompting techniques to improve models' performance on the proposed REHEAT dataset.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer f6wA,\n\nThank you for your positive and constructive feedback. Below, we respond to your questions.\n\n- *Do you think more sophistic prompting strategies such as multi-agent debating can bring larger improvements?*\n\n**We believe more sophisticated prompting strategies could bring improvements, similar to how humans, when faced with heuristic questions, think deeply, and invest more mental resources to avoid heuristic biases.** We will explore more complex prompting strategies in our future work.\n\n- *It would be better to provide experiments of testing models performance across different scales.*\n\nThank you for your advice. **We provide additional experiments on LLaMA2-13B to compare with LLaMA2-70B.**\nThe performance of LLaMA2-13B is slightly lower than that of LLaMA2-70B. We can infer that model size impacts performance on representativeness heuristic questions, with larger models generally enhancing reasoning capabilities to a certain extent.\n\n| Model | Standard | Zero-shot CoT | Self-Consistency | One-Shot ICL |\n| --- | --- | --- | --- | --- |\n| LLaMA2-13B | 24.3 | 28.2 (+3.9) | 23.8 (-0.5) | 41.1 (+16.8) |\n| LLaMA2-70B | 27.2 | 29.2 (+2.0)  | 29.2 (+2.0) | 41.7 (+14.5) |\n- *Is there any results on average human accuracy on the proposed testset?*\n\nAfter receiving IRB approval from our institution, we have completed human evaluations on our proposed ReHeAT dataset. Through the Prolific platform, we recruited 153 participants to answer questions. Participants are asked to answer 3-4 types of representative heuristic questions. Human results are close to those of LLaMA2-70B, whereas the GPT series performs better than the human results. **We will incorporate these new results into our final version and provide a more comprehensive analysis.**\n\n| Type | Base Rate Fallacy | Conjunction Fallacy | Disjunction Fallacy | Insensitivity to Sample Size | Misconceptions of Chance | Regression Fallacy | Average |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Human | 27.7 | 24.4 | 12.5 | 33.3 | 50.0 | 85.7 | 27.2 |\n| GPT-3.5 | 31.9 | 31.1 | 22.9 | 45.1 | 50.0 | 71.4 | 34.7 |\n| GPT-4 | 34.0 | 20.0 | 6.3 | 54.9 | 50.0 | 71.4 | 31.2 |\n| PaLM 2 | 19.1 | 2.2 | 2.3 | 22.0 | 50.0 | 71.4 | 14.8 |\n| LLaMA2-70B | 23.4 | 20.0 | 10.4 | 43.1 | 75.0 | 71.4 | 27.2 |\n\n*Note.* Here we compare Human results with the LMs’ Standard results of all other models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- important research topic \n- thorough experimental results\n- a newly constructed datasets", "reasons_to_reject": "- no obvious shortcomings", "questions_to_authors": "- Do you think more sophistic prompting strategies such as multi-agent debating can bring larger improvements?\n- It would be better to provide experiments of testing models performance across different scales.\n- Is there any results on average human accuracy on the proposed testset?", "extra_scores": {}}, {"review_id": "Lv7JX7pWB6", "reviewer": "Reviewer_eSG3", "summary": "This paper evaluates the impact of the representativeness heuristic on LLMs' reasoning. It proposes a dataset that covers various types of representativeness heuristic. Experiments show that LLMs make representativeness heuristic errors similar to humans and they behave differently when faced with different types of questions. In addition, providing hints in the prompts improves models' performance. The paper highlights the importance of considering the representativeness heuristic when using LMs for reasoning and decision-making.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer eSG3,\n\nWe sincerely thank your insightful comments and valuable suggestions. Below, we address the important concerns you raised:\n\n- *It would be helpful if the authors could conduct similar human studies.*\n\nAfter receiving IRB approval from our institution, we have completed human evaluations on our proposed ReHeAT dataset. Through the Prolific platform, we recruited 153 participants to answer questions. Participants are asked to answer 3-4 types of representative heuristic questions. **We will incorporate these new results into our final version and provide more comprehensive outcomes.**\n\n| Type | Base Rate Fallacy | Conjunction Fallacy | Disjunction Fallacy | Insensitivity to Sample Size | Misconceptions of Chance | Regression Fallacy | Average |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Human | 27.7 | 24.4 | 12.5 | 33.3 | 50.0 | 85.7 | 27.2 |\n| GPT-3.5 | 31.9 | 31.1 | 22.9 | 45.1 | 50.0 | 71.4 | 34.7 |\n| GPT-4 | 34.0 | 20.0 | 6.3 | 54.9 | 50.0 | 71.4 | 31.2 |\n| PaLM 2 | 19.1 | 2.2 | 2.3 | 22.0 | 50.0 | 71.4 | 14.8 |\n| LLaMA2-70B | 23.4 | 20.0 | 10.4 | 43.1 | 75.0 | 71.4 | 27.2 |\n- *The curation process for the new data is unclear. How diverse are the new questions?*\n\nThe 153 new questions were organized to retain the essence of the representativeness heuristic being tested, at the same time changing the situations and contexts of the original items. Because each type of representativeness heuristic question shares the same statistical prototype, we aimed to display diversity in the contexts of the questions. There are approximately 15 to 18 different contexts for each type of question.\n\n- *It would be better if the authors could include results using more examples for SPQs and IQs.*\n\nThank you for this advice. Each type of representativeness heuristic has the same statistical prototype. Therefore, only one SPQ is needed for each type, and we collected the majority of answers for each SPQ through self-consistency. Although IQs could indeed benefit from more examples for demonstration, considering that one of the purposes of IQs is to directly compare with SPQs, we adopted the same test format as the SPQs.\n\n- *How are the ICL examples selected in section 5.1?*\n\n**The examples in Section 5.1 for IC are randomly selected from diverse backgrounds.**\n\n- *Missing reference that compares human and LLMs reasoning, e.g. [1]*\n\nThank you for recommending the missing references. **We will add this reference in our final version.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Representativeness heuristic on LLMs is less explored in prior work, but understanding its impact on LLMs’ reasoning is important. \n- This paper is well-written and easy to follow. The question types and experimental setup are clearly described. Examples of each type of questions are detailed in the appendix.\n- The experiments are well-designed, especially the controlled ones on statistical SPQs and IQs. The qualitative analysis is comprehensive and interesting.", "reasons_to_reject": "- This paper aims to investigate whether LLMs make representativeness heuristic mistakes similar to humans, but understanding the results is challenging without human performance for comparison. It would be helpful if the authors could conduct similar human studies or, at the very least, provide references from previous studies, given that some questions are directly drawn from prior research.  \n- The paper mentions “49 questions drawn directly from previous research and 153 new questions”, but the curation process for the new data is unclear. Also, how diverse are the new questions?", "questions_to_authors": "- The experiments on IQs and SPQs appear to only use one example per question type, but it remains unclear whether this performance would remain consistent with more examples. It would be better if the authors could include results using more examples.\n- How are the ICL examples selected in section 5.1?\n- Missing reference that compares human and LLMs reasoning, e.g. [1]\n\n[1] Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Hannah R. Sheahan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, Felix Hill. Language models show human-like content effects on reasoning tasks.\n\n\n## Comments after Author Discussion\nRating: 7 → 8", "extra_scores": {}}, {"review_id": "091HohWXfi", "reviewer": "Reviewer_4fxR", "summary": "The authors create a new dataset, ReHeAT, that assesses susceptibility to the representativeness heuristic (RH), where humans judge scenarios similar to an exemplar to be more likely. The authors show that many popular LLMs exhibit RH biases, and that this bias can be improved by nudging the models towards using its in-weights knowledge. The paper is clear and polished, and the experiments are thoughtful and extensive.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 4fxrR,\n\nThank you for your constructive comments and for recognizing the overall value of our work.\n\n- *Change Subtitle 5.3: “Improving Performance by Hinting (that LLMs to ?) use Their Knowledge”*\n\n**We have changed Subtitle 5.3 accordingly to “Improving Performance by Hinting that LLMs Use Their Knowledge.”**\n\n- *Discussing your results in light of human base rates would be interesting.*\n\nAfter receiving IRB approval from our institution, we have completed human evaluations on our proposed ReHeAT dataset. Through the Prolific platform, we recruited 153 participants to answer questions. Participants are asked to answer 3-4 types of representative heuristic questions. Human results are close to those of LLaMA2-70B, whereas the GPT series performs better than the human results. **We will incorporate these new results into our final version and provide more comprehensive outcomes.**\n\n| Type | Base Rate Fallacy | Conjunction Fallacy | Disjunction Fallacy | Insensitivity to Sample Size | Misconceptions of Chance | Regression Fallacy | Average |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Human | 27.7 | 24.4 | 12.5 | 33.3 | 50.0 | 85.7 | 27.2 |\n| GPT-3.5 | 31.9 | 31.1 | 22.9 | 45.1 | 50.0 | 71.4 | 34.7 |\n| GPT-4 | 34.0 | 20.0 | 6.3 | 54.9 | 50.0 | 71.4 | 31.2 |\n| PaLM 2 | 19.1 | 2.2 | 2.3 | 22.0 | 50.0 | 71.4 | 14.8 |\n| LLaMA2-70B | 23.4 | 20.0 | 10.4 | 43.1 | 75.0 | 71.4 | 27.2 |\n- *I see the note on page 8 that human base rates are around 10 to 50%. In a way, should we expect LLMs to exhibit this bias, in that they are trained to replicate the judgments of a population of humans?*\n\nThank you for asking this great question. **We do observe similar performance between humans and LMs. However, regarding whether we should expect LLMs to exhibit the representativeness heuristic, we think it depends on the LMs’ potential use.** When LLMs are used to aid in decision-making, we certainly do not want them to display such biases, as this cognitive trap can lead to incorrect judgments. From another viewpoint, this heuristic is a simplification strategy that helps people make quick decisions, which can be beneficial. If our goal is for LLMs to mimic human behavior closely, we should expect them to exhibit the representativeness heuristic. In reality, however, our aim extends beyond merely replicating human-like behavior; we also hope that LLMs will surpass human performance in areas prone to bias and other forms of error.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- New benchmark for assessing language models for a known bias in humans.\n- Concrete takeaways for the reader to fix this bias in their models.\n- Thoughtful core premise supported by logical and interesting experiments.", "reasons_to_reject": "I recommend acceptance.", "questions_to_authors": "Comments:\n- Change Subtitle 5.3: \"Improving Performance by Hinting (that LLMs to ?) use Their Knowledge\"\n- Discussing your results in light of human base rates would be interesting. I see the note on page 8 that human base rates are around 10 to 50%. In a way, should we expect LLMs to exhibit this bias, in that they are trained to replicate the judgments of a population of humans?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is a solid paper that brings a set of well-documented cognitive/social psychology phenomena related to a theoretically important idea in those fields, the representativeness heuristic, into the picture of studying and evaluating LLMs. There's great concurrence among reviewer evaluations and I agree that the paper is well executed and a valuable contribution. I recommend acceptance.

[At least one review was discounted during the decision process due to quality]
Decision: Accept