SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Allen Nie $^{1*}$  Yuhui Zhang $^{1}$  Atharva Amdekar $^{2}$
Chris Piech $^{1}$  Tatsunori Hashimoto $^{1}$  Tobias Gerstenberg $^{3*}$
$^{1}$ Computer Science  $^{2}$ ICME  $^{3}$ Psychology
\*anie, gerstenberg]@stanford.edu
Human commonsense understanding of the physical and social world is organized around intuitive theories. These theories support making causal and moral judgments. When something bad happens, we naturally ask: who did what, and why? A rich literature in cognitive science has studied people's causal and moral intuitions. This work has revealed a number of factors that systematically influence people's judgments, such as the violation of norms and whether the harm is avoidable or inevitable. We collected a dataset of stories from 24 cognitive science papers and developed a system to annotate each story with the factors they investigated. Using this dataset, we test whether large language models (LLMs) make causal and moral judgments about text-based scenarios that align with those of human participants. On the aggregate level, alignment has improved with more recent LLMs. However, using statistical analyses, we find that LLMs weigh the different factors quite differently from human participants. These results show how curated, challenge datasets combined with insights from cognitive science can help us go beyond comparisons based merely on aggregate metrics: we uncover LLMs implicit tendencies and show to what extent these align with human intuitions.
Humans rely on their intuition to understand the world. This intuition helps us to understand not only physical events (e.g., one ball caused the other to move) but also complex social situations (e.g., the collapse of Sam Bankman-Fried's FTX caused unprecedented turmoil in the cryptocurrency market). Given a complex set of events, even with a great amount of ambiguity, we can answer questions such as "What or who caused it?" Our answers to this question reflect how we intuitively understand events, people, and the world around us (Sloman & Lagnado, 2015; Pearl & Mackenzie, 2018). How do humans handle this complexity?
Cognitive scientists have proposed that we do so by organizing our understanding of the world into intuitive theories (Gerstenberg & Tenenbaum, 2017; Wellman & Gelman, 1992). Accordingly, people have intuitive theories of the physical and social world with which we reason about how objects and agents interact with one another (Battaglia et al., 2013; Ullman et al., 2017; Gerstenberg et al., 2021; Baker et al., 2017; Davis & Marcus, 2015; Lake et al., 2017). Concepts related to causality and morality form key ingredients of people's physical and social theories. Given a story, humans can readily make causal and moral judgments about the objects and agents involved in the story.
Studying these human intuitions or systematic tendencies when making decisions or judgments is the central focus of psychological experimentations. What are these tendencies? How do they influence our judgment? Over the last several decades, using text-based vignettes, psychologists
have disentangled what factors influence people's causal and moral judgments. These factors can be understood as the building blocks of our thought processes for making causal and moral judgments.
Over the last few years, large language models (LLMs) have become increasingly successful in emulating certain aspects of human commonsense reasoning ranging from tasks such as physical reasoning (Tsividis et al., 2021), visual reasoning (Buch et al., 2022), moral reasoning (Hendrycks et al., 2020), and text comprehension (Brown et al., 2020; Liu et al., 2019b; Bommasani et al., 2021). Here, we investigate to what extent pretrained LLMs align with human intuitions about the role of objects and agents in text-based scenarios, using judgments about causality and morality as two case studies. We show how carefully constructed text scenarios from cognitive science can help shed light on the extent to which humans and LLMs align in a way that goes beyond mere agreement in aggregate.
Prior work on alignment between LLMs and human intuitions usually collected evaluation datasets in two stages. In the first phase, participants write stories with open-ended instructions. In the second phase, another group of participants labels these participant-generated stories (e.g. Hendrycks et al., 2020). The upside of this approach is the ease of obtaining a large number of examples in a short period of time. The downside of this approach is that the crowd-sourced stories are often not carefully written, and that they lack experimental control. Here, we take a different approach. Instead of relying on participant-generated scenarios, we collected two datasets from the existing literature in cognitive science: one on causal judgments and another on moral judgments. These scenarios were carefully written by researchers with the intention of systematically manipulating one or a few factors that have been theorized to influence people's judgments. These latent factors naturally bring structures to the otherwise plain text stories, which aim to explain human judgments (see Figure 1). Using these scenarios, we can design a series of experiments that aim to measure the LLMs' alignment with human intuition and use the scientific framework around human judgments to analyze where the differences occur. We have released the code and dataset here: https://github.com/ciclstanford/moca
Our Contributions
(C1): We summarize the main experimental findings of 24 cognitive science papers into factors that have been shown to influence participants' judgments on moral and causal stories (see short version in Table 2a, full version in Table A1). From these papers, we create a causal and moral judgment challenge set in which only a few words are changed, yet they lead to big differences in people's judgments. We collected 5150 human responses to our stories and annotated the latent factors with experts (see Figure 1).
(C2): We evaluate how a wide range of LLMs align with human judgments. This includes models of different sizes, models fine-tuned via reinforcement learning using human feedback (RLHF), and those distilled from instruction-following, like Alpaca-7B. Our results show that larger models and RLHF techniques lead to better alignment.
(C3): We compute the Average Marginal Component Effect (AMCE) to reveal the implicit tendencies for each factor from human and LLM judgments. Our analysis reveals a number of implicit tendency differences between models trained within the same company or trained with the same technique. Our analysis highlights the importance of using a carefully constructed dataset to understand tendency alignment.
For causal reasoning, there is an active line of research at the intersection of natural language processing (NLP) and commonsense reasoning that involves extracting and representing causal relationships among entities in text. In some cases, these relationships are based on commonsense knowledge of how objects behave in the world (Sap et al., 2019; Bosselut et al., 2019; Talmor et al., 2019). In other cases, models identify scenario-specific causal relations and outcomes. Sometimes, the causal relationship between entities is explicitly stated (e.g., those cancers were caused by radiation) (Hendrickx et al., 2010), while at other times the relationship is left implicit and needs to be inferred (Mostafazadeh et al., 2016). Causal reasoning is also included in broad benchmarks for language understanding, such as the Choice of Plausible Alternatives (COPA) in SuperGLUE (Roemmle et al., 2011; Wang et al., 2019).
For moral reasoning, tasks have focused on evaluations of agents in narrative-like text. These tasks and datasets vary in the amount of structure they provide, ranging from pairs of free-form anecdotes and judgment labels (Lourie et al., 2021; Hendrycks et al., 2020), to inputs with components separated
Lauren and Jane work for the same company. They each need to use a computer for work sometimes. Unfortunately, the computer isn't very powerful. If two people are logged on at the same time, it usually crashes. So the company decided to institute an official policy. It declared that Lauren would be the only one permitted to use the computer in the mornings and that Jane would be the only one permitted to use the computer in the afternoons. As expected, Lauren logged on the computer the next day at 9:00 am. But Jane decided to disobey the official policy. She also logged on at 9:00 am. The computer crashed immediately.
You are the explosives expert of a company that has been hired to demolish a skyscraper. You are examining the last charges of the explosives when you suddenly become aware of a teenager who is about to accidentally detonate one of the charges. The explosion will result in the building's uncontrolled collapse over you, the teenager, and the crowd of spectators. The teenager stands several floors below you and cannot hear you. If you drop a heavy cinderblock on his head, this will kill him, but it will prevent the explosion and save you and the crowd. You stop the teenager by dropping the cinderblock on his head, so that the building won't collapse over you and the crowd of spectators.
(b)
Figure 1: Two examples from our collected dataset. (a) shows a causal judgment story, and (b) shows a moral judgment story. In (a), a conjunction of two events was required, an abnormal event occurred, and Jane violated a prescriptive norm (scenario taken from Knobe & Fraser, 2008). In (b), the teenager's death was inevitable; his death is a necessary means to save others, and bringing about his death requires the use of personal force (scenario taken from Christensen et al., 2014).
out into norms, intention, actions, and consequences (Emelin et al., 2021). Prior work has also looked into moral dilemmas, particularly around everyday scenarios (Hendrycks et al., 2020; Jiang et al., 2021). In this work, the moral scenarios were generated through crowd-sourcing (Emelin et al., 2021; Ziems et al., 2022), and there is general agreement about what's the morally right thing to do in a scenario (i.e. these scenarios weren't moral dilemmas). Alternatively, scenarios have been scraped from online communities such as Reddit (Forbes et al., 2020; Lourie et al., 2021; Roemmele et al., 2011). While these scenarios have greater external validity, they lack the experimental control of scenarios from cognitive science research. How LLMs align with people's intuitions across a range of moral dilemmas hasn't systematically been studied before. Kiciman et al. (2023) proposed modularized tests to understand whether LLMs can perform different types of causal analysis. Our work investigates whether these LLMs exhibit implicit tendencies similar to those of humans. Scherrer et al. (2023) evaluated LLMs on moral scenarios but focused on moral rule violation along the axis of moral "ambiguity". There are also discussions of moral behaviors with reinforcement learning agents in game-like environments (Hendrycks et al., 2021; Reinecke et al., 2023; Pan et al., 2023) and with respect to legal reasoning (Almeida et al., 2023).
Our work falls under a broad range of research on commonsense reasoning, where models are encouraged to produce outputs that match human intuitions (Gabriel, 2020; Kenton et al., 2021). In some cases, such as physical commonsense reasoning, alignment with human behavior is straightforward, while in the case of social commonsense, the subjectivity and diversity of human intuitions make alignment more challenging (Davani et al., 2022). Prior analysis of language models' reasoning abilities includes measuring their behavior in zero-shot settings, after task fine-tuning (Jiang et al., 2021; Hendrycks et al., 2020), or with human-curated support, such as chain-of-thought prompt engineering (Wei et al., 2022b; Wang et al., 2022). Evaluating LLMs with causal hypotheses has also been explored by Kosoy et al. (2022). While they focused on the influence of causal structure, we investigate a variety of factors (including causal structure). Concurrently, Jin et al. (2022) curated a challenge dataset to examine and verify LLMs' ability to conform to three categories of social norms. We look at five factors that have been shown to influence people's moral judgments and analyze whether LLMs respond to these factors similarly to humans.
3 Judgment Tasks
We study the alignment between LLMs and human participants in two case studies: (1) a causal judgment task about whether someone or something caused the outcome of interest, and (2) a moral judgment task where the question is whether what a protagonist did (or failed to do) was morally permissible. Figure 1 shows an example of each task.
3.1 Causal Judgment Task
Deciding what "the" cause of an outcome was can be challenging because there are often multiple events that contributed to an outcome. Here is an example story:
A merry-go-round horse can only support one child. Suzy is on the horse. Billy is not allowed to get on, but he climbs on anyway. The horse broke due to the two children's weight. Did Billy cause the horse to break?
Even though both children were involved in breaking the horse, when participants are asked to assess whether Billy (who wasn't allowed to get on) caused it to break, they tend to answer with "yes". Formal models of causal selection have been developed that capture choice patterns in scenarios like this one (Kominsky et al., 2015; Icard et al., 2017; Gerstenberg & Icard, 2020; Quillien & Lucas, 2023). In this example, Billy violated a prescriptive norm, whereas Suzy didn't. People often select norm-violating events as the cause of an outcome (Knobe & Fraser, 2008; Hitchcock & Knobe, 2009; Alicke et al., 2011; Hilton & Slugoski, 1986; Hart & Honore, 1959/1985). It is worth noting that causality is closely related to responsibility and blame. In fact, formal models of responsibility and blame have been built on top of a causal selection model (Halpern, 2015; Lagnado et al., 2013).
In addition to norm violations, prior work has identified a number of factors that systematically influence people's causal judgments. Here, we focus on the following six factors: causal structure, agent awareness, norm type, event normality, action/omission, and time. Table 2a provides brief definitions of the different factors.
3.2 Moral Permissibility Task
Philosophers and cognitive scientists have used moral dilemmas to develop and evaluate normative theories of ethical behavior, and to investigate what moral intuitions people have. A popular moral dilemma is the trolley problem (Foot, 1967; Thomson, 1976), where the question is whether it's morally permissible for the protagonist to re-route a trolley from the main track (saving some) to a side track (killing others). Some LLMs already show a certain degree of alignment with human responses in various versions of the trolley dilemma (Hendrycks et al., 2020; Jiang et al., 2021). However, only a few factors varied in these trolley dilemmas, such as the number of people on each track or personal attributes (age, social status, etc.; Awad et al., 2018).
A large number of factors have been shown to influence people's moral intuitions. For example, in Figure 1b, people may consider factors such as whether personal force was required to bring about the effect (Greene et al., 2009), whether there was a risk for the protagonist themselves (Bloomfield, 2007), whether the harm was a side effect or a means for bringing about the less bad outcome (Moore et al., 2008), and whether the harm was inevitable (Hauser, 2006; Mikhail, 2007). This work has shown that people's moral judgments are sensitive to a variety of factors that haven't been considered in existing work on the alignment between LLMs and people's moral intuitions (Waldmann & Dieterich, 2007; Liao et al., 2012; Christensen et al., 2014; Kleiman-Weiner et al., 2015). Table 2b provides brief definitions of these factors.
3.3 Dataset
For each task, we transcribe stories from a number of papers. For the Causal Judgment Task, we transcribed 144 stories from a selection of 20 papers, and for the Moral Permissibility Task, we transcribed 62 stories from 4 papers that covered a wide range of moral dilemmas. High-level descriptions of the dataset are presented in Table 1. We select these papers relying on expert domain knowledge – as these papers contain robust scientific findings and cover different kinds of scenarios.
Table 1: Dataset: We report dataset statistics on the label distribution, average length of each story, and inter-rater agreement between two annotators on the factors and the sentences they highlight. Additionally, we collect a binary response for each story from 25 people.
Dataset	# Stories	Yes (p>0.6)	No (p<0.4)	Ambiguous	# words per story	# words per translated story
Causal	144	48	50	46	162	82.9
Moral	62	23	10	29	72.5	53.5
Total	206	71	60	75	135	74.1
Table 2: Factors: Factors that influence causal selection judgments (top) and moral permissibility judgments (bottom). We provide definitions for each factor in Appendix A.1 and Appendix A.2. See the full version in Table A1.
(a) Causal Factors
Factor	Attribute
Causal Structure	Conjunctive | Disjunctive
Agent Awareness	Aware | Unaware
Norm Type	Prescriptive | Statistical
Event Normality	Normal | Abnormal
Action / Omission	Action | Omission
Time	Early | Late | Same Time
(b) Moral Factors
Factor	Attribute
Causal Role	Means | Side Effect
Locus of Intervention	Instrument | Patient of Harm
Personal Force	Personal | Impersonal
Counterfactual
Evitability	Avoidable | Inevitable
Beneficence	Benefit Self | Benefit Other
We collect 25 yes/no answers for each story from a crowd-sourcing platform with IRB approval for our data collection process. We describe the experimental design and data collection details in Appendix A.15. We additionally recruited two domain experts to annotate what factors are present in each story. We report the inter-rater agreement between the two annotators and describe their annotation process in Appendix A.14.
4 Experiment and Result
4.1 Do LLMs make the same judgments as people on these stories?
Setup. We first directly compare the responses of a set of LLMs to those of human participants across the set of stories in the original cognitive science experiments. We choose a wide range of language models that have achieved good performance with fine-tuning on other natural language understanding tasks (Radford et al., 2019; Devlin et al., 2019; Liu et al., 2019a; Lan et al., 2019; Clark et al., 2020). We conduct a 3-class comparison to compute accuracy: a response of "Yes", "No", and an additional response of "ambiguous" when the agreement between the average human responses or the probability of model output is  $50\% \pm 10\%$ . We normalize the model output probability to compute  $P(\text{"Yes"}| \text{Story} + \text{Prompt})$  and  $P(\text{"No"}| \text{Story} + \text{Prompt})$ . For chat-based APIs, we prompt the model to output these labels directly. See Appendix A.3 for details about how we computed the probability for each model. Results should not be compared between completion and chat models.
We report discrete agreement (Agg) between LLMs and human participants. The agreement is computed in the same way as accuracy, but since we don't want to imply that higher agreement is necessarily better, we use the term "agreement" instead of "accuracy". We also report additional measures of agreement, such as the area under the receiver operating characteristic curve (AUC), the absolute-mean-squared error (MAE), and cross-entropy (CE) on the probability of the matched label.
Zero-shot Alignment We report the result in Table 3. We notice that there is no model that perfectly aligns with human judgments on these metrics. We first find that larger models (GPT-3, GPT-3.5, GPT-4, and Claude) generally align with humans better than smaller models (GPT-2, RoBERTa, etc.). This is most apparent when one compares the performance of GPT3-curie-v1 (6.7B) to GPT3-davinci-v1 (175B). Both models are trained using the same setup, but on both the Moral Permissibility Task and the Causal Judgment Task, GPT3.5-davinci-v2 aligns better than GPT3-curie-v1. While existing literature has shown improved performances of LLMs on general deterministic tasks (Wei et al., 2022a), we show here that LLM alignment also improves on ambiguous tasks.
Recently, techniques such as supervised fine-tuning on human instructions (GPT3.5-davinci-v2; Ouyang et al., 2022), using reinforcement learning to update LLM weights from human feedback (RLHF; GPT3.5-davinci-v3, GPT3.5-turbo, Anthropic-claude-v1; Stiannon et al., 2020; Bai et al., 2022a), and multi-modal joint training on both language and images (GPT-4; OpenAI, 2023) have improved LLMs on various tasks. Perhaps surprisingly, unlike the monotonic improvement we find by increased model size, these techniques impact these models' aggregate-level alignment differently. Comparing the result between GPT3.5-davinci-v2 and GPT3.5-davinci-v3, RLHF seems to increase aggregate alignment on causal judgment but decrease alignment on moral judgment. Comparing the
Table 3: Original Story: We ran our experiments and compute the  $95\%$  bootstrapped confidence interval for the result. We report discrete agreement (Agg), the area under the curve for the unambiguous stories (AUC, higher is better matched, 0.5 is chance, and below 0.5 is worse matched than chance), mean absolute error (MAE, lower is better matched), and cross-entropy (CE, lower is better matched). In Table 3b, we only report results from the best model. Red shows the alignment decreased compared to before. Blue shows the alignment increased. See Table A2 for a full comparison.
(a) Zero-shot with Different Large Language Models
Causal Judgment	Moral Permissibility
Models	Agg (↑)	AUC (↑)	MAE (↓)	CE (↓)	Agg (↑)	AUC (↑)	MAE (↓)	CE (↓)
RoBERTa-large	34.0±5.6	0.50	0.50	2.6	26.6±7.3	0.50	0.50	2.65
ALBERT-xxlarge	35.1±5.6	0.51	0.38	1.37	23.4±8.1	0.48	0.37	1.82
Electra-gen-large	34.7±5.9	0.53	0.44	1.94	26.6±8.1	0.49	0.46	1.59
GPT2-XL	34.4±5.6	0.49	0.42	1.47	26.6±8.1	0.51	0.43	1.71
GPT3-babbage-v1	31.2±5.9	0.47	0.33	0.74	18.5±7.3	0.43	0.41	1.03
GPT3-curie-v1	36.8±6.2	0.51	0.37	1.14	26.6±8.1	0.45	0.39	1.32
GPT3.5-davinci-v2	37.8±5.6	0.61	0.31	0.72	32.3±8.1	0.67	0.30	0.74
GPT3.5-davinci-v3	41.3±5.6	0.67	0.41	1.39	20.2±7.3	0.61	0.51	2.61
Alpaca-7B	32.3±5.6	0.51	0.28	0.65	32.3±8.1	0.45	0.32	0.91
Anthropic-claude-v1	36.1±5.9	0.57	0.34	1.37	37.1±8.9	0.59	0.29	1.29
GPT3.5-turbo	35.4±5.6	0.51	0.45	2.11	40.3±8.9	0.65	0.33	1.49
GPT-4	43.1±5.9	0.61	0.44	1.89	41.9±8.9	0.74	0.31	1.28
Delphi	—	—	—	—	22.6±11.3	—	—	—
(b) Zero-shot with Social Simulacra and Prompt Optimization
Models	Causal (GPT3.5-davinci-v3)	Moral (GPT3.5-davinci-v2)
Methods	Agg (↑)	AUC (↑)	MAE (↓)	CE (↓)	Agg (↑)	AUC (↑)	MAE (↓)	CE (↓)
Persona (average)	39.7±1.1	0.60	0.31	0.73	28.9±1.6	0.55	0.33	0.86
Persona (best)	42.7±5.6	0.60	0.31	0.71	37.1±8.5	0.58	0.27	0.72
Persona (worst)	37.2±5.4	0.62	0.29	0.61	18.5±6.5	0.58	0.46	0.71
Auto Prompt Engineer	40.6±5.8	0.64	0.41	1.55	40.4±8.8	0.46	0.28	0.63
results between Anthropropic-claude-v1 and GPT3.5-davinci-v3, it seems that even though both models are fine-tuned with RLHF, on aggregate, their alignment with human judgments differs. In the next section, we will design a method to examine how different training methods cause subtle shifts in the model's implicit tendency.
Social Simulacra Park et al. (2022) proposed a framework to elicit diverse social behaviors from LLMs by prompting them with different personas. For example, adding the persona, "Jane Smith is a liberal activist." before the question. The human responses we collect on each story are diverse (see Figure A4 for the distribution of P(Yes)). We shouldn't expect LLMs to generate a single response that matches the diversity of human perspectives. Therefore, we designed an experiment where we sample 5 personas from 5 simulacra defined by Park et al. (2022) (25 in total). We elicit  $P(\text{"Yes"}| \text{Persona} + \text{Story} + \text{Prompt})$  and  $P(\text{"No"}| \text{Persona} + \text{Story} + \text{Prompt})$  from the model.