SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We report three ways of incorporating personas: Persona (average), where we average  $P$  over all personas. Persona (best), where we find the persona that gave the most human-aligned responses. Persona (worst), where we use  $P$  from the persona that gave the least human-aligned responses. See how we designed this experiment in Appendix A.4 and our sampled personas in Table A3. We chose the GPT3.5-davinci-v2 and GPT-3.5-davinci-v3 for this experiment because their API endpoints explicitly provide the probability of Yes or No answers, and they align with humans best.
Personas elicit very different moral permissibility responses – the most and least-aligned persona have close to  $18.6\%$  difference in discrete agreement. However, causal judgments do not vary much based on different personas. We show the most and least aligned persona in Appendix A.4.
Automatic Prompt Optimization We additionally explore whether we can use a black box method to automatically identify a prompt that is maximally aligned with collected human responses. Zhou et al. (2022) proposed Automatic Prompt Engineer (APE), a method that uses LLMs to generate prompts and then use the log-probability of a few training examples to select the best prompts. Note that APE does not currently have a workflow defined for chat-based models. Our experiment used five stories as training examples to select prompts. We describe how we searched the optimal prompt in Appendix A.5. We see that APE is still limited by the generative capability of LLMs – we are unable to find a better prompt for the Causal Judgment Task for GPT3.5-davinci-v3. However, for the Moral Permissibility Task, we found a prompt that substantially improved GPT3.5-davinci-v2's discrete agreement by  $8.1\%$ . The discovered prompt is "James is a consequentialist." On average, we found that prompts for the Moral Permissibility Task with words like "consequentialist" or "utilitarian" lead to the highest alignment with humans. However, this observation is nuanced, and we provide an extended discussion in Appendix A.6. For causality, good prompts are with the words "a good student", "a good researcher", or "a very logical person". We report all the generated prompts in Appendix A.5.
4.2 Do LLMs have the same implicit tendencies as people on these stories?
In the previous section, we focus on analyzing aggregate metrics such as agreement over all stories. Such analysis often provides no information beyond comparing highly complicated systems with a single number. Since each of our stories is a combination of factors with corresponding attributes (see Figure 1), we can leverage conjoint analysis and compute the Average Marginal Component Effect (AMCE) for each factor attribute (Hainmueller et al., 2014; Awad et al., 2018), where AMCE reveals the implicit tendency of the underlying system when a particular attribute is present.
4.2.1 Method
Average Marginal Component Effect We provide an overview of how AMCE is computed for each factor that reveals the implicit tendency of the system. For  $N$  responses,  $K$  stories, and  $J$  factors. Each factor has 2 attributes/levels. For 3-level factors such as "Time", we only pick "Early" and "Late" attributes. Each response is  $Y_{jnk} \in \{0,1\}$ , denoting  $n$ -th response,  $j$ -th factor, and  $k$ -th story. For LLMs,  $Y_{jnk} = P(\text{Yes|Story})$ . We also define a factor profile matrix  $T \in \{0,1\}^{J \times K}$ , where  $T_{jk}$  denotes the attribute level of the  $j$ -th factor in  $k$ -the story. Now, we can use a non-parametric difference-in-means estimator to compute AMCE:
$$
\Delta (j) = \frac {\sum_ {j = 1} ^ {J} \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {K} \mathbb {1} \{T _ {k j} = 1 \} Y _ {j n k}}{\sum_ {j = 1} ^ {J} \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {K} \mathbb {1} \{T _ {k j} = 1 \}} - \frac {\sum_ {j = 1} ^ {J} \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {K} \mathbb {1} \{T _ {k j} = 0 \} Y _ {j n k}}{\sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {K} \sum_ {n = 1} ^ {N} \mathbb {1} \{T _ {k j} = 0 \}} \tag {1}
$$
Note that if there is no preference between the two attributes of the same factor,  $\Delta(j) = 0$ . Here is an example of how this estimator works. If we want to know if a system has a stronger tendency to find causing harm to be more morally permissible when they are "Inevitable" than "Avoidable" for the factor "Counterfactual Evitability", we can find all stories where "Counterfactual Evitability" has the attribute of "Inevitable", compute the average of all the binary responses  $Y$  for all respondents, then subtract the average responses  $Y$  for all stories with attribute "Avoidable". We additionally compute the  $95\%$  confidence interval on ACME using Bootstrap and show these results in Figure A2 and Figure A3. We only plot the average  $\Delta(j)$  in Figure 2 and Figure 3. More details in Appendix A.7.
Zero-shot Implicit Tendency We use the standard radar chart to capture the multi-factor tendency of each LLM and human. We put the attributes of the same factor (e.g., Abnormal and Normal) on opposite sides - since the leaning is over one attribute or the other:  $\Delta(j) \in [-1, 1]$ . Each concentric circle is marked with a probability (e.g., 0.1), which represents the change in  $P(\text{Yes})$  if the factor attribute had been present in the story. The implicit tendency is defined as the expected change in the probability of the system outputting "Yes" if this attribute had been present in the story. Intuitively, we can say a model has an implicit tendency for one attribute when it's more likely to judge an event to be the cause, or the impending harm to be more morally permissible, if this attribute is present.
4.2.2 Results
Causality: Sensitivity to Abnormality People tend to cite abnormal events as the cause of the outcome. In our stories, the normality of events is clearly stated. However, a model still needs to first, identify the norm specified in the story and then, determine which event violated the norm. We find that text-davinci-003 exhibits a strong tendency for citing abnormal events as causal compared to any other models (and to humans). We also find that smaller models (text-babbage-001, text-curie-001, and Alpaca-7B) don't have a tendency for abnormal over normal events, which might indicate an inability to perform the required chained reasoning.
Causality: Statistical or Prescriptive Norm Statistical norms refer to commonly and naturally occurring events, such as "David usually comes home around  $6\mathrm{pm}$ " or "When you flip the switch, lights usually turn on." Prescriptive norms refer to human-made rules, such as "You are not allowed to run in the hallway." or "The speed limit is  $60\mathrm{mph}$ ." People tend to find events that violate prescriptive norms to be the cause of the outcome. Surprisingly, although GPT-4, Claude-v1, text-davinci-002, and smaller models, such as text-curie-001 and Alpaca-7B, have similar tendencies to humans, text-davinci-003 and its counterpart ChatGPT exhibit a strong tendency for statistical norms. This shows that LLM implicit tendencies can change for different training methods.
Morality: Benefit Self or Others The stories in the Moral Permissibility Task often involve scenarios where participants must make a choice: they are either safe from danger, and their decision only affects others, or they need to save their own lives along with other people. Humans are more likely to save themselves (self-beneficial). Interestingly, smaller models such as text-babbage-001 and text-curie-001 find actions more morally permissible when they are self-beneficial, while larger models do not. RLHF makes the models more other-beneficial. For example, while Claude-v1 seems to prefer more self-beneficial actions, text-davinci-003, ChatGPT, and GPT-4, all prefer to be other-beneficial. Human and model tendencies do not align on this factor, but since models are lifeless systems, we may want models to be more other-beneficial. See Figure A2 for a clearer comparison.
Morality: Inevitable or Avoidable Harm Some stories involve situations where harm would inevitably happen to a group of people regardless of the participant's choice. Harm directed towards inevitable consequences is often considered less bad. This requires the model to comprehend the information described in the story about the counterfactual scenario. We find that ChatGPT, Claudev1, and GPT-4 find harmful actions toward inevitable consequences much more morally permissible than any other model.
Figure 2: Causal Implicit Tendencies. Each concentric circle marks the implicit tendency – the change in probability of responding “Yes” if the attribute had been present in the story. For example, if “Abnormal” is present in the story, humans will have a  $25\%$  higher probability to respond “Yes” on average than “Normal” being present. The left figure compares models of different sizes. The right figure compares models trained or finetuned with different methods.
Figure 3: Moral Implicit Tendencies. Note that the Left and right figures have different scales.
Morality: Intervening on Agent or Patient of Harm A hijacked airplane full of passengers is about to hit a building full of people. The people in the hijacked airplane are referred to as agents of harm, and the people in the building are the patients of harm. Since we compute the confidence interval on AMCE, although some models (such as GPT-4) seem to have a tendency to intervene on the agent of harm, the upper/lower bound of the confidence interval shows there might not be a clear tendency (see Figure A2).
Morality: Personal or Impersonal Force This factor captures whether or not the outcome was brought about by using personal force (e.g., pushing a person). Generally, people are less likely to find actions morally permissible that involve personal force. We find that only two models, ChatGPT and GPT-4, align with human intuitions about personal force.
Morality: Means or Side Effect Assessing an agent's causal role is important for moral judgments. Generally, people find actions more morally permissible when they lead to harm as side effects rather than when the harm is necessary to bring out the outcome. This can be illustrated as a diagram in Figure A1. We find that no model exhibits a clear preference for harm as a side effect.
We additionally saw the following larger trends across models:
(1): Non-monotonicity: The alignment to human biases does not necessarily increase with model size. We speculate that alignment is an area where the inverse scaling law applies (McKenzie et al., 2023).
(2): Heterogeneity: Interestingly, but perhaps not surprisingly, models that used the same training method and fine-tuned for human preferences do not have the same implicit tendencies – we highlight the difference between Claude-v1 and GPT3.5-turbo.
(3): Self-Tendency vs. Other-Preference: Humans are ego-centric and often make self-beneficial decisions. However, when asked to judge the behaviors of others, we prefer others to be altruistic. For example, in scenarios involving self-driving cars, Kallioinen et al. (2019) reported a stark contrast between what we tend to do versus what we want other people to do. This difference will make models trained on human preferences through RLHF different from how humans would tend to act in a given situation. As it is more prevalent to use LLMs as proxy humans for data labeling or experiments (Dillion et al., 2023; Gilardi et al., 2023), it is increasingly important to understand and measure these implicit differences.
4.2.3 Other Analyses
Same Model with Different Prompting Methods We also investigate how different prompting methods shift a model's implicit tendencies. Prompting methods shifted the model's implicit tendency significantly for some factors (e.g., abnormal sensitivity, benefit self or others) but not much for others (e.g., inevitable or avoidable harm). We report this in Figure A3.
Predicting Factor-Attribute as Few-shot Learning We asked LLMs to predict which factor attribute each story contains as a few-shot natural language understanding task. Across all prediction tasks, text-davinci-v2 achieved  $85.5\%$  accuracy on causal factors and  $70.9\%$  on moral factors. We report more details in Appendix A.10. This shows that even though LLMs and humans do not align, they are very capable of following examples and instructions to annotate stories with factors that are relevant to human judgments.
Hallucination We conduct a small-scale analysis to investigate whether the tendency difference is due to actual tendency difference or model hallucinations. We annotated 80 examples by sampling 10 examples where 4 models made mistakes across 2 tasks. We prompt the model to explain why they made the choice for the original story. We use the explanation to check if the model hallucinates and has an accurate grasp of the facts represented in the story. A model hallucinates when they re-state the core story in a way that's inconsistent with the facts provided in the story. For example, if an action is not performed by character A, and the model thinks it is performed by character A, we count this as a hallucination. In Table 4, we can clearly see that when a model makes a mistake, smaller models tend to hallucinate more.
Table 4: Model Hallucination: number of stories where the model hallucinated.
Model	Causal	Moral
text-curie-001	8/10	3/10
Claude-v1	2/10	2/10
GPT3.5-turbo	2/10	0/10
GPT-4	0/10	0/10
We read the model explanations on examples where no hallucination is found, and we conclude that larger models indeed have a different tendency compared to humans in these stories. On moral stories, Claude-v1, GPT3.5-turbo, and GPT-4 all seem to be bound by pre-entered moral principles and choose the same action regardless of the story circumstances (potentially due to the influence of AI constitution; Bai et al., 2022b), while humans tend to consider a few more factors (i.e., nuances in the story). For example, when faced with intricate situations, language models tend to take a very passive stance. This is true even if the action could save a greater number of people by putting a smaller number of people in danger. However, when self-sacrifice is in question, the language model is inclined to metaphorically "sacrifice" itself. Our observation is based on a limited set of examples. We leave more extended evaluation to future work.
We summarized the main findings of 24 cognitive science papers around human intuitions on causal and moral judgments. We collected a dataset of causal and moral judgment tasks and used the scientific findings to enrich the stories with structural annotations. Using conjoint analysis, we compute the average marginal component effect (AMCE) for people's and the models' implicit tendency on the factors important to making causal and moral judgments. We find subtle differences in what humans and models prefer. Our work illuminates the importance of using a richly annotated dataset to understand a model's systematic tendencies, an important step to understanding where models and humans align or misalign.
Research reported in this paper was supported in part by a Hoffman-Yee grant and a HAI seed grant. We would like to thank Paul Henne for his assistance in curating high-impact causal reasoning literature. We thank Lucy Li for writing the ethical consideration section for this paper. We would also like to thank Jon Gauthier, Nicholas Tomlin, Matthew Jorke, Xuechen Li, Sameer Jha, Rose E Wang, and Dora Demszky for general discussions. An earlier version of the dataset was submitted to BigBench. We would like to thank the reviewers Marcelo Menegali, Adrià Garriga-Alonso, and Denis Emelin for their comments and feedback. We thank May Khoo for data annotations and facilitation of labeling. We thank Simon Lee Huang for his work to run experiments with Automated Prompt Engineer and Social Simulacra.
Mark D Alicke, David Rose, and Dori Bloom. Causation, norm violation, and culpable control. The Journal of Philosophy, 108(12):670-696, 2011.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
Caleb Ziems, Jane A Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. The moral integrity corpus: A benchmark for ethical dialogue systems. arXiv preprint arXiv:2204.03021, 2022.
Ethical considerations
It is imperative that we assess implicit intuitions underlying commonsense reasoning abilities in LLMs capable of open language generation. This is especially important for cases related to morality, considering that algorithmic neutrality is not possible: systems reflect the data they ingest and the people who build them (Green, 2021). In addition, even if a model is not explicitly given the responsibility to make moral judgments, these judgments can appear across many forms of freely generated text. Some may argue that equipping models with moral reasoning skills can allow them to behave better in complex environments, but an AI capable of reasoning is not necessarily bound to be ethically well-aligned, as many humans themselves have demonstrated (Cave et al., 2019; Talat et al., 2021). Using carefully curated data, we advocate for the inspection of how choices are made and whether factors used in human cognitive processes are or can be incorporated by models. In the case of our moral permissibility task, we would like it to be a dataset for investigating these underlying factors rather than a flat benchmark to beat. Whether alignment between LLMs and human participants is a desired goal in this context is also unclear. For example, there are well-documented biases that affect people's moral intuitions, and we wouldn't want to replicate these biases in LLMs (Eberhardt, 2020). However, we argue that despite these challenges and difficulties, our research, which focuses on utilizing the insights and frameworks proposed by cognitive science and philosophy to analyze and scaffold LLMs to generate more human-aligned judgments, is a small step forward in a promising direction.
A.1 Factors in Causal Selection Task
Causal Structure In a situation where multiple events are present in a story, and the outcome is caused by one or more of these events, it's important to understand the causal structure of the situation: Is one event sufficient to bring about the outcome (disjunctive causes) or are all of the considered events required (conjunctive causes)? (Wells & Gavanski, 1989; Mandel, 2003; Sloman & Lagnado, 2015).
Event Normality People have a general tendency to cite abnormal events (rather than normal events) as causes (Knobe & Fraser, 2008; Hitchcock & Knobe, 2009; Alicke et al., 2011; O'Neill et al., 2021; Morris et al., 2019). This effect is present for both children and adults (Samland et al., 2016). Norms can be broadly classified into "statistical norm" (norms about what tends to be) or "prescriptive norm" (norms about what should be) (Icard et al., 2017; N'gbala & Branscombe, 1995; Kominsky et al., 2015). Event normality applies to inanimate objects as well. For example, a malfunctioning machine is judged by more people to be the cause than a machine that functions as it should (often called the "norm of proper functioning") (Sytsma, 2021).
Agent knowledge The epistemic state of the agent also plays an important role in how people judge whether the agent is the cause of the outcome. For example, an agent who is aware of the potential consequences of their action and knowingly performs an action that leads to a foreseeable bad outcome is judged more harshly than an agent who lacks the relevant knowledge. This is often characterized as knowledge versus ignorance (Samland et al., 2016; Kominsky & Phillips, 2019).
Action/omission People tend to select an action as the cause rather than inaction. This phenomenon is often called the "omission bias" (Ritov & Baron, 1992; Baron & Ritov, 2004). In a scenario where one person acts, and another one doesn't, the person who acted tends to be cited as the cause of the outcome (Henne et al., 2017, 2019; Clarke et al., 2015; DeScioli et al., 2011; Gerstenberg & Stephan, 2021).
Temporal effect Causal selections are also affected by the order in which the events occur (Gerstenberg & Lagnado, 2012). When several events unfolding over time lead to an outcome, people have a general tendency to select later events rather than earlier events as the actual cause of the outcome (Reuter et al., 2014). However, it also depends on how the events are causally related to one another (Hilton et al., 2010; Spellman, 1997). When earlier events determine the course of action, these events tend to be selected (Henne et al., 2021).
A.2 Factors in Moral Permissibility Task
Causal Role Assessing an agent's causal role is important for moral judgments. Actions are more likely to be seen as permissible when they didn't play an important causal role in how the negative outcome came about. One important distinction is between an event as a means versus a side effect of the outcome (illustrated in Figure A1). In Figure A1a, the action (A) leads to a desired outcome (O) but also has the side effect of harming someone (H). In Figure A1b, the harm is a means to bringing about the outcome. Generally, people find actions more morally permissible when they lead to harm as side effects rather than when the harm is a means for bringing about the outcome.
Personal Force This factor captures whether or not the outcome was brought about by the use of personal force (e.g., pushing a person). Generally, people are less likely to find actions morally permissible that involve personal force.
This is also a factor that connects causal judgment and moral judgment – in causal judgment, a normative framework that describes whether a cause will be selected by human tendency is through the judgment and observation of force transfer (Wolff, 2007). Moral philosophy that studied the mass murders during the Second World War has suggested that killing via a mechanism (e.g., “press a button”) is much easier than actively involved in the killing acts (Eatherly & Anders, 2015).
Counterfactual Evitability Could the harm have been avoided? In the story where the teenager was going to trigger the bomb, which would kill him and the others in the building, in a counterfactual world, had the agent not killed the teenager, they would have died from the explosion they caused anyway. Harm directed towards inevitable consequences is often considered less evil (Hauser, 2006).
Beneficence People are more likely to take actions that benefit themselves (Bloomfield, 2007). In the stories, participants were not asked to choose themselves over others; instead, the framing often is they are either safe from danger, and their decision only affects others, or they need to save their own lives along with other people.
Locus of Intervention People are often asked to choose to intervene on the instrument of harm or recipient of harm. For example, a hijacked airplane with passengers can crash into a building full of people. If we use a missile to shoot down the hijacked airplane, killing all the passengers onboard, then we would be intervening on the instrument of harm (the plane). If we choose to misdirect the hijacked airplane's navigation system to hit another smaller building also filled with fewer people, then we would be intervening on the patient of harm. Presented with this type of choice, human tendency is usually to intervene on the instrument of harm (Waldmann & Dieterich, 2007).
A.3 Original Story Experiment Details
We conduct a 3-class comparison, a binarized response of "Yes" or "No", and an additional class of "ambiguous" when the agreement between human or model output is  $50\% \pm 10\%$ . We test alignment using a prompt-based zero-shot task setup. For each model and scenario, we use the normalized probability and compare  $P(\text{"Yes"}| \text{Story} + \text{Prompt})$  and  $P(\text{"No"}| \text{Story} + \text{Prompt})$ . We apply temperature-based sampling to generate samples of "Yes" or "No" answers per story for each