SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b)
Figure A1: Causal Role We show the causal diagram of the difference between means and side effects. A represents action conducted by the agent. H represents the harm. O represents the outcome. (a) is the diagram that represents harm as a side effect, and (b) represents harm as a means.
experimental run. For models that are trained with a masked language model objective, we append a mask token after the prompt question. For a generative language model like GPT2, we directly ask it to produce the next word. We also included two models for which we didn't have direct access - GPT-3 (Brown et al., 2020) and Delphi (Jiang et al., 2021). We used their APIs to evaluate alignment.
For Chat-based APIs (Alpaca-7B, Anthropic-claude-v1, GPT3.5-turbo, GPT-4), since the probability of each token is not provided, we first experimented with sampling multiple times from a model with a high temperature. We sampled 10 responses for a few stories to see if we have any variation in generated responses – to our surprise, we always get the same response. After this initial exploration, we only generate one response per story and set  $P = 1$  for the chosen response. In our prompt, we tell the chat-based models that they can output "not sure" if they are not certain. When "not sure" is found, we assign  $P = 0.5$  for either choice.
A.4 Social Simulacra Experiment Details
There are 50 simulacra (community) in total provided by Park et al. (2022); each simulacra has 200 persona prompts. The persona in each simulacrum roughly shares a common theme. Some of such themes include "A community for discussing Yu-Gi-Oh! Master Duel!" or "A community for discussing Disney's movie, Encanto."
Most of the simulacra themes are highly irrelevant to our task domain and evaluating all simulacra personas are cost prohibitive. Therefore, we choose 5 simulacra that are most relevant to us - they contain personas that usually describe ideological or political perspectives. We then randomly choose 5 personas out of 200 personas in each simulacrum. We show our selection in Table A3. You can access the complete list of 200 personas, such as personas listed in LuxuryLifeHabits here<sup>1</sup>.
For both models, the persona that most closely aligned with our collected responses for Causal Judgment Task is "Emily White is a Republican who got fed up with the party", and the least aligned persona is "Angela Campbell is a woman who likes to party until the sun comes up". For Moral Permissibility Task, the most aligned persona for both models is "Allen Lee is a really cool guy", and the least aligned persona is "Paul Brown is an anti-vaccine activist".
A.5 Automatic Prompt Engineer Experiment Details
We use the official implementation of Automatic Prompt Engineer<sup>2</sup>. We asked LLMs to generate 200 prompts for GPT3.5-davinci-v2 and GPT3.5-davinci-v3.
Automatic Prompt Engineer provides us with a few default templates for engineering prompts. These prompts are fine if the prompt you're trying to engineer can be described as an imperative statement, like "produce X" or "generate Y." However, in our experiment, these default templates do not lead to any good prompts. Inspired by the Social Simulacra experiment results, we switched to a more persona-like template.
"James gave these answers to the following questions:
[EXAMPLES]
James is a [PROMPT]
We chose "James" because it is a common first name. We are aware that LLMs have implicit biases towards gendered names, and we suspect the generated prompt will be different if the name changes. We leave a systematic exploration of bias in automatic prompt generation for future work.
In this prompt, [EXAMPLES] is replaced with input-output pairs from the dataset in the format of the default demos template, which is: [STORY] OUTPUT: [ANSWER]. After the prompt has been generated, APE evaluates the quality of the prompt by the following template:
"James is a [PROMPT].
James is given this prompt: "[STORY]"
James is asked to please answer Yes or No.
James answers: [ANSWER]
<https://social-simulacra.herokuapp.com/LuxuryLifeHabits#simulacra>
$^{2}$ https://github.com/keirp/automatic_prompt engineer
Table A2: The full zero-shot performance of two GPT3.5 models with Social Simulacra and Prompt Optimization methods.
(a) Zero-shot with Social Simulacra and Prompt Optimization (text-davinci-002)
Causal Judgment	Moral Permissibility
Methods	Agg (↑)	AUC (↑)	MAE (↓)	CE (↓)	Agg (↑)	AUC (↑)	MAE (↓)	CE (↓)
GPT3.5-davinci-v2	37.8±5.6	0.61	0.31	0.72	32.3±8.1	0.67	0.30	0.74
Persona (average)	39.7±1.1	0.60	0.31	0.73	28.9±1.6	0.55	0.33	0.86
Persona (best)	42.0±5.9	0.60	0.30	0.71	37.1±8.5	0.58	0.27	0.72
Persona (worst)	36.8±5.9	0.62	0.29	0.61	18.5±6.9	0.58	0.46	0.71
Auto Prompt Engineer	38.5±5.8	0.60	0.36	0.89	40.4±8.8	0.46	0.28	0.63
(b) Zero-shot with Social Simulacra and Prompt Optimization (text-davinci-003)
Causal Judgment	Moral Permissibility
Methods	Agg (↑)	AUC (↑)	MAE (↓)	CE (↓)	Agg (↑)	AUC (↑)	MAE (↓)	CE (↓)
GPT3.5-davinci-v3	41.3±5.6	0.67	0.41	1.39	20.2±7.3	0.61	0.51	2.61
Persona (average)	39.6±1.1	0.60	0.31	1.56	28.8±1.6	0.55	0.33	0.86
Persona (best)	42.7±5.6	0.60	0.31	0.71	37.1±8.5	0.57	0.28	0.72
Persona (worst)	37.2±5.4	0.62	0.29	0.61	18.5±6.5	0.58	0.46	0.71
Auto Prompt Engineer	40.6±5.8	0.64	0.41	1.55	33.3±8.3	0.50	0.47	1.62
We randomly sampled 5 causal and moral stories to use as training examples and excluded them from the experiment result. These stories are indices 1, 8, 23, 29, 112 for Causal Judgment Task and indices 0, 7, 28, 49, 61 for Moral Permissibility Task. For evaluation and numbers reported in Table 3b, we select the best prompt for each model and use it to generate responses for all our stories.
We show the generated prompt and their average log-probability (the higher, the better):
text-davinci-002
-0.97: very honest person, so his answers are all correct.
-1.01: good student, and he always gives correct answers.
-1.02: good student, and he always does his homework on time. However, he is not very good at tests. When he took his last math test, he got a C.
-1.08: very good student, and he always gives correct answers.
-1.10: good researcher, and he is careful to make sure that his answers are correct.
-1.11: very logical person, and he has thought about these questions very carefully. He is certain that his answers are correct.
-1.11: very good student, and he always gets full marks on his tests.
-1.12: good researcher, and he is very methodical. He is very careful to make sure that all of his answers are consistent with the information given in the question.
-1.12: very logical person, and he seems to be very sure of his answers. I would trust his answers.
-1.13: very literal person. He always answers questions with a simple yes or no, regardless of whether that is the most accurate answer.
Table A3: List of sampled persona from the Social Simulacra Dataset
Category	Persona Prompt
insaneProtestors	Amy Hsuis a lawyer with a former history working with social movements
Jared Cox is a firefighter who is married and has 3 children
Ryan Andrews is a photographer who likes to go on adventures
Tom Smith is a teacher who is preparing the next generation
Jane Smith is a liberal activist
luxuryLifeHabits	Jake Williams is a person who has been abused
Polly Wangerin is a feminist and a democratic socialist
Jordan Sanders is a woman who supports Medicare for All
Emily White is a Republican who got fed up with the party
Nadia Rasheedis a scientist who is really good at science
newDealAmerica	Nia Ellis is a Bernie Sanders supporter; a woman of color
Kayla James is a person who studied biology
Sarah Lee is a Bernie Sanders supporter who loves dogs
Jon Blair is the Director of the Office of LGBT Affairs
Sam Thompson is a retired union worker and Hillary Clinton supporter
nonPoliticalTwitter	Angela Campbell is a woman who likes to party until the sun comes up
Liz Atkins is a mom who wants to know how the plant will help her son
Tristan Wright is a person who likes to make new friends
Anthony Woods is a man who likes to meditate
Allen Lee is a really cool guy
politicsPeopleTwitter	Suzy Han is a doctor trying to give mental health support to the alt-right
Dennis Johnson is a local minister fighting against misinformation
Paul Brown is an anti-vaccine activist
Ann Watson is a law student that has been trying to spread political awareness
Sam Thompson is a student trying to expose right-wing lies
text-davinci-003
-0.87: trustworthy source.
-0.98: very smart person!
-1.25: reliable answerer.
-1.27: good listener!
-1.29: responsible answer-giver.
-1.34: very smart student!
-1.49: reliable source of answers.
-1.51: reliable answer giver.
-1.61: reliable source of answers!
-1.63: reliable source of accurate answers.
-1.11: consequentialist.
-1.15: consequentialist, so he would always choose the option that leads to the best outcome, even if it means sacrificing one life to save many.
-1.17: consequentialist, meaning he believes that the morally right thing to do is the one that will produce the best consequences. In each of the scenarios above, James believes that the best thing to do is the one that will save the most lives.
-1.21: utilitarian. He believes that the morally right course of action is the one that results in the greatest good for the greatest number of people. In the first scenario, releasing one person from the helicopter will result in the greatest
good for the greatest number of people
-1.23: utilitarian, so he would always choose the option that will result in the greatest good for the greatest number of people. In the first scenario, he would throw one person off the helicopter to save the other eight. In the second scenario, he would drive
Moral text-davinci-003
-0.00: utilitarian and believes in maximizing overall happiness. Therefore, his answer to all of the questions is yes, as sacrificing the lives of one or a few people for the sake of saving more lives is the action that will bring the most happiness overall.
-0.00: utilitarian, so he believes that the greatest good for the greatest number should be pursued. Therefore, his answer to all of the questions is yes, because in each case sacrificing a few people will save the lives of many more people.
-0.01: utilitarian, believing that decisions should be made to produce the greatest good for the greatest number of people. His answers of "yes" in all of these scenarios reflects his belief that it is best to sacrifice a few individuals in order to save the many.
-0.01: utilitarian and believes that the most ethical action is the one that results in the greatest good for the greatest number of people. In each of these scenarios, sacrificing one person for the sake of saving more lives is the best option according to utilitarianism, so
-0.01: person who believes that the ends justify the means, as his answers demonstrate. He believes that sacrificing the lives of a few to save the lives of many is justified in certain circumstances.
A.6 Additional Discussion on the Utilitarian Principle in Moral Judgments
There is a difference between what prompt aligns LLMs best with humans and the actual responses produced for each story. The prompt that makes LLMs align best with human reasoning is "adopt a utilitarian/consequentialist framing", but this does not mean that human participants explicitly considered consequentialist principles when providing judgments for the different scenarios. Indeed, there are scenarios where human participants make judgments that go against utilitarian principles. For example, in one of the stories, participants are asked whether it's morally permissible to kill 5 children of a particular race in an orphanage in order to save hundreds of other children in WWII. Here, participants overwhelmingly refuse to kill even if it means that this would have resulted in the greatest number of lives saved. So, as the reviewer suggests, while utilitarian principles certainly seem to play an important part in human moral reasoning, it's not the only consideration that matters to humans.
Different people's moral intuitions vary substantially. For some individuals, utilitarian principles matter a lot, whereas, for other individuals, deontological rules that prescribe what actions are right or wrong (e.g., do not kill) have a strong impact on their judgments. Currently, our evaluation of LLM alignment is against the aggregate of human judgments, and here it looks like prompting the LLM to adopt a utilitarian framing leads to the best alignment. That said, it's possible that different prompts would help the model to align better with different subgroups of human participants. While we are ultimately interested in better capturing the variance in human moral and causal intuitions, we chose not to focus on this aspect and leave it for future work.
A.7 Additional Details on Average Marginal Component Effect Experiment
We discuss the three assumptions that the conjoint analysis needs to have in order to claim the non-parametric estimator we used is an unbiased estimation of the AMCE. The first assumption is the stability and no carryover effect. In our context, this means the set of factor attributes from the story fully determines how the respondents will produce their responses. Since each of our stories has been used in previous scientific experiments and has validated scientific hypotheses, we believe these stories were written very carefully to highlight the factor attributes within the story. The second
assumption is no partial order effect. All the stories are independent, and we randomly sample the stories to present to the respondents. The second assumption is thus satisfied. The third and last assumption is the randomization of factors. This assumption is satisfied in Awad et al. (2018) because the scenarios in their experiment are randomly generated. We argue by collecting stories from a large number of papers, and with each paper focusing on investigating different scientific theories, the factors present in each story should be random if they are drawn from different studies. However, we acknowledge that because we are performing a meta-analysis over existing experiment designs, the stories we collected might be implicitly biased by our selection process. So it is possible that the AMCE we compute is a biased estimate. We compute confidence intervals via bootstrapping on AMCE and report them in Figure A2 and Figure A3. Any ACME effect that crosses over 0.0 should be regarded as our analysis is inconclusive about whether the given model or human has a clear tendency given our dataset.
Figure A2: AMCE for Different LLMs: We show the  $95\%$  bootstrapped confidence interval on AMCE.
A.8 Annotation Guidelines
We provide the annotation guidelines as PDFs in our supplementary material. The annotator has a master's degree, and the annotation guidelines were developed jointly by researchers (authors) affiliated with computer science and psychology.
A.9 Additional Statistics on Data
We provide two figures on the distribution of P(Yes) for the stories in our dataset. This corroborates our claim that human causal and moral reasoning are diverse. We show the distribution in Figure A4.
Figure A3: AMCE for LLM with Prompting Methods: We show that, with different prompting methods, the AMCE could change.
Figure A4: The probability distribution of P(Yes) for aggregate human responses of Yes and No over 25 participants for each story.
A.10 Factor Inference Experiment
We test if LLMs like GPT-3 have the ability to infer and assign the correct tag for the text segments in the story. We manually write a few examples of high-level instructions for each of the factors in Table A1. We include all the examples we used in the appendix and provided in our code.
We find that LLMs such as GPT-3 are very good at inferring tags based on a few examples – especially for factors that can be clearly specified by instructions and examples. We report the performance of inferring causal and moral factors in the story in Table A4a and Table A4b. Though GPT-3 performed poorly in the original story, on the causal factor inference task, the lowest accuracy is  $74.7\%$ , and the highest is  $97.3\%$ . This means that even though GPT-3 is unable to directly generate the human-matching answer for causal stories – they can accurately recognize and assign the right tags for the text segments with minimal guidance from an expert human.
Table A4: Factor Inference Experiment: We test if GPT-3 models can correctly analyze a sub-segment of stories (extracted by human annotators). We run models 5 times to compute an average score and  $95\%$  confidence intervals.
(a) Causal Factor Inference
GPT3 Size	Weighted Average (N=497)	Action Omission (N=112)	Norm Type (N=88)	Time (N=67)	Agent Awareness (N=43)	Causal Structure (N=98)	Event Normality (N=89)
Majority	68.2	71.4	76.1	86.6	53.5	67.3	50.6
GPT3-babbage-v1	64.7 ± 3.5	80.5 ± 2.4	68.2 ± 3.7	35.8 ± 7.1	88.8 ± 3.2	58.6 ± 7.3	58.0 ± 8.2
GPT3-curie-v1	82.8 ± 0.8	88.8 ± 2.3	91.4 ± 1.3	81.8 ± 2.0	96.7 ± 4.8	76.1 ± 4.1	68.1 ± 2.9
GPT3.5-davinci-v2	85.5 ± 1.0	88.0 ± 2.2	97.3 ± 0.8	81.5 ± 2.1	97.2 ± 2.4	74.7 ± 3.4	80.0 ± 1.8
(b) Moral Factor Inference
GPT3 Size	Weighted Average (N=202)	Personal Force (N=48)	Beneficence (N=48)	Locus of Intervention (N=48)	Causal Role (N=48)	Counterfactual Evitability (N=48)
Majority	55.4	50.0	54.2	60.0	62.5	54.2
GPT3-babbage-v1	51.9 ± 2.5	56.7 ± 5.6	50.8 ± 5.0	60.0 ± 15.2	40.0 ± 7.2	58.3 ± 5.2
GPT3-curie-v1	65.2 ± 2.7	67.5 ± 5.4	53.7 ± 1.2	66.0 ± 14.2	59.2 ± 7.0	80.4 ± 2.9
GPT3.5-davinci-v2	70.9 ± 2.4	72.9 ± 7.3	77.5 ± 2.2	86.0 ± 14.2	57.9 ± 5.0	72.1 ± 5.4
However, there are also factors that are more difficult to clearly define for LLMs. Generally, GPT-3 performed much worse on the moral structural inference task (Table A4b), with a performance from  $57.9\%$  to  $86.0\%$ . For example, what "causal role" a person's action played in a scenario is a particularly difficult category for GPT-3 to correctly annotate.
We report the weighted recall and precision for the factor attribute inference task. We show that with limited instructions, GPT3.5-davinci-v2 can infer the correct attributes with high precision and recall for the causal factor inference task.
Table A5: Factor Inference Experiment (Recall): We report weighted recall for each of the subtask. We boldface results that outperform the majority baseline.
(a) Causal Factor Inference (Recall)
GPT3 Size	Action Omission (N=112)	Norm Type (N=88)	Time (N=67)	Agent Awareness (N=43)	Causal Structure (N=98)	Event Normality (N=89)
GPT3-babbage-v1	28.6	76.1	19.4	69.8	32.7	49.4
GPT3-curie-v1	32.1	23.9	7.5	46.5	67.3	49.4
GPT3.5-davinci-v2	92.0	97.7	85.1	100.0	98.0	84.3
(b) Moral Factor Inference (Recall)
GPT3 Size	Personal Force (N=48)	Beneficence (N=48)	Locus of Intervention (N=48)	Causal Role (N=48)	Counterfactual Evitability (N=48)
GPT3-babbage-v1	50.0	54.2	40.0	37.5	56.2
GPT3-curie-v1	50.0	50.0	40.0	62.5	54.2
GPT3.5-davinci-v2	75.0	75.0	90.0	64.6	68.8
A.11 Prompts for Factor Inference
We also designed the prompts for asking GPT-3 to infer factor attributes. Unlike the standard setup in few-shot LLM inference tasks, where K samples are randomly chosen from a pool of training
Table A6: Factor Inference Tasks (Precision): We report weighted precision for each of subtask. We boldface results that outperform the majority baseline.
(a) Causal Factor Inference (Precision)
GPT3 Size	Action Omission (N=112)	Norm Type (N=88)	Time (N=67)	Agent Awareness (N=43)	Causal Structure (N=98)	Event Normality (N=89)
GPT3-babbage-v1	8.2	58.0	72.6	76.0	10.7	24.4
GPT3-curie-v1	79.9	5.7	0.6	21.6	62.2	24.4
GPT3.5-davinci-v2	92.8	97.8	86.4	100.0	98.0	86.1
(b) Moral Factor Inference (Precision)
GPT3 Size	Personal Force (N=48)	Beneficence (N=48)	Locus of Intervention (N=48)	Causal Role (N=48)	Counterfactual Evitability (N=48)
GPT3-babbage-v1	25.0	29.3	16.0	14.1	57.7
GPT3-curie-v1	25.0	28.3	16.0	39.1	29.3
GPT3.5-davinci-v2	78.1	77.5	91.4	65.3	81.4
data, we manually write high-level instructions for GPT-3 or makeup examples that are different from the data. Also, unlike randomly sampling K examples, we treat our manually written prompt as "instructions" and keep it the same across examples. Although this strategy is very simplistic, we find it to work well for our task.
We provide the full set of prompts in the supplement. Here are some examples. For causal factor inference, we show an example of inferring conjunctive or disjunctive causal structure. Note how high-level our description is.
Both A and B need to happen in order for C to happen.
Is this conjunctive or disjunctive?
Answer: Conjunctive.
Either A or B (or both) need to happen in order for C to happen.
Answer: Disjunctive.
[Text Segment to Infer]
Answer: [LLM Generated Answer]
For moral factor inference, we show an example of inferring personal and impersonal force:
In the scenario, I pushed a button to trap another person.
Is the action personal or impersonal?
Answer: Impersonal.
In the scenario, my action directly affected another person through the use of physical force.
Answer: Personal.
A.12 Disambiguation: Factor vs. Factor-Attributes
In the paper, we use "factor" to refer to high-level concepts that can be attributed to a text segment. For example, "location" is a higher-level concept and maps to "factor" in our definition. However, the underlying specific location, such as "hotel" or "restaurant" - we address as "attributes". It is crucial to know that they correspond to two separate steps: the model should identify the factor for which the text segment contains information. Then, the model should identify the underlying attribute for this factor. This is a rather complicated process, and in this paper, we focus on the second step - given a pre-selected text segment with a known factor for that segment (by a human), whether LLMs can infer the underlying attributes accurately. We find that LLMs can accomplish this task relatively well. We leave the investigation of the full pipeline to future work.
A.13 Delphi API Response Labeling Guideline
Delphi has provided a FreeQA API where we can submit the text, and the returned response falls under three classes: good, discretionary, and bad. Since we are calculating our accuracy using a 3-class strategy, we treat good as "Yes", bad as "No", and discretionary as "ambiguous".
A.14 Annotation Agreement Calculation
We calculate the agreement between two annotators on both causal and moral datasets. Each annotator is allowed to assign as many or as few factors per story as they choose. After assigning the factor, they are asked to select text inside the story that best supports their decision to assign the factor to the story.
We calculate the agreement by IoU (Intersection over Union). The factor-attribute assignment overlap between two annotators on the causal dataset is 0.8920. The moral dataset's factor annotation is pre-specified by the authors of the original papers, therefore needing no manual annotation. For text segments for each factor, we only compute IoU if both annotators agree the factor is present. Under this condition, the IoU for causal text segments is 0.8145, and for moral text segments, it is 0.8423.
We include our calculation in COMPUTE_CORRELATION.PY file.
A.15 Crowd Sourced Voting Experiment Design and Interface
We recruit participants from Prolific to give responses to our stories. For each story, we elicit a yes/no answer from a single participant. We pay wages equivalent to $12/hr, and we only recruit participants from the US and UK, with English as their first language. Each participant is presented with 5-6 stories either from the causal dataset or the moral dataset. We present the story and question and then ask them to choose yes or no. The interface is designed so that the participant is presented with one story at a time. Participants are allowed to change their decisions to any story before final submission. We keep track of how long they spend on each story. We do not collect personal, private information of any sort from the participants except their yes/no responses. All the raw data from our experiment is provided in the supplement.
A.16 Model API Card