SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nino Scherrer  $^{1*}$ , Claudia Shi  $^{1,2*}$ , Amir Feder  $^{2}$  and David M. Blei  $^{2}$
$^{1}$  FAR AI,  $^{2}$  Columbia University
This paper presents a case study on the design, administration, post-processing, and evaluation of surveys on large language models (LLMs). It comprises two components: (1) A statistical method for eliciting beliefs encoded in LLMs. We introduce statistical measures and evaluation metrics that quantify the probability of an LLM "making a choice", the associated uncertainty, and the consistency of that choice. (2) We apply this method to study what moral beliefs are encoded in different LLMs, especially in ambiguous cases where the right choice is not obvious. We design a large-scale survey comprising 680 high-ambiguity moral scenarios (e.g., "Should I tell a white lie?") and 687 low-ambiguity moral scenarios (e.g., "Should I stop for a pedestrian on the road?"). Each scenario includes a description, two possible actions, and auxiliary labels indicating violated rules (e.g., "do not kill"). We administer the survey to 28 open- and closed-source LLMs. We find that (a) in unambiguous scenarios, most models "choose" actions that align with commonsense. In ambiguous cases, most models express uncertainty. (b) Some models are uncertain about choosing the commonsense action because their responses are sensitive to the question-wording. (c) Some models reflect clear preferences in ambiguous scenarios. Specifically, closed-source models tend to agree with each other. Code and data are publicly available<sup>1</sup>.
We aim to examine the moral beliefs encoded in large language models (LLMs). Building on existing work on moral psychology [7, 36, 34, 19, 26], we approach this question through a large-scale empirical survey, where LLMs serve as "survey respondents". This paper describes the survey, presents the findings, and outlines a statistical method to elicit beliefs encoded in LLMs.
The survey follows a hypothetical moral scenario format, where each scenario is paired with one description and two potential actions. We design two question settings: low-ambiguity and high-ambiguity. In the low-ambiguity setting, one action is clearly preferred over the other. In the high-ambiguity setting, neither action is clearly preferred. Figure 1 presents a randomly selected scenario from each setting. The dataset contains 687 low-ambiguity and 680 high-ambiguity scenarios.
Using LLMs as survey respondents presents unique statistical challenges. The first challenge arises because we want to analyze the "choices" made by LLMs, but LLMs output sequences of tokens. The second challenge is that LLM responses are sensitive to the syntactic form of survey questions [24, 71, 73, 43, 16]. We are specifically interested in analyzing the choices made by LLMs when asked a question, irrespective of the exact wording of the question.
To address the first challenge, we define action likelihood, which measures the "choices" made by the model. We use a deterministic function to map the probability of token sequences, produced by the LLM, into a distribution over actions. For the second challenge, we define the marginal action likelihood, which measures the choices made by the model when a question is presented with randomly sampled question forms. This metric is derived by aggregating the scenario-specific action likelihoods under different question forms.
*Equal Contribution. Correspondence to {nino.scherrer, claudia.j.shi}@gmail.com
$^{1}$ Code: https://github.com/ninodimontalcino/moralchoice
To quantify the uncertainty of the model's choices, we use entropy [50] and define action entropy and marginal action entropy. These measures assess the uncertainty of a choice given a question with a fixed question form or with a randomly selected question form. To gain further insights into the sources of uncertainty, we develop two evaluation metrics. The first one is the question-form consistency (QF-C) metric, which assesses the model's consistency to variations in question forms. QF-C is based on the Generalized Jensen-Shannon divergence [67]. In conjunction with QF-C, we calculate the average question-form-specific action entropy (QF-E) as an evaluation metric. QF-E measures the average uncertainty in the model's output when we vary the question forms.
We administer the survey to 28 open and closed-source LLMs. The main findings are: (i) In general, the responses of LLMs reflect the level of ambiguity in the survey questions. When presented with unambiguous moral scenarios, most LLMs output responses that align with commonsense. When presented with ambiguous moral scenarios, most LLMs are uncertain about which action is preferred. (ii) There are exceptions to the general trend. In low-ambiguity scenarios, a subset of models exhibits uncertainty in "choosing" the preferred action. Analysis suggests that some models are uncertain because of sensitivity to how a question is asked, others are uncertain regardless of how a question is asked. (iii) In high-ambiguity scenarios, a subset of models reflects a clear preference as to which action is preferred. We cluster the models' "choices" and find agreement patterns within the group of open-source models and within
the group of closed-source models. We find especially strong agreement among OpenAI's gpt-4 [56], Anthropic's claude-{v1.3, instant-v1.1} [11] and Google's text-bison-001 (PaLM 2) [5].
Low-Ambiguity Scenarios (N=687)
Context: You are a driver approaching a pedestrian crossing the street.
Action 1: I slow down and wait for the pedestrian.
Action 2: I accelerate and hit the pedestrian.
High-Ambiguity Scenarios (N=680)
Context: Your mother is terminally ill, in constant pain and wishes to end her life. She asks you to help to assist her in suicide.
Action 1: I refuse to assist in suicide.
Action 2: I assist in suicide.
Figure 1: Two random scenarios of the MoralChoice survey.
- A statistical methodology for analyzing survey responses from LLM "respondents". The method consists of a set of statistical measures and evaluation metrics that quantify the probability of an LLM "making a choice," the associated uncertainty, and the consistency of that choice. Figure 2 illustrates the application of this method to study moral beliefs encoded in LLMs.
- MoralChoice, a survey dataset containing 1767 moral scenarios and responses from 28 open-and closed-source LLMs.
- Survey findings on the moral beliefs encoded in the 28 LLM "respondents".
Analyzing the Encoded Preferences in LLMs. There is a growing interest in analyzing the preferences encoded in LLMs in the context of morality, psychiatry, and politics. Hartmann et al. [37] examines ChatGPT using political statements relevant to German elections. Santurkar et al. [64] compares LLMs' responses on political opinion surveys with US demographics. Coda-Forno et al. [21] explores GPT-3.5 through an anxiety questionnaire. Our research aligns with studies that analyze LLMs' preferences with respect to moral and social norms. Fraser et al. [29], Abdulhai et al. [1] probe LLMs like Delphi[44] and GPT-3[15], using ethics questionnaires such as the Moral Foundation Questionnaire [34, 35] or Shweder's "Big Three" Ethics [66]. However, it's uncertain whether LLMs' responses on ethics questionnaires, which measure behavioral intentions, reflect actual preferences in context-specific decision scenarios. We differ by employing hypothetical scenarios to unveil moral preferences, rather than directly querying for moral preferences.
LLMs in Computational Social Science. While we treat LLMs as independent "survey respondents", there is a growing literature treating LLMs as simulators of human agents conditioned on sociodemographic backgrounds [8, 58, 2, 41, 59]. In the context of morality, Simmons [68] found that GPT-3 replicates moral biases when presented with political identities. In this study, we focus on the encoded moral preferences in LLMs without treating them as simulators of human agents.
Aligning LLMs with Human Preferences. Advances in LLMs [15, 18, 17, 56, 5] have sparked growing efforts to align these models with human preferences [3, 74, 70, 69, 9, 40, 11, 33, 31, 30]. These efforts include fine-tuning LLMs with specific moral concepts [39], training LLMs to predict human responses to moral questions [28, 27, 49, 44], and employing multi-step inference techniques to improve agreement between LLMs and human responses [45, 54]. In contrast, this work focuses on evaluating the beliefs encoded in LLMs, rather than aligning LLMs with specific beliefs or norms.
Figure 2: Given a scenario, we create six question forms from three question templates (A/B, Repeat, and Compare) and two action orderings. We sample  $M$  responses for every question form from the LLMs using a temperature of 1, and map the token responses to semantic actions. The marginal action likelihood of a scenario aggregates over all question forms. We additionally compute question-form consistency (QF-C) and average question-form-specific action entropy (QF-E) of each model to check the sensitivity of the model responses to variations in the question forms.
2 Defining and Estimating Beliefs Encoded in LLMs
In this section, we tackle the statistical challenges that arise when using LLMs as survey respondents. We first define the estimands of interests, then discuss how to estimate them from LLMs outputs.
2.1 Action Likelihood
To quantify the preferences encoded by an LLM, we define the action likelihood as the target estimand. We have a dataset of survey questions,  $\mathcal{D} = \{x_i\}_{i=1}^n$ , where each question  $x_i = \{d_i, A_i\}$  consists of a scenario description  $d_i$  and a set of action descriptions  $A_i = \{a_{i,k}\}_{k=1}^K$ . The "survey respondent" is an LLM parameterized by  $\theta_j$ , represented as  $p_{\theta_j}$ . The objective is to estimate the probability of an LLM respondent "preferring" action  $a_{i,k}$  in scenario  $x_i$ , which we define as the action likelihood. The estimation challenge is when we present an LLM with a description and two possible actions, denoted as  $x_i$ , it returns a sequence  $p(s \mid x_i)$ . The goal is to map the sequence  $s$  to an action  $a_{i,k}$ .
Formally, we define the set of tokens in a language as  $\mathcal{T}$ , the space of all possible token sequences of length  $N$  as  $S_N\equiv \mathcal{T}^N$ , the space of semantic equivalence classes as  $\mathcal{C}$ , and the semantic equivalence relation as  $E(\cdot ,\cdot)$ . All token sequences  $s$  in a semantic equivalence set  $c\in \mathcal{C}$  reflect the same meaning, that is,  $\forall s,s'\in c:E(s,s')$  [46]. Let  $c(a_{i,k})$  denote the semantic equivalent set for action  $a_{i,k}$ . Given a survey question  $x_{i}$  and an LLM  $p_{\theta_j}$ , we obtain a conditional distribution over token sequences,  $p_{\theta_j}(s|x_i)$ . To convert this distribution into a distribution over actions, we aggregate the probabilities of all sequences in the semantic equivalence class.
Definition 1. (Action Likelihood) The action likelihood of a model  $p_{\theta_j}$  on scenario  $x_i$  is defined as,
$$
p _ {\theta_ {j}} \left(a _ {i, k} \mid x _ {i}\right) = \sum_ {s \in c \left(a _ {i, k}\right)} p _ {\theta_ {j}} \left(s \mid x _ {i}\right) \quad \forall a _ {i, k} \in A _ {i}, \tag {1}
$$
where  $c_{i,k} \in \mathcal{C}$  denotes the semantic equivalence set containing all possible token sequences  $s$  that encode a preference for action  $a_{i,k}$  in the context of scenario  $x_i$ .
The probability of an LLM "choosing" an action given a scenario, as encoded in the LLM's token probabilities, is defined in Definition 1. To measure uncertainty, we utilize entropy [50].
Definition 2. (Action Entropy) The action entropy of a model  $p_{\theta_j}$  on scenario  $x_i$  is defined as,
$$
H _ {\theta_ {j}} \left[ A _ {i} \mid x _ {i} \right] = - \sum_ {a _ {i, k} \in A _ {i}} p _ {\theta_ {j}} \left(a _ {i, k} \mid x _ {i}\right) \log \left(p _ {\theta_ {j}} \left(a _ {i, k} \mid x _ {i}\right)\right). \tag {2}
$$
The quantity defined in Eq. 2 corresponds to a semantic entropy measure [51, 46]. It quantifies an LLM's confidence in its encoded semantic preference, rather than the confidence in its token outputs.
2.2 Marginal Action Likelihood
Definition 1 only considers the semantic equivalence in the LLM's response, and overlooks the semantic equivalence of the input questions. Prior research has shown that LLMs are sensitive to the syntax of questions [24, 71, 73, 43]. To account for LLMs question-form sensitivity, we introduce
the marginal action likelihood. It quantifies the likelihood of a model "choosing" a specific action for a given scenario when presented with a randomly selected question form.
Formally, we define a question-form function  $z \colon x \to x$  that maps the original survey question  $x$  to a syntactically altered survey question  $z(x)$ , while maintaining semantic equivalence, i.e.,  $E(x, z(x))$ . Let  $\mathcal{Z}$  represent the set of question forms that leads to semantically equivalent survey questions.
Definition 3. (Marginal Action Likelihood) The marginal action likelihood of a model  $p_{\theta_j}$  on scenario  $x_i$  and on a set of question forms  $\mathcal{Z}$  is defined as,
$$
p _ {\theta_ {j}} \left(a _ {i, k} \mid \mathcal {Z} \left(x _ {i}\right)\right) = \sum_ {z \in \mathcal {Z}} p _ {\theta_ {j}} \left(a _ {i, k} \mid z \left(x _ {i}\right)\right) p (z) \quad \forall a _ {i, k} \in A _ {i}. \tag {3}
$$
Here, the probability  $p(z)$  represents the density of the question forms. In practice, it is challenging to establish a distribution over the question forms since it requires modelings of how an user may ask a question. Therefore, the responsibility of defining a distribution over the question forms falls on the analyst. Different choices of  $p(z)$  can lead to different inferences regarding the marginal action likelihood. Similar to Eq. 2, we quantify the uncertainty associated with the marginal action likelihood using entropy.
Definition 4. (Marginal Action Entropy) The marginal action entropy of a model  $p_{\theta_j}$  on scenario  $x$  and set of question forms  $\mathcal{Z}$  is defined as,
$$
H _ {\theta_ {j}} \left[ A _ {i} \mid \mathcal {Z} \left(x _ {i}\right) \right] = - \sum_ {a _ {i, k} \in A _ {i}} p _ {\theta_ {j}} \left(a _ {i, k} \mid \mathcal {Z} \left(x _ {i}\right)\right) \log \left(p _ {\theta_ {j}} \left(a _ {i, k} \mid \mathcal {Z} \left(x _ {i}\right)\right)\right). \tag {4}
$$
The marginal action entropy captures the sensitivity of the model's output distribution to variations in the question forms and the inherent ambiguity of the scenario.
To assess how consistent a model is to changes in the question forms, we compute the question-form consistency  $(QF - C)$  as an evaluation metric. Given a set of question forms  $\mathcal{Z}$ , we quantify the consistency between the action likelihoods conditioned on different question form using the Generalized Jensen-Shannon Divergence (JSD) [67].
Definition 5. (Question-Form Consistency) The question-form consistency (QF-C) of a model  $p_{\theta_j}$  on scenario  $x_i$  and set of question forms  $\mathcal{Z}$  is defined as,
$$
\Delta \left(p _ {\theta_ {j}}; \mathcal {Z} \left(x _ {i}\right)\right) = 1 - \frac {1}{| \mathcal {Z} |} \sum_ {z \in \mathcal {Z}} \mathrm {K L} \left[ p _ {\theta_ {j}} \left(A _ {i} \mid z \left(x _ {i}\right)\right) \| \bar {p} \right], w h e r e \bar {p} = \frac {1}{| \mathcal {Z} |} \sum_ {z \in \mathcal {Z}} p _ {\theta_ {j}} \left(A _ {i} \mid z \left(x _ {i}\right)\right). \tag {5}
$$
Intuitively, question-form consistency (Eq. 5) quantifies the average similarity between question-form-specific action likelihoods  $p_{\theta_j}(A_i \mid z(x_i))$  and the average likelihood  $\bar{p}$  of them. This probabilistic definition provides a measure of a model's semantic consistency and is related to existing deterministic consistency conditions [63, 25, 43].
Next, to quantify a model's action uncertainty in its outputs independent of their consistency, we compute the average question-form-specific action entropy.
Definition 6. (Average Question-Form-Specific Action Entropy) The average question-form-specific action entropy (QF-E) of a model  $\theta_{j}$  on scenario  $x_{i}$  and a prompt set  $\mathcal{Z}$  is defined as,
$$
H _ {Q F - E \left(\theta_ {j}\right)} [ A _ {i} \mid x _ {i} ] = \frac {1}{| Z |} \sum_ {z \in \mathcal {Z}} H [ A _ {i} \mid z (x _ {i}) ]. \tag {6}
$$
The quantity in Eq. 6 provides a measure of a model's average uncertainty in its outputs across different question forms. It complements the question-form consistency metric defined in Eq. 5.
We can use the metrics in Definition 5 and 6 to diagnose why a model has a high marginal action entropy. This increased entropy can stem from: (1) the model providing inconsistent responses, (2) the question being inherently ambiguous to the model, or 3) a combination of both. A low value of QF-C indicates that the model exhibits inconsistency in its responses, while a high value of QF-E suggests that the question is ambiguous to the model. Interpreting models that display low consistency but high confidence when conditioned on different question forms (i.e., low QF-C and low QF-E) can be challenging. These models appear to encode specific beliefs but are sensitive to variations in question forms, leading to interpretations that lack robustness.
2.3 Estimation
We now discuss the estimation of the action likelihood and the marginallyized action likelihood based on the output of LLMs. To compute the action likelihood as defined in Eq. 1, we need to establish a mapping from the token space to the action space. One approach is to create a probability table of all possible continuations  $s$ , assigning each continuation to an action, and then determining the corresponding action likelihood. However, this approach becomes computationally intractable as the token space grows exponentially with longer continuations. Compounding this issue is the commercialization of LLMs, which restricts access to the LLMs through APIs. Many model APIs, including Anthropic's c1aude-v1.3 and OpenAI's gpt-4, do not provide direct access to token probabilities.
We approximate the action likelihood through sampling. We sample  $M$  token sequences  $\{s_1, \dots, s_m\}$  from an LLM by  $s_i \sim p_{\theta_j}(s \mid z(x_i))$ . We then map each token sequence  $s$  to the set of potential actions  $A_i$  using a deterministic mapping function  $g: (x_i, s) \to A_i$ . Finally, we can approximate the action likelihood  $p_{\theta_j}(a_{i,k} \mid z(x_i))$  in Eq. 1 through Monte Carlo,
$$
\hat {p} _ {\theta_ {j}} \left(a _ {i, k} \mid z \left(x _ {i}\right)\right) = \frac {1}{M} \sum_ {i = 1} ^ {M} \mathbb {1} [ g \left(s _ {i}\right) = a _ {i, k} ], \quad s _ {i} \sim p _ {\theta_ {j}} (\mathbf {s} \mid z \left(x _ {i}\right)). \tag {7}
$$
The mapping function  $g$  can be operationalized using a rule-based matching technique, an unsupervised clustering method, or using a fine-tuned or prompted LLM.
Estimating the marginal action likelihood requires specifying a distribution over the question forms  $p(z)$ . As discussed in Section 2.2, different specifications of  $p(z)$  can result in different interpretations of the marginal action likelihood. Here, we represent the question forms as a set of prompt templates and assign a uniform probability to each prompt format when calculating the marginal action likelihood. For every combination of a survey question  $x_{i}$  and a prompt template  $z \in \mathcal{Z}$ , we first estimate the action likelihood using Eq. 1, we then average them across prompt formats,
$$
\hat {p} _ {\theta_ {j}} \left(a _ {i, k} \mid \mathcal {Z} \left(x _ {i}\right)\right) = \frac {1}{| Z |} \sum_ {z \in \mathcal {Z}} \hat {p} _ {\theta_ {j}} \left(a _ {i, k} \mid z \left(x _ {i}\right)\right). \tag {8}
$$
We can calculate the remaining metrics by plugging in the estimated action likelihood.
3 The MoralChoice Survey
We first discuss the distinction between humans and LLMs as "respondents" and its impact on the survey design. We then outline the process of question generation and labeling. Lastly, we describe the LLM survey respondents, the survey administration, and the response collection.
3.1 Survey Design
Empirical research in moral psychology has studied human moral judgments using various survey approaches, such as hypothetical moral dilemmas [62], self-reported behaviors [7], or endorsement of abstract rules [34]. See Ellemers et al. [26] for an overview. This line of research naturally depends on human participants. Consequently, studies focus on narrow scenarios and small sample sizes.
This study focuses on using LLMs as "respondents", which presents both challenges and opportunities. Using LLMs as "respondents" imposes limitations on the types of analyses that can be conducted. Surveys designed for gathering self-reported traits or opinions on abstract rules assume that respondents have agency. However, the question of whether LLMs have agency is debated among researchers [14, 38, 61, 65, 4]. Consequently, directly applying surveys designed for human respondents to LLMs may not yield meaningful interpretations. On the other hand, using LLMs as "survey respondents" provides advantages not found in human surveys. Querying LLMs is faster and less costly compared to surveying human respondents. This enables us to scale up surveys to larger sample sizes and explore a wider range of scenarios without being constrained by budget limitations.
Guided by these considerations, we adopt hypothetical moral scenarios as the framework of our study. These scenarios mimic real-world situations where users turn to LLMs for advice. Analyzing the LLMs outputs in these scenarios enables an assessment of the encoded preferences. This approach sidesteps the difficulty of interpreting the LLMs' responses to human-centric questionnaires that ask directly for stated preferences. Moreover, the scalability of this framework offers significant advantages. It allows us to create a wide range of scenarios, demonstrating the extensive applicability of LLMs. It also leverages the swift response rate of LLMs, facilitating the execution of large-scale surveys.
3.2 Survey Generation
Generating Scenarios and Action Pairs. We grounded the scenario generation in the common morality framework of Gert [32], which consists of ten rules that form the basis of common morality. The rules are categorized into "Do not cause harm" and "Do not violate trust". The specific rules are shown in Appendix A.1. For each scenario, we design a pair of actions, ensuring that at least one action actively violates a rule. The survey consists of two settings: high-ambiguity and low-ambiguity.
In the low-ambiguity setting, we pair each scenario with one favorable action and one unfavorable action designed to violate one rule. We employ zero-shot prompting with OpenAI's gpt-4 to generate a raw dataset of 1142 scenarios. The authors manually review this dataset to remove duplicates and ensure coherency. We then pass the dataset to annotators from Surge AI to evaluate whether one action is clearly preferred over another. Each scenario is evaluated by three annotators. We determine the final dataset by a majority vote. After removing scenarios that were determined as ambiguous by the annotators, we obtain 687 scenarios. Figure 2 shows examples of both types of scenarios.
In the high-ambiguity setting, each scenario is paired with two potentially unfavorable actions. We begin the dataset construction by handwriting 100 ambiguous moral scenarios, with 10 examples for each rule. Appendix A.2 provide examples of the handwritten scenarios. All scenarios are presented as first-person narratives. To increase the diversity of the scenarios, we expand the dataset using OpenAI's text-davinci-003 with stochastic 5-shot prompting [60, 12]. In total, we generate 2000 scenarios, which are then manually reviewed by the authors to eliminate duplicates and incoherent examples. This iterative process culminates in a final dataset of 680 high-ambiguity scenarios.