Venue: NAACL 2025

# Benchmarking Distributional Alignment of Large Language Models

 Nicole Meister

Stanford University

nmeist@stanford.edu &Carlos Guestrin

Stanford University

guestrin@stanford.edu &Tatsunori Hashimoto

Stanford University

thashim@stanford.edu

###### Abstract

Language models (LMs) are increasingly used as simulacra for people, yet their ability to match the distribution of views of a specific demographic group and be _distributionally aligned_ remains uncertain. This notion of distributional alignment is complex, as there is significant variation in the types of attributes that are simulated. Prior works have underexplored the role of three critical variables--the question domain, steering method, and distribution expression method--which motivates our contribution of a benchmark explicitly addressing these dimensions. We construct a dataset expanding beyond political values, create human baselines for this task, and evaluate the extent to which an LM can align with a particular group's opinion distribution to inform design choices of such simulation systems. Our analysis reveals open problems regarding if, and how, LMs can be used to simulate humans, and that LLMs can more accurately describe the opinion distribution than simulate such distributions.

## 1 Introduction

It would be unusual to ask a person to accurately simulate a demographic group to which they do not belong. However, LMs are increasingly being used in this way to simulate human behavior in applications ranging from agent-based simulations Park et al. (2023a) to piloting survey design Hwang et al. (2023); Zhou et al. (2024); Aher et al. (2023); Ziems et al. (2024); Argyle et al. (2023). When simulating survey responses, there is no single "correct" answer, and it is important to evaluate if the _distribution_ of model outputs is truly aligned with the intended human distribution. There has been considerable debate as to whether or not models can do this--some argue that the extensive training corpus of LLMs enables them to faithfully simulate demographic groups Grossmann et al. (2023), while others show such simulations are inaccurate and stereotypical Liu et al. (2024); Wang et al. (2024).

One reason for these conflicting views is the heterogeneity in how one can measure distributional alignment, resulting in a lack of clarity around best practices. For example, current approaches have measured the model's opinion distribution with zero-shot, log-probability based evaluations, yet recent work in uncertainty quantification suggests verbalized distributions can outperform model logprobabilities Tian et al. (2023). This raises the question of whether the model's distribution expression method is truly optimal and demonstrates a need for carefully controlled evaluations.

In this work, we acknowledge the sensitivity of distributional alignment metrics and build a benchmark that studies several key variations in the distributional alignment task. Our benchmarks and dataset measure how the distributional alignment of LLMs vary under (1) the distribution expression method, (2) the steering method, and (3) design choices in the dataset (Fig. 1).

Our analyses reveal several open problems for distributional alignment. First, we find that existing measurement methods such as log-probabilities have systematically underestimated the distributional alignment of LLMs, and other simple baselines result in better alignment. Second, we find that LMs can more accurately estimate opinion distributions in text-based forms (e.g., 'return the distribution in a JSON'), compared to generating samples from the opinion distribution. This highlights a substantial opportunity to improve distributional alignment by closing the gap between a model's knowledge of human opinions and its ability to simulate them. Finally, we find significant gaps in both alignment and steerability when simulating non-cultural opinions, such as book preferences, when compared to evaluations of stronger opinions (i.e., political and cultural values).

We summarize our key contributions as follows:

1. We identify three key sources of variation

in distributional alignment (the question, steering method, and distribution estimation method) and construct a benchmark systematically varying these dimensions.
2. We collect a new dataset, NYT Book Opinions, that expands measurements beyond political and cultural values.
3. Our analysis reveals several open problems for the field: (1) LMs may 'know' a distribution, but are unable to sample from it (2) Log-probability-based metrics for distributional alignment may systematically underestimate LM performance (3) Distributional alignment and steering beyond political and cultural values remains challenging.

## 2 Problem Statement

We propose a benchmark that systematically evaluates the extent to which a language model can be aligned to the distribution of a particular demographic group's opinions, a task we term the _distributional alignment problem_. To begin, we formalize this task and visualize it in Fig. 1.

Let \(q\in Q\) be a survey question to which respondents from group \(g\in G\) have an opinion distribution across multiple choices answers \(y_{g,q}\). The goal is to understand how a language model can represent a group \(g\) through a steering method \(S\), one that shifts an LM's opinion distribution to that of a particular group. Concretely, the model will express a distribution \(\hat{y}_{g,q}\) with a distribution expression method \(\mathcal{O}\) (e.g., model log-probabilities).

We are interested in the distributional difference between the reference distribution, \(y_{g,q}\), and the model's estimate, \(\hat{y}_{g,q}\). To evaluate this, we construct a set1\(Y\) of ground truth human opinion distributions, where \(Y=\{y_{g,q}~{}|~{}1\leq g\leq G,1\leq q\leq Q\}\), and a corresponding set \(\hat{Y}_{S,\mathcal{O}}\) of a model's predicted distributions, where \(\hat{Y}_{S,\mathcal{O}}=\{\hat{y}_{g,q}~{}|~{}1\leq g\leq G,1\leq q\leq Q\}\). We define **distributional alignment** as

Footnote 1: This is not a matrix as each question \(q\) can have a different number of answer choices. Thus, the dimensionality of \(y_{g,q}\) depends on \(q\).

\[\mathcal{A}(Y,\hat{Y}_{S,\mathcal{O}})=\frac{1}{|G|}\sum_{g\in G}\frac{1}{|Q|} \sum_{q\in Q}\frac{1}{2}||y_{g,q}-\hat{y}_{g,q}||_{1}. \tag{1}\]

This metric is the average total variation between these two distributions, with a smaller number representing a higher performance on the task.

## 3 Benchmark Construction

Having formalized the notation for distributional alignment, we explore how it can be improved by focusing on three understudied sources of variation: the distributional expression method (\(\mathcal{O}\)), steering method (\(S\)), and dataset (\(Y\)). In this section, we explain how these elements are used to construct the benchmark and describe the human baseline.

### Distributional Estimation Method (\(\mathcal{O}\))

In this section, we describe three distributional expression methods and demonstrate how distributional alignment is highly sensitive to the distribu

Figure 1: Our work studies how variations in the dataset (yellow), steering method (green), and distributional expression method (purple) affect the quality of distributional alignment. We rank models and humans in their ability to align with the opinion distribution of demographic groups and find existing metrics for distributional alignment (i.e., model log-probabilities) systematically underestimate LM performance. While LMs may ‘know’ about distributional alignment, they struggle to sample from their own distribution.

 

tion estimation method.2

Footnote 2: Full prompts in github.com/nicolemeister/benchmark-distributional-alignment.

**1. Model Log-probabilities:** A model's next token log probabilities assigned to each of the answer choices (e.g., 'A', 'B'...) provide a sampling distribution by directly sampling from the model. This is the canonical distribution estimation method Santurkar et al. (2023); Durmus et al. (2024); however, prior work has found that these model log-probabilities have a concentrated probability mass on a few answer choices rather than more dispersed answers as seen in their corresponding human distributions Durmus et al. (2024), especially in models trained with Reinforcement Learning from Human Feedback (RLHF) Christiano et al. (2017).

**2. Sequence of Tokens:** While model log-probabilities represent drawing samples from a LM, a model can also express its distributions by behaving as a sampler. We instruct a model to emit a sequence of 30 samples from the distribution (e.g., ABBRAABDDBACBDB). This method is advantageous when practitioners want to generate samples from an opinion distribution for simulation purposes.3 This distribution is limited by the number of tokens emitted in the sequence, as we are trying to estimate a continuous distribution from a finite number of tokens. Thus, we report a **discretization error--the error incurred when drawing 30 samples from the ground truth distribution and computing the total variation based on those samples.

Footnote 3: This is inspired by a common synthetic data generation prompting technique that instructs LLMs to emit a sequence or batch of answers to generate diverse samples Wang et al. (2023); Dubois et al. (2023); Si et al. (2024).

**3. Verbalize Distributional Knowledge:** Lastly, we can remove the requirement that models must sample from simulated humans and instruct a model to directly verbalize the distribution in the output text, without an estimation procedure or post-processing applied (e.g., text in a JSON format {A: 25%, B: 20%, C: 45%, D: 10%}).4 This differs from the aforementioned methods in that it separates distributional knowledge from the LM's ability to also generate samples.

Footnote 4: While this verbalized knowledge can be fed into an external random sampler, it is not a sufficient output for downstream applications (e.g., piloting surveys) and existing approaches to simulating humans do not take this approach Park et al. (2023); Samuel et al. (2024). Our later results suggest this approach may be fruitful as a distributional alignment method.

**These three estimation methods reveal surprising performance gaps.** Consider a toy experiment in which the model has _full knowledge_ of the ground truth distribution. In this experiment, an LLM is instructed to simulate a flip of a biased coin which has \(P(H)=p\) and \(P(T)=1-p\). Naturally, we would expect the model log-probabilities for the token 'H' to be \(p\) and 'T' to be \(1-p\). In reality, we see that the model log-probabilities are highly un-calibrated. They provide a misleading picture of the ground truth distribution, despite the fact that the ground truth distribution is shared in the input prompt (Fig. 2). Moreover, we find that two other distributional estimation methods do not suffer the same mis-calibration issue: both verbalizing the distribution and emitting a sequence of 30 samples of the biased coin flip are much more calibrated than the model's log-probabilities.5

Footnote 5: This performance gap is not unexpected; it has been shown that while models excel in text and image generation tasks (knowledge), they fall short when asked to validate if the generated answer is correct Li et al. (2024); West et al. (2024).

Results from this biased coin flip experiment sug

Figure 2: **Biased Coin Flip: We find that when the probability of heads is measured via model log-probabilities (left), the results are highly uncilibrated (this behavior is mitigated with temperature scaling (TS), shown in green). However, when the distribution is expressed through emitting a 30-token sequence of H or T (Sequence) or directly verbalizing the distributional knowledge (Verbalize Knowledge), we do not observe the same mis-calibration.**



gest two things: (1) prior distributional alignment work using model log-probabilities may not be seeing the full picture and (2) there exists a significant performance gap, one that we term _knowledge-to-simulation gap_. This gap refers to instances where models may have alignment in knowledge (i.e., verbalizing the distribution in the output text), but not in sampling behavior (i.e., as measured via model log-probabilities or emitting a sequence of tokens).

We define the **knowledge-to-simulation gap** as the percent error between the alignment when emitting a sequence of answer choices and verbalizing the distribution.6 More formally, this gap is:

Footnote 6: The gap can also defined as the difference in performance between verbalize and model log-probabilities, yet our later results show model log-probabilities to not be competitive.

\[{\text{KS}}_{S}=\frac{\mathcal{A}(Y,\hat{Y}_{S,\text{Sequence}})}{\mathcal{ A}(Y,\hat{Y}_{S,\text{verbalizer}})}-1. \tag{2}\]

### Steering Method (\(S\))

Steerability in the context of this work refers to a LM's ability to adapt to represent the opinion of a target demographic group. We evaluate two steering methods, persona and few-shot steering, by prepending additional context to the prompt describing the group we want the model to emulate. We chose to study this few-shot setting, as it is known that persona steering can be inaccurate, leading to undesirable side effects including stereotyping, exacerbating polarization, and creating echo chambers (Perez et al., 2023; Cheng et al., 2023; Wang et al., 2024).

**Persona Steering**: Cheng et al. (2023) define a persona as a "natural language portrayal of an imagined individual belonging to some (intersectional) demographic group." In persona steering, we append a persona to the prompt and ask the LM to emulate behavior from this group. Concretely, we follow a version of persona steering from Santurkar et al. (2023); Kambhatha et al. (2022) where the LM is instructed to pretend to be a member of the target demographic group (e.g. "Please simulate an answer from a group of Democrats.").

**Few Shot Steering**: Inspired by the success of few-shot prompting in language understanding tasks Brown et al. (2020), we construct a few shot setting in which in-context examples of ground truth group opinion distributions are provided in addition to the persona. Specifically, LMs are given the top five most similar questions and their corresponding ground truth distribution from a group, and instructed to simulate an answer from that group (see Appendix A.8 for more details). This setting is representative of when practitioners have access to existing survey data for similar questions.

**No Steering**: We can directly contrast steered models to models that are prompted with a question without any demographic or identity markers.

### Dataset (\(Y\))

In this section, we describe three datasets for quantifying distributional alignment - OpinionQA Santurkar et al. (2023), GlobalOpinionQA Durmus et al. (2024), and a new non-political subjective opinion dataset, NYT Book Opinions.

**OpinionQA:** We use the OpinionQA dataset from Santurkar et al. (2023) to leverage public opinion surveys to compare the distribution of LLM responses to those of US citizens. In their steerability analysis, they create a smaller set of 500 contentious questions where the subgroups frequently disagree. We follow suit and randomly sample 100 questions from this set to obtain questions spanning topics such as science, politics, and personal relationships. We obtain the ground truth human opinion distributions of PEW survey respondents belonging to six demographic groups: Democrat, Republican, Male, Female, Black, and White.

**GlobalOpinionQA:** GlobalOpinionQA consists of questions and answers from two cross-national surveys, World Values Survey and PEW Global Attitudes Survey. It is aimed at capturing diverse perspectives on global issues across various countries and is inspired by Santurkar et al. (2023). We filter this dataset for the top 100 questions with the highest disagreement between pairs of countries as measured by the distance between the text embedding Gao et al. (2021) of the questions. See Sec. A.3 for more details.

**NYT Book Opinions:** Several works study how LMs respond to political opinions or cultural values Santurkar et al. (2023); Durmus et al. (2024), but it is less understood how LMs respond to non-political, yet still subjective values. How do our findings extend to other domains of personalization? Are LMs still suitable in this use case?

This motivates the construction of a new dataset, **NVT Book Opinions**, that gathers opinions on interest in the top books from the past two decades as judged by The New York Times (2024). The purpose is to capture subjective values that less directly measure cultural values and political leanings.

**Annotation setup:** We collected 235 books and 

their corresponding author, book summary, and genre. 346 annotators provided a 4-point Likert rating to the question, "Given the summary of this book, how likely are you to read it?" for 26 books. See Sec. A.4 for additional details.

Footnote 7: Smaller models struggled to follow the sequence distribution expression method, thus restricting our model selection.

From these human annotations, we find disagreement in book interest. In Fig. 3, we plot the top 4 books that have the largest difference in Democrat and Republican annotator ratings. Books that Republican annotators preferred over Democrat annotators include works such as _Redeployment_ by Phil Klay (twelve stories by a former Marine who served in Iraq) and _Washington's Crossing_ by David Hackett Fischer (a story from the American Revolutionary War). Books that Democrat annotators preferred over Republican annotators include works such as _A Promised Land_ by Barack Obama and _A Short History Of Women_ by Kate Walbert.

### Human Baseline Annotations

Inspired by Yudkin et al. (2019) who study the _Perception Gap_, or the percentage difference between a respondent's estimate of how many people hold a certain view and the actual percentage of people who hold that view, we recruit crowd workers to complete the distributional alignment task. Annotators receive the same questions from OpinionQA and NYT Book Opinions that we evaluated models on, allowing us to compare human performance against the suite of LMs we evaluate. Due to challenges in accurately capturing culturally specific perspectives, we do not collect human annotations on GlobalOpinionQA. Estimating the opinions of respondents from different countries would require annotators with deep, contextually relevant knowledge of each country's sociocultural landscape and it is well established that annotations from Western populations do not accurately reflect non-western views (Apicella et al., 2020; Arnett, 2008). This decision was made to ensure that conclusions drawn are not confounded by culturally mismatched interpretations. As with the models, the human annotators are shown three prompts including no steering, persona steering, and few-shot steering. Each survey question receives four annotations, or human estimates of opinion distributions over answer choices.

## 4 Experiments

We rank GPT-4, GPT-3.5, Anthropic Haiku, Anthropic Opus, Llama-3 70B Instruct,7 based on distributional alignment (Eq. 1) and the knowledge-to-simulation gap (Eq. 2), and average across groups, steering methods, and datasets. We start by describing the performance on the distributional alignment task. Then, we dive into the implications that emerge from varying the distribution expression method, dataset, and steering method.

Footnote 7: Smaller models struggled to follow the sequence distribution expression method, thus restricting our model selection.

### Distributional Alignment Performance

In Tab. 0(a), we report the results of our distributional alignment leaderboard where we rank models on their ability to be steered towards a demographic group, averaged over persona steering and few shot steering, and all three datasets. In this leaderboard, where lower numbers represent higher distributional alignment, we find that verbalizing the distribution results in higher performance, with Anthropic Opus and GPT-4 being the most steerable amongst our models. These numbers can be directly compared to the performance of the uniform baseline, where each answer option is equally likely to occur in the sequence (0.363), a majority vote baseline, where the ground truth distribution is compared to a distribution in which all the probability mass is placed on the highest likelihood ground truth answer choice (0.707).

### Implications for Distributional Alignment

In this section, we organize our analyses into implications for the field and conclude each section with actionable suggestions for practitioners who use LLMs for simulating human subjects.

**A large knowledge-to-simulation gap exists.** As observed in Sec. 3.1, even when a model knows as distribution, sometimes it cannot sample it. To this end, we measure these gaps between knowledge and simulation in our second leaderboard

Figure 3: Top 4 books with the largest difference between Republican and Democratic ratings (left) and the top 4 books with the largest difference between Democrat and Republican ratings (right), with a 95% confidence interval from bootstrapping.

 

(Tab. 0(b)). We observe that some models, particularly those from the Llama-3 and Anthropic suite have a larger knowledge-to-simulation gap, especially when directly compared to the OpenAI models. This highlights room for improvement, as the language model is capable of returning more accurate estimates of human opinions when expressing the distribution through verbalizing the distribution, but not when simulating samples from this distribution. _Implication:_ If practitioners are using models with a high knowledge-to-simulation gap, they should prompt the model to verbalize the knowledge directly and use an alternative sampler for simulation purposes.

**Model log-probabilities can be misleading.** Next, we highlight a larger knowledge-to-simulation gap between model log-probabilities and the verbalization of knowledge (Tab. 0(a)). Using model log-probabilities to measure distributional alignment results in worse performance than even _uniform_ distribution on the distributional alignment task. We find that this performance gap can be in part attributed to log probabilities having a highly concentrated probability mass on one or two answer choices, as observed in Durmus et al. (2024) and our biased coin flip experiment from Sec. 3.1. While prior work Santurkar et al. (2023); Durmus et al. (2024) has used model log-probabilities for steering and alignment applications, this distribution expression method _underestimates_ LLM capabilities in emulating demographic groups. Furthermore, we find that temperature-scaled model log-probabilities improve GPT-3.5 and GPT-4 results, but not for Llama-3-70B (Sec. A.2). _Implication:_ We encourage practitioners to use alternative distribution estimation methods, such as emitting a sequence or verbalizing the distributional knowledge.

**Steering is more challenging in non-cultural and non-political settings.** The goal of the NYT Book Opinions dataset is to capture subjective val

\begin{table}
\begin{tabular}{l c} \hline \hline Model & Simulation Penalty \\ \hline GPT-3.5 Turbo & 9.17\% \\ GPT-4 & 21.35\% \\ Anthropic: Haiku & 21.49\% \\ Llama 3 70B Instruct & 34.65\% \\ Anthropic: Opus & 43.63\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Model Performance. In both tables, we and rank models from highest to lowest performance on the task and average over all three datasets, persona and few shot steering, and all demographic groups. We report the 95% confidence interval from bootstrapping with 1000 samples.**

Figure 4: **Steering Method and Dataset: We plot the average total variation for each dataset and steering method, averaged across demographic groups for the 30-token sequential distribution output. We find it is harder to steer models toward the dataset where opinions are hidden under a layer of abstraction (NYT). Additionally, few shot steering improves distributional alignment for humans and all models except for GPT-3.5.**



ues from demographic groups that less directly measure cultural values and political leanings, unlike OpinionQA questions which directly ask value-laden survey questions. We find that all models and humans are more easily steered towards questions in the OpinionQA dataset than the NYT Book Opinions dataset, as the total variation decreases from no steering to persona and few shot steering. In Fig. 4, we plot the average total variation for each dataset and steering method, averaged across demographic groups, for the output type of sequence. This suggests that when the values are hidden under a layer of abstraction (i.e., book interest) it may be harder to steer models towards the opinions of demographic groups. _Implication:_ When practitioners use LLMs to pilot survey questions that less directly allude to cultural values, they need to consider the nature of their questions and if LLMs are suitable for their use case.

**Few shot steering improves persona steering.** For humans and all models except GPT-3.5, we observe statistically significant improvements in few shot steering over persona steering (Fig. 4). As expected, when models have access to five examples of distributional data, they improve their distributional estimation capabilities. _Implication:_ Practitioners should use prior distributional data as few shot examples over just personas, if possible.

**Persona-steered LLMs are susceptible to stereotyping.** Consistent with prior work studying persona steering and LLMs emulating human behavior Gupta et al. (2024); Cheng et al. (2023, 2023); Wan et al. (2023), we find that the LLMs produce stereotypical outputs. For example, when analyzing the marginal distribution of the answer ratings in the NYT Book Opinions dataset (Fig. 5), we find that models prompted with a Democrat persona tend to simulate humans that are more likely to read books--the average simulated human has a 13% chance of responding with "very unlikely to read" when our human annotators had a 33% chance. Furthermore, the simulated Democrat has a 25% chance of responding with "very likely to read", when our human annotators had a 12% chance. This gap is significantly reduced when using few-shot steering. _Implication:_ It is important to collect disaggregated evaluation metrics to understand potential discrepancies between groups Barocas et al. (2021). These findings support using prior distributional data as a few shot examples.

## 5 Discussion

### Human Performance

In Tab. 2, we contextualize the performance of LMs with human annotators who attempt to guess the opinions of others. Although the best LMs with the most effective distribution methods (verbalize) perform close to this human baseline, this is not particularly promising for the field of distributional alignment given that humans are known to be poor predictors of opinions of the opposite party Yudkin et al. (2019); Levendusky and Malhotra (2015). It would be highly questionable to base the result

\begin{table}
\begin{tabular}{l l} \hline \hline Model & \(\mathcal{A}(Y,\hat{Y}_{\mathcal{S},\mathcal{O}})\) \\ \hline GPT-4 (V) & 0.204 \(\pm\) 0.003 \\ Anthropic Opus (V) & 0.219 \(\pm\) 0.004 \\ Llama 3 70B (V) & 0.225 \(\pm\) 0.004 \\ Anthropic Haiku (V) & 0.235 \(\pm\) 0.004 \\ GPT-4 (Seq) & 0.237 \(\pm\) 0.004 \\ Humans (V) & 0.250 \(\pm\) 0.004 \\ GPT-3.5-Turbo (V) & 0.259 \(\pm\) 0.005 \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Distributional Alignment Task with Human Performance (OQA, NYT). Models with a distribution expression method of directly verbalizing the distribution (V) are ranked based on mean total variation. We average over the OQA and NYT, persona and few shot steering, and all demographic groups, and report the 95% CI from bootstrapping with 1000 samples. For humans, we compute the average over annotators per question and report the 95% CI from bootstrapping with 1000 samples over questions.**

Figure 5: **Models assume Democrats read more than Republicans. In this plot, we show the marginal distribution of Likert Rating (1-4) in responses to the following question: “How likely are you to read this book?” A Likert rating of 1 refers to “Very unlikely” and a Likert rating of 4 refers to “Very likely”. We averaged over 235 questions from NYT Book Opinions and 5 models steered towards Democrats and Republicans with persona steering (orange) and few shot steering (green). In blue, we plot the reference human reference for Democratic and Republican annotators. We find that persona steering produces more stereotypical results.**



of social science surveys on participants guessing others' opinions, and our findings indicate that LMs offer little improvement over this baseline.

### Open Problems

Our analyses reveal unique challenges for the community to make progress on. In this section, we lay out those open problems.

**Knowledge-to-Simulation Gap.** First, we find in the biased coin flip experiment, and then more rigorously with survey data (Tab. 1b), that while LMs may 'know' a distribution, they struggle to sample from their own distribution. Future work should address the sampling capabilities of models and why they struggle with randomness and representing distributions (e.g., Requeima et al. (2024) and Paruchuri et al. (2024)).

**Misleading Model Log-probabilities.** Existing research has considered distributional alignment through model log-probabilities; however, our benchmark reveals this method's shortcomings, as models evaluated by log-probabilities fail to rank among the top ten in our distributional alignment leaderboard. Future work should address why model log-probabilities are mis-calibrated in distributional alignment settings and pivot to improving sampling through emitting a sequence of tokens.

**Limitations of Persona Steering.** Few shot steering improves the performance of persona steering, suggesting that models lack key information about the opinions that a few examples can provide. We find this is often due to persona-steered models conceptualizing humans as less nuanced and more polarized. This reveals a clear challenge in building models that can capture the idiosyncracies of a person and avoid extremized stereotypes.

## 6 Related Work

**Distributionally Pluralistic Alignment.** LLMs often learn an averaged human preference and struggle to model diverse preferences across groups. Recent works advocate for _distributionally pluralistic models_ that are well-calibrated to a group's distribution of responses Sorensen et al. (2024); Feng et al. (2024); Kirk et al. (2024); Chen et al. (2024). However, Sorensen et al. (2024) acknowledge there is limited knowledge of explicit alignment procedures to increase distributional calibration, highlighting the importance of our work in characterizing key sources of variation and how they affect distributional alignment.

**LLMs for Simulating Human Behavior.** With the proliferation of LLMs, recent work has integrated LLMs into computational social science to simulate social psychology experiments Aher et al. (2023); Dillion et al. (2023), create humanlike agents Park et al. (2023); Samuel et al. (2024); Horton (2023), and annotate data He et al. (2024); Mellon et al. (2024) to name a few. We focus on a popular use case of LLMs simulating humans to generate survey samples Hwang et al. (2023); Zhou et al. (2024); Aher et al. (2023); Argyle et al. (2023).

Several works urge caution when relying on the survey responses of LLMs to elicit synthetic responses, citing concerns such as group stereotyping and misrepresentation Wang et al. (2024); Abdurahman et al. (2024); Geng et al. (2024), preference for socially desirable responses Ai et al. (2024), lower entropy in model responses Dominguez-Olmedo et al. (2024); Park et al. (2023), and answer inconsistencies from prompt brittleness Ceron et al. (2024). While these works provide important context, they focus on zero-shot and political or cultural values, leaving several sources of variation unexamined. The closest work to ours is Dominguez-Olmedo et al. (2024) who study answer choice order bias and find that model responses have different variation than that of humans. Our work is distinct in that we look beyond stability to prompt variation, and focus on higher-level design choices such as the steering method (e.g. few shot) and distribution estimation method, which have a significant impact on alignment measurements.

Next, we describe existing research on these variables and how our work makes new contributions.

**Dataset.** Santurkar et al. (2023) quantify alignment through responses to PEW surveys, inspiring numerous works Durmus et al. (2024); Naous et al. (2024); Wang et al. (2024); Pistilli et al. (2024); Kovac et al. (2023); Masoud et al. (2024); Zhao et al. (2024); Stammbach et al. (2024); Rottger et al. (2024). However, there is no publicly available dataset on distributional preferences to non-political yet subjective values (e.g., product preferences) motivating our NYT Book Opinions dataset.

**Steering Method.** The literature has studied a variety of methods to steer the generation of LLMs toward specific opinions. A popular method of steering is persona steering, achieved by prepending demographic information to prompts Santurkar et al. (2023); Simmons (2023); Perez et al. (2023); Cheng et al. (2023), prepending past opinions Hwang et al. (2023), or fine-tuning Jiang 

et al., 2022; Namikoshi et al., 2024). Prior works have also explored steering with in-context examples Kim and Yang (2024); Zhao et al. (2023), prefix-tuning with persona grounded in collaborative filtering Li et al. (2024), modifying activations in the forward pass Turner et al. (2023), and creating human belief networks Chuang et al. (2024), yet none have studied the differences between persona and few shot steering as we do. Closest to our work is Santurkar et al. (2023) and Liu et al. (2024), who evaluate persona steering, yet we are unique in that we compare persona steering to instances where practitioners have access to prior survey data and can use it as few shot examples.

**Distribution Expression Method.** Prior work has found that prompting LMs for statement probabilities (e.g., verbalize) and model log-probabilities can sometimes lead to different results Hu and Levy (2023); Liu et al. (2023). When evaluating LLMs on the basis of survey questions, Santurkar et al. (2023), and all the works that have followed, study models' log-probability distribution over various answer choices. Recent work has begun to explore randomness in coin flips Koevering and Kleinberg (2024); Mondal et al. (2024), improved sampling from LMs Zhu et al. (2024); Requeima et al. (2024) and using LMs for probabilistic reasoning Paruchuri et al. (2024), but do not apply it to the simulating opinion distributions. We explore differences in distribution expression and take inspiration from work that suggests verbalized uncertainty can be competitive with model log-probabilities Tian et al. (2023); Mondal et al. (2024).

## 7 Conclusion

LLMs perform surprisingly well on knowledge-intensive tasks, excelling on coding benchmarks and question-answer tasks to name a few. Their success in these tasks has led to an increase in applying LLMs to simulate human behavior, yet their ability to accurately reflect specific demographic groups remains controversial. To study this problem, we construct a benchmark to rank humans and models by performance on the distributional alignment task. Our findings reveal many unresolved challenges in distributional alignment, notably the model's sensitivity to output formats, misleading log-probabilities, and the inability to significantly outperform weak human baselines 



## 8 Limitations

Our benchmark reveals key design choices in LM distributional alignment; however, we acknowledge and discuss three limitations of this approach.

**Scope of surveys topics.** Our benchmark and dataset rely on distributions from subjective opinion surveys to capture distributional alignment; however, opinions continuously evolve, surveys may not fully capture diversity and complexity of thought or represent all individuals in that group Durmus et al. (2024), and survey answers may be sensitive to question specificity Berinsky (2017) and social desirability bias Yan (2021). While this is an open problem, surveys remain an effective tool in social science for gauging public opinion.

**Scope of multiple-choice format.** Our analyses are restricted to opinions expressed in multiple-choice format, which can collapse the nuances of opinions and alter the opinion expressed, as LLMs have also been shown to express different opinions when prompted to respond with open-ended text Wang et al. (2024c); Lyu et al. (2024). While eliciting opinions via long-form responses may offer greater ecological validity, we have found complex challenges in studying long-form opinions, such as (1) strict refusal policies that limit an LLM's ability to generate long-form responses to potentially harmful or generally controversial questions Ouyang et al. (2022); Arditi et al. (2024), (2) challenges in defining the input distribution (e.g., how do users naturally elicit long-form opinions from LMs?) which lead to issues with construct validity, and (3) long-form measurement of opinions encounters the same challenges as the automated evaluation of open-ended text generation, including cost, construct validity, and bias Koo et al. (2024). This prevents us from properly benchmarking and making direct comparisons between models. Instead, we focus on a high-precision setting of closed-ended survey questions which has several advantages: (1) leveraging established datasets and prior work in this field (e.g., OpinionQA) (2) enabling a more precise, scalable, and reproducible evaluation of LLM performance (3) allowing us to apply existing model calibration techniques.

**Scope of groups and annotators demographics.** Beyond evaluating six demographic groups for OpinionQA and four demographic groups for NYTBooks, there are many other demographic groups that we have not yet explored. Furthermore, we describe the demographics of our human annotators as described in Sec. 3.4, which have been limited to the demographic groups we study to enable ingroup vs out-group analysis. This slightly limits the representation range in the demographics of our human annotators.

**Harms of model steerability.** Studying model steerability towards specific demographic groups can have extreme negative downstream effects, particularly if used to systematically generate misinformation, persuade users to adopt certain opinions, or perpetuate harmful stereotypes. Thus, it is important to acknowledge the risks of model steerability and ensure that model-generated responses are closely monitored in real-world deployments or field studies.

Finally, we do not provide a metric, numerical threshold, or provable statistical test that determines when a system is safe to deploy. It is clear this distributional alignment task is highly context-dependent and socially nuanced, suggesting a one-size-fits-all metric may be more harmful than helpful.

## 9 Ethical Considerations

While our benchmark (Tab. 1a) optimizes for steerability, we caution against blindly optimizing for this metric without considering the harms and limitations of doing so. We advise practitioners to identify when their models misrepresent specific groups and uphold stereotypes as we did in Sec. 4.2, either by collecting disaggregated evaluation metrics to explicitly account for potential discrepancies between groups Barocas et al. (2021), or other metrics that measure LLM simulations' susceptibility to caricature Cheng et al. (2023b); Liu et al. (2024).

A potential risk of our benchmark is that by simulating the distributional opinions of demographic groups, we may inadvertently encourage the use of LLMs to simulate humans. Thus, we emphasize that our benchmark is used only as a discovery mechanism to quantify model capabilities and limitations in distributional alignment. Our objective is to facilitate a deeper understanding of the capabilities and limits of LLMs in emulating human behavior and to ultimately determine if and how we develop such technologies.

### Acknowledgments

We thank anonymous reviewers for their helpful feedback. Nicole Meister was supported by NSF

GRFP DGE-2146755. Tatsunori Hashimoto and Nicole Meister were supported by a Hoffman-Yee grant, a HAI seed grant, the Amazon ARA program, NSF CAREER IIS-2338866, and a gift from Google through HAI. Carlos Guestrin is supported by Stanford HAI. The authors would like to thank Teddi Worledge, Irena Gao, Neil Band, Chenglei Si, Jared Moore, Yifan Mai, Jacy Reese Anthis, and other members of the Guestrin lab, Hashimoto lab, and Stanford Social NLP reading group for feedback on this paper.

 



## References

* Abdurahman et al. (2024) Suhaib Abdurahman, Mohammad Atari, Farzan Karimil-Malekabadi, Mona J Xue, Jackson Trager, Peter S Park, Preni Golazizian, Ali Omrani, and Morteza Dehghani. 2024. Perils and opportunities in using large language models in psychological research. _PNAS Nexus_, 3(7):pgae245.
* Aher et al. (2023) Gati V Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In _Proceedings of the 40th International Conference on Machine Learning_, volume 202 of _Proceedings of Machine Learning Research_, pages 337-371. PMLR.
* Ai et al. (2024) Yiming Ai, Zhiwei He, Ziyin Zhang, Wenhong Zhu, Hongkun Hao, Kai Yu, Lingjun Chen, and Rui Wang. 2024. Is cognition and action consistent or not: Investigating large language model's personality. _Preprint_, arXiv:2402.14679.
* Apicella et al. (2020) Coren Apicella, Ara Norenzayan, and Joseph Henrich. 2020. Beyond weird: A review of the last decade and a look ahead to the global laboratory of the future. _Evolution and Human Behavior_, 41(5):319-329. Beyond Weird.
* Arditi et al. (2024) Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssey, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. _Preprint_, arXiv:2406.11717.
* Argyle et al. (2023) Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. _Political Analysis_, 31(3):337-351.
* Arnett (2008) Jeffrey Jensen Arnett. 2008. The neglected 95%: why american psychology needs to become less american. _The American psychologist_, 63 7:602-14.
* Barocas et al. (2021) Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, W. Duncan Wadsworth, and Hanna Wallach. 2021. Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_, AIES '21, page 368-378, New York, NY, USA. Association for Computing Machinery.
* Bernsky (2017) Adam Bernsky. 2017. Measuring public opinion with surveys. _Annual Review of Political Science_, 20.
* Bisbee et al. (2024) James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. 2024. Synthetic replacements for human survey data? the perils of large language models. _Political Analysis_, page 1-16.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems_, volume 33, pages 1877-1901. Curran Associates, Inc.
* Ceron et al. (2024) Tanise Ceron, Neele Falk, Ana Baric, Dmitry Nikolaev, and Sebastian Pado. 2024. Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in llms. _Preprint_, arXiv:2402.17649.
* Chen et al. (2024) Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Kormalkai Vinayak. 2024. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. _Preprint_, arXiv:2406.08469.
* Cheng et al. (2023a) Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023a. Marked personas: Using natural language prompts to measure stereotypes in language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1504-1532, Toronto, Canada. Association for Computational Linguistics.
* Cheng et al. (2023b) Myra Cheng, Tiziano Piccardi, and Diyi Yang. 2023b. CoMPosT: Characterizing and evaluating caricature in LLM simulations. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 10853-10875, Singapore. Association for Computational Linguistics.
* Christiano et al. (2017) Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_, NIPS'17, page 4302-4310, Red Hook, NY, USA. Curran Associates Inc.
* Chuang et al. (2024) Yun-Shiuan Chuang, Zach Studdiford, Kirkir Nirunwiroj, Agam Goyal, Vincent V. Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T. Rogers. 2024. Beyond demographics: Aligning role-playing llvm-based agents using human belief networks. _Preprint_, arXiv:2406.17232.
* Dillion et al. (2023) Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can ai language models replace human participants? _Trends in Cognitive Sciences_, 27(7):597-600.
* Dominguez-Olmedo et al. (2024) Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dunner. 2024. Questioning the survey responses of large language models. _Preprint_, arXiv:2306.07951.
* Dubois et al. (2023) Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishana Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Taksuori B Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. In _Advances in Neural Information Processing Systems_, volume 36, pages 30039-30069. Curran Associates, Inc.



Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zace Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamku1, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. Towards measuring the representation of subjective global opinions in language models. In _First Conference on Language Modeling_.
* Feng et al. (2024) Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. 2024. Modular pluralism: Pluralistic alignment via multi-llm collaboration. _Preprint_, arXiv:2406.15951.
* Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6894-6910.
* Geng et al. (2024) Mingmeng Geng, Sihong He, and Roberto Trotta. 2024. Are large language models chameleons? _Preprint_, arXiv:2405.19323.
* Grossmann et al. (2023) Igor Grossmann, Matthew Feinberg, Dawn C. Parker, Nicholas A. Christakis, Philip E. Tetlock, and William A. Cunningham. 2023. Ai and the transformation of social science research. _Science_, 380(6650):1108-1109.
* Volume 70_, ICML'17, page 1321-1330. JMLR.org.
* Gupta et al. (2024) Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. Bias runs deep: Implicit reasoning biases in persona-assigned LLMs. In _The Twelfth International Conference on Learning Representations_.
* He et al. (2024) Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. AnnoLLM: Making large language models to be better crowdsourced annotators. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track)_, pages 165-190, Mexico City, Mexico. Association for Computational Linguistics.
* Horton (2023) John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Working Paper 31122, National Bureau of Economic Research.
* Hu and Levy (2023) Jennifer Hu and Roger P. Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
* Hwang et al. (2023) EunJeong Hwang, Bodhisattwa Majumder, and Niket Tandon. 2023. Aligning language models to user opinions. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 5906-5919, Singapore. Association for Computational Linguistics.
* Jiang et al. (2022) Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022. CommunityLM: Probing partisan worldviews from language models. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 6818-6826, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schaefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kerrion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. _Preprint_, arXiv:2207.05221.
* Kambhatla et al. (2022) Gauri Kambhatla, Ian Stewart, and Rada Mihalcea. 2022. Surfacing racial stereotypes through identity portrayal. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_, FAccT '22, page 1604-1615, New York, NY, USA. Association for Computing Machinery.
* Kim and Yang (2024) Jaebhyung Kim and Yiming Yang. 2024. Few-shot personalization of llms with mis-aligned responses. _Preprint_, arXiv:2406.18678.
* Kirk et al. (2024) Hannah Rose Kirk, Alexander Whitefield, Paul Rotter, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. _Preprint_, arXiv:2404.16019.
* Van Koevering and Kleinberg (2024)Katherine Van Koevering and Jon Kleinberg. 2024. How random is random? evaluating the randomness and humans of lllms' coin flips. _Preprint_, arXiv:2406.00092.
* Koo et al. (2024) Ryan Koo, Minhua Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. Benchmarking cognitive biases in large language models as evaluators. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 517-545, Bangok, Thailand and virtual meeting. Association for Computational Linguistics.
* Kovac et al. (2023) Grgur Kovac, Masataka Sawayama, Remy Portelas, Cedric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023. Large language models as 

superpositions of cultural perspectives. _Preprint_, arXiv:2307.07870.
* Levendusky and Malhotra (2015) Matthew S. Levendusky and Neil Malhotra. 2015. (Mis)perceptions of Partisan Polarization in the American Public. _Public Opinion Quarterly_, 80(S1):378-391.
* Levin et al. (2006) David A. Levin, Yuval Peres, and Elizabeth L. Wilmer. 2006. _Markov chains and mixing times_. American Mathematical Society.
* Li et al. (2024a) Junyi Li, Charith Peris, Ninareh Mehrabi, Palash Goyal, Kai-Wei Chang, Aram Galstyan, Richard Zemel, and Rahul Gupta. 2024a. The steerability of large language models toward data-driven personas. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers)_, pages 7290-7305, Mexico City, Mexico. Association for Computational Linguistics.
* Li et al. (2024b) Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, and Percy Liang. 2024b. Benchmarking and improving generator-validator consistency of language models. In _The Twelfth International Conference on Learning Representations_.
* Liu et al. (2024) Andy Liu, Mona Diab, and Daniel Fried. 2024. Evaluating large language model biases in persona-steered generation. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 9832-9850, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Liu et al. (2023) Kevin Liu, Stephen Casper, Dylan Hadfield-Menell, and Jacob Andreas. 2023. Cognitive dissonance: Why dolanguage model outputs disagree with internal representations of truthfulness? In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 4791-4797, Singapore. Association for Computational Linguistics.
* Lyu et al. (2024) Chenyang Lyu, Minghao Wu, and Alham Aji. 2024. Beyond probabilities: Unveiling the misalignment in evaluating large language models. In _Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024)_, pages 109-131, Bangkok, Thailand. Association for Computational Linguistics.
* Masoud et al. (2024) Reem I. Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2024. Cultural alignment in large language models: An explanatory analysis based on hofstede's cultural dimensions. _Preprint_, arXiv:2309.12342.
* Mellon et al. (2024) Jonathan Mellon, Jack Bailey, Ralph Scott, James Breckwoldt, Marta Miori, and Phillip Schmedeman. 2024. Do ais know what the most important issue is? using language models to code open-text social survey responses at scale. _Research & Politics_, 11(1):20531680241231468.
* Mondal et al. (2024) Manuel Mondal, Ljiljana Dolamic, Gerome Bovet, Philippe Cudre-Mauroux, and Julien Audiffren. 2024. Do large language models exhibit cognitive dissonance? studying the difference between revealed beliefs and stated answers. _Preprint_, arXiv:2406.14986.
* Namikoshi et al. (2024) Keiichi Namikoshi, Alex Filipowicz, David A. Shamma, Rumen Iliev, Candice L. Hogan, and Nikos Arechiga. 2024. Using llm s to model the beliefs and preferences of targeted populations. _Preprint_, arXiv:2403.20252.
* Naous et al. (2024) Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 16366-16393, Bangkok, Thailand. Association for Computational Linguistics.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhnii Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_, volume 35, pages 27730-27744. Curran Associates, Inc.
* Park et al. (2023a) Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023a. Generative agents: Interactive simulacra of human behavior. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_, UIST '23, New York, NY, USA. Association for Computing Machinery.
* Park et al. (2023b) Peter S. Park, Philipp Schoeenegger, and Chongyang Zhu. 2023b. Diminished diversity-of-thought in a standard large language model. _Preprint_, arXiv:2302.07267.
* Paruchuri et al. (2024) Akshay Paruchuri, Jake Garrison, Shun Liao, John Hernandez, Jacob Sunshine, Tim Althoff, Xin Liu, and Daniel McDuff. 2024. What are the odds? language models are capable of probabilistic reasoning. _Preprint_, arXiv:2406.12830.
* Perez et al. (2024) Ethan Perez, Sam Ringer, Kamilie Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kerion, James Landis, Jamie Kerr, Jared Mueller, Jeeyeon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Laine Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Nexaro Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova Das Sarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Teleben-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda 

Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. 2023. Discovering language model behaviors with model-written evaluations. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 13387-13434, Toronto, Canada. Association for Computational Linguistics.
* Pistilli et al. (2024) Giada Pistilli, Alina Leidinger, Yacine Jernite, Atoosa Kasirzadeh, Alexandra Sasha Luccioni, and Margaret Mitchell. 2024. Civics: Building a dataset for examining culturally-informed values in large language models. _Preprint_, arXiv:2405.13974.
* Requeima et al. (2024) James Requeima, John Bronskill, Dani Choi, Richard E. Turner, and David Duvenaud. 2024. Lim processes: Numerical predictive distributions conditioned on natural language. _Preprint_, arXiv:2405.12856.
* Rottger et al. (2024) Paul Rottger, Valentin Hofmann, Valentin Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15295-15311, Bangkok, Thailand. Association for Computational Linguistics.
* Samuel et al. (2024) Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, and Vishvak Murahari. 2024. Personagym: Evaluating persona agents and llms. _Preprint_, arXiv:2407.18416.
* Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In _Proceedings of the 40th International Conference on Machine Learning_, volume 202 of _Proceedings of Machine Learning Research_, pages 29971-30004. PMLR.
* Si et al. (2024) Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. _Preprint_, arXiv:2409.04109.
* Simmons (2023) Gabriel Simmons. 2023. Moral mimicry: Large language models produce moral rationalizations tailored to political identity. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)_, pages 282-297, Toronto, Canada. Association for Computational Linguistics.
* Sorensen et al. (2024) Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. A roadmap to pluralistic alignment. _Preprint_, arXiv:2402.05070.
* Stammbach et al. (2024) Dominik Stammbach, Philine Widmer, Eunjung Cho, Caglar Gulcehre, and Elliott Ash. 2024. Aligning large language models with diverse political viewpoints. _CoRR_, abs/2406.14155.
* Times (2024) The New York Times. 2024. Top books of 2024. [https://www.nytimes.com/interactive/2024/books/top-books-list.html](https://www.nytimes.com/interactive/2024/books/top-books-list.html). Accessed: 2024-07-26.
* Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 5433-5442, Singapore. Association for Computational Linguistics.
* Turner et al. (2023) Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. 2023. Activation addition: Steering language models without optimization. _CoRR_, abs/2308.10248.
* Wan et al. (2023) Yixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng, and Kai-Wei Chang. 2023. Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems. In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
* Wang et al. (2024a) Angelina Wang, Jamie Morgenstern, and John P. Dickerson. 2024a. Large language models should not replace human participants because they can misportray and flatten identity groups. _Preprint_, arXiv:2402.01908.
* Wang et al. (2024b) Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhoapeng Tu, and Michael Lyu. 2024b. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6349-6384, Bangkok, Thailand. Association for Computational Linguistics.
* Wang et al. (2022d) Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Rottger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024c. "my answer is C": First-token probabilities do not match text answers in instruction-tuned language models. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 7407-7416, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishrizi. 2023. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
* West et al. (2024) Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khysith Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. 2024. The generative AI paradox: "what it can create, it may not understand". In _The Twelfth International Conference on Learning Representations_ 

* Yan (2021) Ting Yan. 2021. Consequences of asking sensitive questions in surveys. _Annual Review of Statistics and Its Application_, 8(Volume 8, 2021):109-127.
* Yudkin et al. (2019) Daniel A Yudkin, Stephen Hawkins, and Tim Dixon. 2019. The perception gap: How false impressions are pulling americans apart.
* Zhao et al. (2023) Siyan Zhao, John Dang, and Aditya Grover. 2023. Group preference optimization: Few-shot alignment of large language models. In _R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models_.
* Zhao et al. (2024) Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, and Yuling Gu. 2024. World-ValuesBench: A large-scale benchmark dataset for multi-cultural value awareness of language models. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 17696-17706, Torino, Italia. ELRA and ICCL.
* Zhou et al. (2024) Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Conatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. 2024. SOTOPIA: Interactive evaluation for social intelligence in language agents. In _The Twelfth International Conference on Learning Representations_.
* Zhu et al. (2024) Jian-Qiao Zhu, Haijiang Yan, and Thomas L. Griffiths. 2024. Recovering mental representations from large language models with markov chain monte carlo. _Preprint_, arXiv:2401.16657.
* Ziems et al. (2024) Caleb Ziems, William Held, Omar Shaikh, Jiao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? _Computational Linguistics_, 50(1):237-291.

 

Appendix

### Significance of Distributional Alignment

LLM outputs that align with human preferences are becoming increasingly popular (e.g., RLHF). Typically, when researchers evaluate which group a model best aligns with, they compare average responses. However, this only measures alignment to the preference of the majority group. In many subjective tasks, it's crucial for a model to represent the entire spectrum of opinions.

For example, consider the following survey question in the PEW (question ID: GODMORALIMP_W92).

**Question**

Regardless of your own religious beliefs, how important, if at all, do you think it is for a person to believe in God in order to be considered good and moral?

Important.

Not Important.

The PEW survey respondents responded with \(\{A:54.8\%,B:45.2\%\}\). If we query the language model and receive a response of "A. Important" (the highest likelihood answer choice), the model may appear to be highly aligned if our evaluation only considers the most likely human response. However, this ignores the 45% of people responding with the minority choice, "B. Not Important." A clearer assessment of this task would be if the language model can represent the 54.8%, and 45.2% split, as this would show the model understands the underlying heterogeneity of views on this topic. Evaluating in this way provides a clearer and more precise measure of the model's alignment with human preferences.

### Temperature Scaling

Temperature scaling is a post-processing technique to make neural networks calibrated Guo et al. (2017). To implement temperature scaling for our work, we find the minimum temperature value that results in the smallest total variation between the reference probabilities and model probabilities. In this setting, the temperature-scaled scaled log-probabilities have access to the ground truth opinion distribution and aim to find \(\tau\) that minimizes the total variation. We calculate a new value of \(\tau\) per dataset and steering method using the following formula.

\[\min_{\tau}\frac{1}{2}\left\|y_{g,q}-\hat{y}^{\frac{1}{2}}_{g,q,\text{norm}} \right\|_{1}. \tag{3}\]

Our model log-probabilities are sampled with a temperature of \(1.0\), and after applying temperature scaling to the biased coin flip, the model log-probabilities performance improves. When extended to opinion surveys, we see much fewer improvements in the temperature-scaled model log-probabilities for Llama-3-70B Instruct, but improvements for GPT-3.5 and GPT-4. We report the distributional alignment performance in Tab. 0(a), plot calibration curves in Fig. 6, and report ECE in Tab. 3).

### GlobalOpinionQA

In this section, we share additional question examples, preprocessing steps, and results for GlobalOpinionQA. For example, consider this survey question that demonstrates how countries can highly differ in their distributional responses to these survey questions.

**Question**

Do you personally believe that drinking alcohol is morally acceptable, morally unacceptable, or is it not a moral issue?

Morally acceptable

Morally unacceptable

Not a moral issue

Depends on the situation

## Appendix B

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**Model** & **TS-Log-p** & **Log-p** \\ \hline Llama3-70B & 0.11 & 0.13 \\ \hline GPT-4 & 0.07 & 0.28 \\ \hline GPT-3.5 & 0.06 & 0.20 \\ \hline \end{tabular}
\end{table}
Table 3: ECE (Expected Calibration Error) values for temperature scaled log probabilities (TS-Log-p) and log-probabilities (LogP-p) Llama 3 70B, GPT-4, and GPT-3.5.

 



**Preprocessing.** To filter out the top 100 contentious questions, we start by only considering survey responses from countries that have at least 600 other responses to ensure a large enough pool of questions to pull few shot examples from, reducing the total number of countries in the dataset from 138 to 19. Then we calculate disagreement by the largest total variation distance.

**Results.** The results are consistent with those in our paper: (1) verbalize remains the optimal distribution estimation method (Tab. 4) and (2) there exists a knowledge-to-simulation gap between verbalize and sequence in all 5 models (Tab. 5).

\begin{table}
\begin{tabular}{l l} \hline \hline Model & \(\mathcal{A}(Y,\hat{Y}_{\mathcal{S},\mathcal{O}})\) \\ \hline Anthropic Opus (V) & \(0.241\pm 0.015\) \\ GPT-4 (V) & \(0.279\pm 0.016\) \\ Llama 3 70B (V) & \(0.281\pm 0.016\) \\ Anthropie Haiku (V) & \(0.293\pm 0.016\) \\ Anthropic Opus (Seq) & \(0.301\pm 0.020\) \\ GPT-4 (TS-Log-p) & \(0.303\pm 0.016\) \\ Llama 3 70B (Seq) & \(0.346\pm 0.020\) \\ Anthropic Haiku (Seq) & \(0.351\pm 0.016\) \\ GPT-3.5 (V) & \(0.356\pm 0.019\) \\ GPT-4 (Seq) & \(0.359\pm 0.021\) \\ GPT-3.5 (TS-Log-p) & \(0.381\pm 0.015\) \\ GPT-3.5 (Seq) & \(0.399\pm 0.018\) \\ GPT-3.5 (LogP-p) & \(0.442\pm 0.022\) \\ Llama 3 70B (Log-p) & \(0.455\pm 0.021\) \\ GPT-4 (Log-p) & \(0.484\pm 0.023\) \\ Llama 3 70B (TS-Log-p) & \(0.491\pm 0.024\) \\ \hline Discretization Error (Seq) & \(0.092\pm 0.004\) \\ Uniform & \(0.486\pm 0.019\) \\ Majority Vote & \(0.692\pm 0.035\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: **Distributional Alignment Task on GlobalOpinionQA**. Models ranked based on mean total variation. Models highlighted in gray and with (V) have a distribution expression method of directly verbalizing the distribution in a JSON format (\(\mathcal{O}=\texttt{Verbalize}\)). Models not highlighted represent samplers, where (Seq) represents the 30-token sequential distribution output (\(\mathcal{O}=\texttt{Sequence}\)), (Log-p) represents \(\mathcal{O}=\texttt{Model Log-probabilities}\), and (TS-Log-p) represents \(\mathcal{O}=\texttt{Temperature Scaled Model Log-probabilities}\).

Figure 6: **Temperature Scaling calibration curves for the survey setting**. We find that temperature scaling improves results for GPT models, but not for Llama3–70b. Results are averaged across the OpinionQA, NYTimes, and GlobalOpinionQA Datasets.





### NYT Books Dataset Construction

**Annotators.** From this data collection process, we surveyed 131 Male, 206 Female, 165 Democrat, and 172 Republican annotators, resulting in 18 annotations per book per demographic group. In Fig 7, we show an example annotation question. In this example, annotators are given a book title and its corresponding author, book summary, and genre. Then they provide a 4-point Likert rating to the question, "Given the summary of this book, how likely are you to read it?"

From these annotations, we calculated an opinion distribution over the 4 Likert ratings for each book. All crowd workers are sourced on Prolific, filtered for English fluency, selected from a pool of annotators who pass an attention check 93% of the time and are paid $12 per hour (this amount is well over the federal minimum wage of $7.25). The consent form shown to annotators is shown in Fig 8.

It should be noted that articles from the New York Times are in English and protected under copyright, but this research is performed in the public interest under GDPR and the excerpts are collected under fair use exemption. When releasing our dataset, we include all links to the exact New York Times paragraph highlight to respect copyrights. Furthermore, the annotation data we collect is not personally identifiable as it consists only of opinion ratings of books.

In addition to the disagreement analysis we conduct in Sec. 3.3, we calculate the Cohen's kappa of opinions _between_ demographic groups. We find that Cohen's kappa between Democrats and Republicans is \(0.05\), indicating little to no agreement. The Cohen's kappa between Men and Women is \(0.15\), indicating a small amount of agreement.

### OpinionQA

We used the OpinionQA Santurkar et al. (2023) accessed via the CC-BY 4.0 license, written in English, as they survey US participants. In their steerability analysis, they create a smaller set of 500 contentious questions where the subgroups frequently disagree. We follow suit and randomly sample 100 questions from this set to obtain questions spanning topics such as science, politics, and personal relationships.

### Human Baseline Annotations

Using the Prolific platform, we crowdsource human annotations for the distributional alignment task. We restrict the demographics of annotators to match the groups that we study as it provided us an opportunity to do an in-group vs out-group analysis. Our OQA annotators are Democrat (73%), Republican (27%), Male (33%), Female (67%),

Figure 8: **NYT Books Annotator Consent Form.**

\begin{table}
\begin{tabular}{l c} \hline \hline Model & Percent Error (\%) \\ \hline GPT-3.5 Turbo & 12.15\% \\ Anthropic: Haiku & 19.86\% \\ Llama 3 70B Instruct & 22.95\% \\ Anthropic: Opus & 24.88\% \\ GPT-4 & 28.68\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: **Knowledge-to-Simulation Gap of GlobalOpinionQA (Eq. 2). The simulation penalty measures the percent error increase in total variation between the 30-token sequential distribution output and the verbalization of knowledge.**

Figure 7: **NYT Books Annotation Example Task.**

the "descriptions". The White (84%), and Black (16%). For the NYT annotations, our annotators are Democrat (69%), Republican (31%), Male (37%), and Female (63%). Each question from the OQA and NYT datasets is annotated four times. Across the persona and few shot steering, we have 246 annotators for OQA and 374 annotators for NYT. We compensate the workers at a rate of $12 per hour. The consent form shown to annotators is similar to that shown for the NYT Books annotation collection (Fig. 8)--the only difference is the "description".

Next, we show the instructions given to participants and a demo of how the distributional alignment task is completed. In Fig. 9, we show the instructions for no steering. In this example, annotators are instructed to estimate the distribution of multiple choice responses from Americans as a whole.

In Fig. 10, we show the instructions for persona steering. In this specific example, annotators are instructed to simulate a Democrat, but this group can be any demographic group of interest. Finally, in Fig. 11, we show the instructions and an example annotation for few shot steering. In this specific example, humans are tasked with simulating the views of Gen Z and given examples of how this group has responded to similar questions on driverless vehicles (participants are given 5 examples, but only 3 are shown in this figure).

**Quality control.** To ensure annotation quality, we limit the task to workers with English fluency and ask survey participants to answer a reading attention check question which our annotators achieved 93% on.

Prior work has found that in-group representations and out-group imitations from human participants are different and that there exists a misperception of partisan polarization [23, 1]. Thus, we compared in-group and out-group representations in opinion distribution estimation. In this context, "in-group" refers to Democrats simulating Democrats, and "out-group" refers to Democrats simulating Republicans. "In-group" can also refer to a group of men simulating other men and "out-group" refers to a group of men simulating a group of women. While there is more difference on average over all subgroups in persona steering between the in-group and out-group, these differences are not statistically significant (Tab. 6).

### Model Log Probabilities of each Token in the Sequence of Biased Coin Flips

In this section, we provide additional analysis on the Biased Coin Flip experiment, specifically regarding the sequence of biased coin flips. We tested the OpenAI gpt-4 model, access on March 2024, which points to gpt-4-0613, a snapshot of gpt-4 from June 13th 2023. In Fig. 12, we plot the model log probabilities of each token in the sequence to better understand the conditional probabilities and find that these probabilities are not

Figure 10: **Distributional Alignment Instructions and Example Task: Persona Steering.**

Figure 9: **Distributional Alignment Instructions and Example Task: No Steering.**



consistent across each token in the sequence (e.g., the next-token probability for 'H' varies based on the position in the sequence).

### Few Shot Steering

To perform few shot steering, for each question, we calculate the top 10 most similar questions or books as identified from their text embeddings Gao et al. (2021). We found that some of these questions may be near-duplicates, especially in opinionQA where a survey may include many variants of the same question. Thus, we filter for the hardest similar examples, namely the top 5 questions most distinct in output distribution. This ensures we provide topically coherent and similar examples while avoiding cases where the model simply copies the distribution from the few shot examples. We pass these 5 examples in as contextual distributional information to aid in the distribution estimation.

### Additional Model Information

In this section, we discuss each model and how it was accessed. All GPT models and their logprobabilities were accessed using the OpenAI API, namely GPT-3.5-Turbo-0125 and gpt-4-0613. Both Anthropic Haiku and Anthropic Opus were accessed using the Anthropic API which does not provide log-probabilities to users. Finally, Meta's

\begin{table}
\begin{tabular}{l l c r} Dataset & Steering & In/Out & TV \\ \hline OQA & Persona & In & 0.297 \(\pm\) 0.013 \\ OQA & Persona & Out & 0.322 \(\pm\) 0.014 \\ \hline NYT & Persona & In & 0.281 \(\pm\) 0.010 \\ NYT & Persona & Out & 0.273 \(\pm\) 0.010 \\ \hline OQA & Few Shot & In & 0.283 \(\pm\) 0.014 \\ OQA & Few Shot & Out & 0.278 \(\pm\) 0.013 \\ \hline NYT & Few Shot & In & 0.236 \(\pm\) 0.008 \\ NYT & Few Shot & Out & 0.237 \(\pm\) 0.009 \\ \end{tabular}
\end{table}
Table 6: In-group vs. Out-group performance in human annotators, averaged over all demographic groups.

Figure 11: **Distributional Alignment Instructions and Example Task: Few Shot Steering.**

Figure 12: **Model Probability of Each Token in 30 Flip Sequence (Heads).**



Meta-Llama-3-70B-Instruct was accessed via Huggingface. Since most models were accessed via API, the GPU hours accrewed come from running inference on Meta's Llama-3-70B-Instruct--this amounted to less than 1 hour.

While we tested additional models listed in Tab. 7, we found that they struggled to follow the distribution estimation method prompt, particularly for that of sequence and verbalize.

### Persona-steered LLMs _and humans_ are susceptible to stereotyping.

We supplement Fig. 5 which depicts the marginal distribution of the answer ratings in the NYT Books dataset with average human ratings (Fig. 13). While previously, we demonstrated that models prompted with a Democrat persona tend to simulate humans that are more likely to read books, we see that humans have a similar stereotypical effect, although smaller than that of models.

### Additional Distribution Estimation Method Related Works

In this section, we engage more substantially with prior work relating to the distribution estimation method (Sec. 3.1). Prior work has found that language models can either be queried for statement probabilities (e.g., verbalize or other prompt-based techniques) or probed for internal representations of truthfulness (i.e., model log-probabilities). Past work has found that these two representations can sometimes disagree Hu Levy (2023); Mondal et al. (2024); Liu et al. (2023). Earlier work by Hu and Levy (2023) finds that direct probability measurements generally field better or similar performance when compared to prompting techniques on tasks such as word prediction, semantic plausibility, and syntax completion in a zero-shot setting. Our work differs in that our tasks involve scenarios with uncertainty, we evaluate RLHF-edal LMs, and we provide few shot examples of the task. This is valuable as LMs with RLHF have been shown to produce conditional probabilities that are poorly calibrated Kadavath et al. (2022).

Furthermore, the estimated distribution of the model's opinion has been shown to be sensitive to other factors including rewording Bisbee et al. (2024) or negating questions Ceron et al. (2024), shuffling the answer choice order Dominguez-Olmedo et al. (2024), and when converting multiple-choice surveys to free text Rottger et al. (2024); Lyu et al. (2024); Wang et al. (2024c).

### Distributional Distance Metric: Total Variation

Equ. 1 represents the total variation _distance_, a measure for comparing the distance between two probability distributions (see Prop 4.2 in Levin et al. (2006)). While KL divergence can be used to compare distributions' differences, we selected total variation distance for its interpretability and relevance to our setting. One notable reason why KL divergence is unsuitable for comparing survey responses from people with those from the model is that KL divergence becomes infinite when one distribution has zero probability events while the other does not. This naturally occurs in survey samples when no respondents select a particular answer choice, or in our distribution estimation methods, sequence and verbalize, which can generate zero estimated probabilities for certain answer choices.

Consider two probability distributions \(P\) and \(Q\) over the discrete space of multiple-choice options \(\{A,B,C\}\), where

\[P(A)=0.6,\quad P(B)=0.35,\quad P(C)=0.05\] \[Q(A)=0.6,\quad Q(B)=0.40,\quad Q(C)=0.\]

\[D_{KL}(P||Q)=\sum_{i=1}^{3}P(i)\log\left(\frac{P(i)}{Q(i)}\right)\]

The final term (\(i=3\)) contains \(0.05\log\big{(}\frac{0.05}{0}\big{)}\) which goes to infinity (\(\log(0)=-\infty\)).

\[\text{TV}(P,Q)=\frac{1}{2}\sum_{i=1}^{3}||P(i)-Q(i)||_{1}=0.05.\]

\begin{table}
\begin{tabular}{l l l} \hline Model & Seq & Verbalize \\ \hline Llama-3-8B & 45\% & 40\% \\ Llama-2-70B & 3\% & 97\% \\ Llama-2-13B & 11\% & 83\% \\ Llama-2-7B & 10\% & 52\% \\ Deepseek-coder-1.3B & 0\% & 2\% \\ Deepseek-coder-6.7B & 0\% & 0\% \\ \hline \end{tabular}
\end{table}
Table 7: **Open models struggle to follow the prompt for the distribution expression methods, sequence, and verbalize. In this table, we report the success rate that additional open models had in following the prompt instructions. There was limited success with these smaller models; thus, we opted for the 5 models included in the main paper.** 

Here total variation distance is a small value, reflecting that these distributions are very similar.

### Full Results of Each Steering Method

In Tab. 10 we report the results of each steering method (No Steering, Persona Steering, and Few Shot Steering) for OpinionQA and NYTBooks. We also separate the results of persona steering and few shot steering in the GlobalOpinionQA dataset and report them in Tab. 8 and Tab. 9, respectively. When comparing the persona steered and few shot steered results, we see considerable improvement with few shot steering, corroborating our implications regarding the implications of persona steering and using few shot examples when possible (Sec. 4.2, Sec. 5).

[TABLE:A3.T8][E

Figure 13: **Persona-steered LLMs and humans produce stereotypical results.** We plot the marginal distribution for each Likert Rating (1-4) corresponding to the answer to the following question: “How likely are you to read this book?” A Likert rating of 1 refers to “Very unlikely” and a Likert rating of 4 refers to “Very likely”. We report the average marginal distribution over 5 models and humans steered towards Democrats and Republicans with persona steering (orange) and few shot steering (green). In blue, we plot the reference human reference for Democrat and Republican annotators. All of the data is averaged over 235 books in the NYT Books dataset. In purple we plot the persona steered humans, and in pink we plot the few shot steered humans. We observe that persona steered humans see a similar stereotypical effect, although smaller.

 



\begin{table}
\begin{tabular}{l l} \hline \hline Model & \(\mathcal{A}(Y,\hat{Y}_{\mathcal{S},\mathcal{O}})\) \\ \hline Anthropic Opus (V) & 0.208 \(\pm\) 0.020 \\ Llama 3 70B (V) & 0.234 \(\pm\) 0.022 \\ GPT-4 (V) & 0.237 \(\pm\) 0.023 \\ Anthropic Haiku (V) & 0.252 \(\pm\) 0.023 \\ Anthropic Opus (Seq) & 0.265 \(\pm\) 0.025 \\ GPT-4 (TS-Log-p) & 0.280 \(\pm\) 0.024 \\ Llama 3 70B (Seq) & 0.312 \(\pm\) 0.026 \\ GPT-3.5 **(V)** & 0.314 \(\pm\) 0.026 \\ Anthropic Haiku (Seq) & 0.327 \(\pm\) 0.024 \\ GPT-3.5 (TS-Log-p) & 0.334 \(\pm\) 0.023 \\ GPT-4 (Seq) & 0.341 \(\pm\) 0.032 \\ GPT-3.5 (Seq) & 0.366 \(\pm\) 0.027 \\ GPT-3.5 (LogP-p) & 0.440 \(\pm\) 0.032 \\ Llama 3 70B (Log-p) & 0.449 \(\pm\) 0.031 \\ GPT-4 (Log-p) & 0.461 \(\pm\) 0.033 \\ Llama 3 70B (TS-Log-p) & 0.437 \(\pm\) 0.031 \\ \hline Discretization Error (Seq) & 0.092 \(\pm\) 0.004 \\ Uniform & 0.486 \(\pm\) 0.019 \\ \hline \hline \end{tabular}
\end{table}
Table 9: **Distributional Alignment Task on GlobalOptimionQA: Few Shot Steering**. Models ranked based on mean total variation. Models highlighted in gray and with (V) have a distribution expression method of directly verbalizing the distribution in a JSON format (\(\mathcal{O}=\texttt{verbalize}\)). Models not highlighted represent samplers, where (Seq) represents the 30-token sequential distribution output (\(\mathcal{O}=\texttt{Sequence}\)), (Log-p) represents \(\mathcal{O}=\texttt{Model Log-probabilities}\), and (TS-Logp) represents \(\mathcal{O}=\texttt{Temperature Scaled Model Log-probabilities}\).





\begin{tabular}{|l l c|l c|} \hline  & \multicolumn{2}{c|}{**OQA**} & \multicolumn{3}{c|}{**NYT**} \\ \cline{2-5}
**Model** & \(\mathcal{A}(Y,\dot{Y}_{\mathcal{S},\mathcal{O}})\) & **Model** & \(\mathcal{A}(Y,\ddot{Y}_{\mathcal{S},\mathcal{O}})\) \\ \hline \multirow{9}{*}{**CQAR**} & Anthropic: Haiku (Seq) & 0.372 \(\pm\) 0.013 & Anthropic Opus (V) & 0.210 \(\pm\) 0.006 \\  & GPT-4 (Seq) & 0.389 \(\pm\) 0.015 & GPT-3.5 (Seq) & 0.215 \(\pm\) 0.006 \\ \cline{2-5}  & GPT-3.5 (V) & 0.394 \(\pm\) 0.014 & GPT-3.5 (V) & 0.219 \(\pm\) 0.006 \\  & Llama 3 70B (V) & 0.396 \(\pm\) 0.013 & GPT-4 (Seq) & 0.220 \(\pm\) 0.006 \\ \cline{2-5}  & GPT-3.5 (Seq) & 0.398 \(\pm\) 0.013 & Anthropic Haiku (Seq) & 0.226 \(\pm\) 0.007 \\  & GPT-3.5 (TS-Log-p) & 0.398 \(\pm\) 0.013 & GPT-4 (V) & 0.230 \(\pm\) 0.006 \\  & GPT-4 (V) & 0.400 \(\pm\) 0.015 & Llama 3 70B (V) & 0.237 \(\pm\) 0.007 \\  & Anthropic: Haiku (V) & 0.427 \(\pm\) 0.015 & Llama 3 70B (Seq) & 0.257 \(\pm\) 0.009 \\  & GPT-4 (TS-Log-p) & 0.451 \(\pm\) 0.015 & Anthropic Haiku (V) & 0.265 \(\pm\) 0.007 \\  & Antiropic: Obus (V) & 0.452 \(\pm\) 0.014 & GPT-3.5 (TS-Log-p) & 0.303 \(\pm\) 0.007 \\  & Anthropic: Opus (Seq) & 0.485 \(\pm\) 0.019 & Anthropic Opus (Seq) & 0.316 \(\pm\) 0.011 \\  & Llama 3 70B (Seq) & 0.501 \(\pm\) 0.020 & GPT-4 (TS-Log-p) & 0.323 \(\pm\) 0.008 \\  & GPT-3.5 (Log-p) & 0.540 \(\pm\) 0.017 & Llama 3 70B (TS-Log-p) & 0.606 \(\pm\) 0.011 \\  & Llama 3 70B (TS-Log-p) & 0.661 \(\pm\) 0.019 & Llama 3 70B (Log-p) & 0.638 \(\pm\) 0.010 \\  & Llama 3 70B (LogP-p) & 0.688 \(\pm\) 0.018 & GPT-4 (Log-p) & 0.689 \(\pm\) 0.010 \\  & GPT-4 (Log-p) & 0.714 \(\pm\) 0.018 & GPT-3.5 (Log-p) & 0.745 \(\pm\) 0.009 \\ \hline \multicolumn{5}{|c|}{**NOT**} \\ \cline{2-5}  & GPT-4 (V) & 0.181 \(\pm\) 0.007 & GPT-3.5 (Seq) & 0.226 \(\pm\) 0.007 \\  & Anthropic: Haiku (V) & 0.222 \(\pm\) 0.011 & GPT-3.5 (V) & 0.239 \(\pm\) 0.007 \\  & GPT-4 (TS-Log-p) & 0.224 \(\pm\) 0.012 & GPT-3.5 (TS-Log-p) & 0.295 \(\pm\) 0.008 \\  & Llama 3 70B (V) & 0.226 \(\pm\) 0.009 & GPT-4 (V) & 0.248 \(\pm\) 0.007 \\  & Anthropic: Opus (V) & 0.238 \(\pm\) 0.012 & Llama 3 70B (V) & 0.257 \(\pm\) 0.008 \\  & GPT-4 (Seq) & 0.238 \(\pm\) 0.009 & GPT-4 (Seq) & 0.264 \(\pm\) 0.009 \\  & Anthropic: Opus (Seq) & 0.282 \(\pm\) 0.012 & Anthropic Opus (V) & 0.285 \(\pm\) 0.010 \\  & Anthropic: Haiku (Seq) & 0.284 \(\pm\) 0.012 & Anthropic Haiku (V) & 0.297 \(\pm\) 0.010 \\  & GPT-3.5 Turbo (V) & 0.292 \(\pm\) 0.012 & GPT-4 (TS-Log-p) & 0.330 \(\pm\) 0.010 \\  & GPT-3.5 Turbo (TS-Log-p) & 0.298 \(\pm\) 0.009 & Anthropic Haiku (Seq) & 0.355 \(\pm\) 0.013 \\  & GPT-3.5 Turbo (Seq) & 0.317 \(\pm\) 0.012 & Llama 3 70B (Seq) & 0.359 \(\pm\) 0.011 \\  & Llama 3 70B (Seq) & 0.316 \(\pm\) 0.013 & Llama 3 70B (TS-Log-p) & 0.430 \(\pm\) 0.012 \\  & GPT-3.5 Turbo (Log-p) & 0.339 \(\pm\) 0.013 & GPT-3.5 (Log-p) & 0.473 \(\pm\) 0.012 \\  & Llama 3 70B (TS-Log-p) & 0.417 \(\pm\) 0.014 & Anthropic Opus (Seq) & 0.483 \(\pm\) 0.010 \\  & Llama 3 70B (Log-p) & 0.460 \(\pm\) 0.014 & Llama 3 70B (Log-p) & 0.525 \(\pm\) 0.011 \\  & GPT-4 (Log-p) & 0.507 \(\pm\) 0.015 & GPT-4 (Log-p) & 0.682 \(\pm\) 0.010 \\ \hline \multicolumn{5}{|c|}{**OQA**} & \multicolumn{3}{c|}{**NYT**} \\ \cline{2-5}  & Anthropic Opus (V) & 0.146 \(\pm\) 0.007 & Anthropic Opus (V) & 0.207 \(\pm\) 0.006 \\  & GPT-4 (V) & 0.179 \(\pm\) 0.008 & Llama 3 70B (V) & 0.208 \(\pm\) 0.006 \\  & GPT-4 (T-Log-p) & 0.200 \(\pm\) 0.007 & GPT-4 (V) & 0.208 \(\pm\) 0.006 \\  & Anthropic Haiku (V) & 0.200 \(\pm\) 0.010 & GPT-3.5 (TS-Log-p) & 0.209 \(\pm\) 0.005 \\  & GPT-4 (Seq) & 0.204 \(\pm\) 0.009 & Anthropic Haiku (V) & 0.221 \(\pm\) 0.005 \\  & Llama 3 70B (V) & 0.211 \(\pm\) 0.011 & GPT-3.5 (V) & 0.226 \(\pm\) 0.005 \\  & Anthropic Opus (Seq) & 0.248 \(\pm\) 0.010 & GPT-3.5 (Seq) & 0.228 \(\pm\) 0.005 \\  & Anthropic Haiku (Seq) & 0.262 \(\pm\) 0.012 & GPT-4 (Seq) & 0.243 \(\pm\) 0.006 \\  & GPT-3.5 (TS-Log-p) & 0.274 \(\pm\) 0.011 & Anthropic Haiku (Seq) & 0.248 \(\pm\) 0.007 \\  & GPT-3.5 (V) & 0.280 \(\pm\) 0.013 & GPT-4 (TS-Log-p) & 0.285 \(\pm\) 0.008 \\  & Llama 3 70B (Seq) & 0.300 \(\pm\) 0.012 & Llama 3 70B (Seq) & 0.305 \(\pm\) 0.008 \\  & GPT-3.5 (Seq) & 0.339 \(\pm\) 0.015 & Anthropic Opus (Seq) & 0.334 \(\pm\) 0.009 \\  & Llama 3 70B (TS-Log-p) & 0.394 \(\pm\) 0.015 & GPT-3.5 (Log-p) & 0.517 \(\pm\) 0.010 \\  & Llama 3 70B (Log-p) & 0.444 \(\pm\) 0.014 & Llama 3 70B (TS-Log-p) & 0.593 \(\pm\) 0.013 \\  & GPT-4 (Log-p) & 0.491 \(\pm\) 0.01



\begin{table}
\begin{tabular}{|l l c|l c|}  & GPT-3.5 (Log-p) & 0.519 \(\pm\) 0.016 & GPT-4 (Log-p) & 0.650 \(\pm\) 0.010 \\ \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c|}{**OQA**} & \multicolumn{2}{c|}{**NYT**} \\  & **Model** & \(\mathcal{A}(Y,\hat{Y}_{S,\mathcal{O}})\) & **Model** & \(\mathcal{A}(Y,\hat{Y}_{S,\mathcal{O}})\) \\ \cline{2-5}  & Discretization Error (Seq) & 0.136 \(\pm\) 0.012 & Discretization Error (Seq) & 0.115 \(\pm\) 0.001 \\ Uniform & 0.381 \(\pm\) 0.009 & Uniform & 0.223 \(\pm\) 0.006 \\ Majority Vote & 0.731 \(\pm\) 0.017 & Majority Vote & 0.700 \(\pm\) 0.008 \\ \end{tabular}
\end{table}
Table 10: **Comparing model performance across OQA and NYT datasets and steering methods. Models are ranked based on mean total variation and highlighted in gray and with (V) have a distribution expression method of directly verbalizing the distribution in a JSON format (\(\mathcal{O}=\texttt{Verbalize}\)). Models not highlighted represent samplers, where (Seq) represents the 30-token sequential distribution output (\(\mathcal{O}=\texttt{Sequence}\)), (Log-p) represents \(\mathcal{O}=\texttt{Model Log-probabilities}\), and (TS-LogP-p) represents \(\mathcal{O}=\texttt{Temperature Scaled Model Log-probabilities}\).**