Questioning the Survey Responses of Large Language Models
Authors: Ricardo Dominguez-Olmedo
Moritz Hardt
Celestine Mendler-Dünner
Venue: NeurIPS 2024 oral (2024)
Keywords: large language models
surveys
Primary area: natural_language_processing

Abstract: Surveys have recently gained popularity as a tool to study large language models. By comparing models’ survey responses to those of different human reference populations, researchers aim to infer the demographics, political opinions, or values best represented by current language models. In this work, we critically examine language models' survey responses on the basis of the well-established American Community Survey by the U.S. Census Bureau. Evaluating 43 different language models using de-facto standard prompting methodologies, we establish two dominant patterns. First, models' responses are governed by ordering and labeling biases, for example, towards survey responses labeled with the letter “A”. Second, when adjusting for these systematic biases through randomized answer ordering, models across the board trend towards uniformly random survey responses, irrespective of model size or training data. As a result, models consistently appear to better represent subgroups whose aggregate statistics are closest to uniform for the survey under consideration, leading to potentially misguided conclusions about model alignment.

Questioning the Survey Responses of Large Language Models
Ricardo Dominguez-Olmedo 1.2
Moritz Hardt
Celestine Sendler-Dünnerlaaee 1,2,3
Max-Planck Institute for Intelligent Systems, Tübingen
Tübingee Al Center
$ELLIS Institute Tübingen
{rdo, hardt,, cwendler (tteennerrrttuubnnnen......
Abstract
Surveys have recently gained popularity as tool to study large language models. By comparing survey responses models to those of human reference popula- tions, researchers aim to infer the demographics, political opinions, values best represented by current language models. In this work, we critically examine this methodology on the basis of the well-established American Community Survey by the U.S. Census Bureau. Evaluating 43 different language models using de-facto standard prompting methodologies, models' responses are governed by ordering and labeling biases, for example, towards survey responses labeled with the letter 'A'. Second, when adjusting for these systematic biases through randomized answer ordering, models across the board trend towards uniformly random survey responses, irrespective of model size or pre-training data. As result, in contrast to conjectures from prior work, survey-derived alignment measures often permit a simple explanation: models consistently appear to better represent subgroups whose aggregate statistics are closest to uniform for any survey under consideration.
establish
dominant patterns. First,
1 Introduction
Surveys have long tradition in social science research means for gathering statistical information about the characteristics, values, and opinions of human populations [Groves et al., 2009]. Insights from surveys inform policy interventions, business decisions, and science across various domains. Surveys typically consist of a series of well-curated questions in multiple-choice format, with unambiguous framing and set of answer choices carefully selected by domain experts. Surveys then presented to groups of individuals and their answers are aggregated to gain statistical insights about the populations that these groups of individuals represent.
Many established survey questionnaires together with the carefully collected answer statistics are publicly available. Machine learning researchers have identified the potential benefits of building on this valuable data resource to study large language models (LLMMs.. Survey questions offer way systematically prompt LLMs, and the aggregate statisties human populations serve as a reference point for evaluation. As a result, the use of surveys has recently gained popularity for studying LLMs' biases (Santurkar al. 2023 Durmus al. 2023 Also prompting LLMs with survey questions, researchers in the social sciences have explored using LLMs emulate the survey responses of human populations [Argyle al. 2023 Lee al. 2023 If effective proxies, simulated responses could augment or replace the expensive data collection process involving human subjects and provide insights into subpopulations that are otherwise hard reach.
answers collected by surveying
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

American population
Aggregate responses
American Community
28
the PAST 12 MONTHS, has givan birth to any chillnee??
Relatiue
Frequency
1.0
?
Yea
0.0
Survey (ACS)
Large Language Model
Output probabilities
DE 000
Questton:: the past 12 months, has this person given birth any childen?
Narmalized Probabiiity
1.0
response distributions comparable?
es
A.
Yes
eo
B. No
Answer:
Figure 1: We prompt language models with questions from the American Community Survey (ACS). We systematically compare models' survey responses to those of the U.S. Census.
It is tempting to prompt LLMs with survey questions, due to their syntactic similarity to question answering tasks [Brown et al. 2020 Liang et al., (022). However, a priori unclear how to interpret their answers. Rather than knowledge testing, surveys seek to elicit aggregate statistics over individuals, providing an unbiased view on the properties of the population they are targeting. The quality of survey data hinges the validity and robustness of the conclusions that can be drawn from it. Clearly, running a survey on LLMs is different from interrogating humans and thus it comes with distinct challenges. While much research has gone into carefully designing surveys to ensure faithful human responses, it is unclear whether prompting LLMs with the same surveys satisfies similar premises out-of-the-box. We devote this work to gain systematic insights into the survey responses of LLMs, what we can expect to learn from them, and to what extent they resemble those of human populations.
1.1
Our work
The basis of our investigation is the American Community Survey (ACS), a demographic survey conducted by the U.S. Census Bureau at a national level, yearly basis. We curate a questionnaire composing of 25 multiple choice questions from the 2019 ACS. We prompt 43 language models of varying size with these questions, individually and in sequence, and record their probability distribution over answers. Based on the collected data, we investigate the following two questions: What can we infer about LLMs, and the data they have been trained on, from their survey responses? Does the data generated by prompting models to answer the ACS questionnaire qualitatively resemble the census data collected by surveying the U.S. population? See Figure
We start by inspecting models' distributions over answers to individual survey questions when the questions are asked independently. We observe that the entropy of response distributions differs substantially across models of varying size. Entropy tends to increase log linearly with model size, and it is preserved across different questions asked. We find that this differences arise because strong ordering and labeling biases confound models' answers. fact, after adjusting for such systematic biases through randomized choice ordering, we find that response distributions are very similar across models and tend to correspond to highly balanced answers.
Comparing models' adjusted responses to those of the U.S. census population, we find that natural variations in entropy across questions are not reflected in the responses. Instead, on average across questions, models' responses are no closer to the census population, or the population of any state within the US, than to a fixed uniform baseline. This qualitative difference between model responses and human data puts into question the insights that can be gained from such comparisons. We find that even after instruction-tuning this trend persists, and model responses have consistently higher entropy than any human population we compare to, independent of the survey used. Only for models of size larger than 70 billion parameters we can recognize trend that the divergence between model responses and the census data decreases after instruction-tuming.
With these insights in mind, we inspect conjectures from prior work related to survey derived alignment metrics, that is, that differences in similarity between models' and populations' responses might be attributable certain demographics being better represented in the training data. Instead, our results suggest much simpler explanation: the relative alignment of model responses with different

---

demographic subgroups can be explained by the entropy of the subgroups' responses, irrespective of the data or training procedure employed train the model. We demonstrate this beyond the ACS on other surveys considered by prior work. As such, our findings provide important context to prior studies that employ surveys to examine the biases of L.LMs.
More broadly, our findings suggest caution when treating language models' survey responses as faithful representation of any human population, at least a present time, as it could lead to potentially misguided conclusions about alignment.
1.2 Related work
Despite the syntactical similarities, there are important differences between evaluating LLMs on the basis of their survey responses and traditional question answering evaluations [Liang et al. 2022]. Question answering (QA) tasks predominantly serve the purpose of knowledge testing [e.g.. Kwiatkowski et al. 2019 Rajpurkar et al. 2016 Talmor et af., 2019, Mihaylov et al. 2018). such setting, language model's 's answer to some unambiguous input question is extracted by computing its most likely completion. Similarly, for questions that lack a clear answer (c.g., "Angela and Patrick are sitting together. Who is an entrepreneur?") models' most likely response have been used to
investigate various biases of LLMs Li et al. 2020 Mao et al. 2021 Perez et al. 2023 Abid et al. 2021
Jiang et al., 2022].
When evaluating LLMs on the basis of survey questions, the focus is not on the model's most likely completion but rather on the probability distribution that the model assigns to various answer choices. For example, not whether the model is more likely to answer "Yes" than "No" to a given survey question, but the normalized probability assigned to each of the two answer choices. See Figure [ More concretely, Santurkar et al. [2023] study LLMs' answer distributions for multiple-choice opinion polling questions, measuring their similarity to those of various U.S. demographic groups. They extract models' answer distributions from the next token probabilities corresponding to each answer choice. Subsequent works employ similar methodology but instead consider transnational opinion surveys [Durmus et al., 2023 AlKhamissi et al., 2024] and moral beliefs surveys (Scherrer et al. 2024]. We adopt this popular methodology to systematically investigate the properties of models answer distributions on the basis of a cell-established demographic survey.
Instead of asking questions individually. Hartmann et al. 2202)] Rutinowski et al. [2023] Motoki et al. [2023). Feng et al.) (2023) sequentially prompt language models to answer entire political compass or voting advice questionnaires. Rather than aggregating answers into a political affinity score, our focus is instead on examining whether models' responses qualitatively resemble those of human populations. We discuss this sequential generation setting in detail in Appendix Fr
Lastly, there is an emerging body of research that integrates LLMs into computational social sci- ence [Ziems et al. 2024). This includes tasks such as taxonomic labeling, where language models are employed for tasks such as opinion prediction [Kim and Lee [023 Mellon et al. [2022], and free-form coding, where language models are used to generate explanations for social science constructs [Nelson jet al. 202T]. Recent studies have also investigated the feasibility of using L.LMs to simulate human participants in psychological, psycholinguistic, and social psychology experiments (Dillion et al.) |023]Aher
2023), ), or as proxies for specific human populations social science research [Ar gyle et al.) 2023 Lee et al. [203 Sanders et al.) [023] and economics [Brand et al. [2023] Horton 2023]. Within this context, our work suggests caution in relying on the survey responses of LLMs to
2023, /Aher et al.
elicit synthetic responses that resemble those of human populations and highlights potential pitfalls.
2 Surveying language models
We employ the de-facto standard methodology to survey language models introduced by Santurkar et al. [2023.. For every survey question, we generate a prompt containing the multiple-choiee question and we collect language models' probability distribution over answer choices. Formally, for a given model m and survey question g we define the model's survey response as a categorical random variable Re which can take k, values corresponding to the number of answer choices to question q. The respective answer distributions are then contrasted with those of human populations align various dimensions. The overall setup is illustrated in Figure] I
3

---

Prompting, We determine the event probabilities of R by prompting model m as follows:
1. We construct an imput prompt of the form "Question: <question> \n A. <choice 1 > >\n B.
<choice 2> An <choice ki> An Answer:".
2. We query language models with the input prompt and obtain their output distribution over next-token probabilities. We select the ki output probabilities corresponding to each answer choice (e.g., the tokens "A", "B", etc.), and we renormalize to obtain the probability distribution over survey answers.
The chosen style of prompt is standard for question answering tasks [Hendrycks et al. 202 used
in OpinionQA [Santurkar et al. 2023 J. and follows the best practices for social science research recommended by Ziems et al. [2024]. For completeness we perform several prompt ablations. including the prompt variations used by Argyle et al. [2023], Santurkar al. [2023) and Durmus
et al. [023). We find our takes to be robust to such changes, see Appendix However, note
that our goal is not to engineer better prompts, but to critically examine popular scientifie practices.
Survey questions. We use a representative subset of 25 multiple-choice questions from the 2019 ACS questionnaire. We denote the set of questions by Q. The questions cover basic demographic information, education attainment, healthcare coverage, disability status, family status, veteran status, employment status, and income. We generally consider the questions and answers as they appear in the ACS questionnaire. Figure depicts an example question. We refer to Appendix A.1 for our list of questions and the exact framing used for each question.
Models surveyed. We survey 43 language models of size varying from 110M to 175B parameters: the base models GPT-2 Radffrdd et al., 2019), GPT-Neo (Black et al.) 2021 Pythia [Biderman et al. 2023], MPT [MosaicML 2023]. Llama 2 [Touvron et al. 2023 ,Llama 3 (Dubey et al. 2024 and
GPT- Brown et al. 2020]; as well the instruct variants of MPT 7B and GPT 20B, the
Dolly fine-tune of Pythia 12B [Databricks 2003). Llama 2 Chat, Llama 3 Instruct, the text-davinci
variants of GPT-3 (uuyang al. 2022 and GPT-4 OpenAl 2023
Reference data evaluation. We use the responses collected by the U.S. Census Bureau when reference data. In particular,
use the 2019 ACS public use microdata sample (henceforth census data). The data contains the anonymized responses of around 3.2 million individuals in the United States. For each survey question Q, denote the census' population-lese -leell response as a categorical random variable Cg whose event probabilities are the relative frequency of each answer choice among survey respondents. We use U, to denote the uniform distribution over answers. Given these two reference points, we evaluate language models' responses Ry along two dimensions:
surveying the U.S. population p
We use entropy to measure the degree of variation in models' responses. We denote the entropy of random variable as (R). To meaningfully compare the entropy of responses across questions with varying number of choices kn. we report normalized entropy, that is, the entropy implies that model m's ''s survey response to
relative to the uniform distribution. H (RR.) question q is uniformly distributed (i.e., H (U,)) 1).
We use the Kullback-Leiller (KL) divergence to measure the "similarity" between two distri- butions over answers. We write (L(R,, || C,) for the KL divergence between the response distribution I25 of model m to question g and the corresponding aggregate response distribution C, observed in the census data. The larger the KL distance between two distributions, the more dissimilar the two distributions are.
Note that the KL divergence between any distribution and the uniform distribution corresponds to the entropy difference. For normalized entropy this yields KL(C, || Ug) k,, (C,)))
For OpenAl's models, only have API. this case, we assign to unseen probabilities (if any) minimum between the remaining probability mass and smallest observed probability, following the methodology of Santurkar et (0233
the top-5 next-token log probabilities through the OpenAI

---

Response entropy
SEX
HICOV
FER
CIT
ENG
1.0
0.5
102
10"
10th
10th
10a
10" 10th 10t1
10a
10" 10th 10th
102 10" 10th 10t1
10a 102 10th 10
Model size
Model size
Model size
Model size
Model size
Model
U.S. census
Uniform distribution
(a) Entropy of base models' responses, for five of the ACS questions.
Response entropy
Survey question
1.0
GPT2 110M
GPT Neo 125M
0.0
GPT2
GFT2
355M
GPT Neo
77
774M
GPT2
1.38
GPT3
1.58
GPT Neo
2.78
6.78
MPT
Pythia 78
LLaMA 78
Liama 77
Pythin
88
LLSMA
128
138
GPT3
Liaa
GPT Neox 208
138
138
LLaMA 308
LLSMA 658
Ulama 708
Uama
GPT3 1758
708
U.S. Census
(b) Entropy of base models' responses to the ACS, ordered by model size.
Figure 2: Entropy model responses across the ACS questions for naive prompting. Entropy of models' responses (+) tends increase log-linearly with model size, irrespective of the underlying response entropy observed in the U.S. census (-).
Randomized choice ordering. For several investigations we survey models under randomized choice ordering. This means, for a given question the answer choice ordering, i.c., the assignment of answers (c.g., "male", "femalc") to choice labels ("A", "B", etc), while the choice labels are kept in alphabetic order. We evaluate modcls' survey responses under all possible choice orderings and we use R to denote the expected distribution over answers and O to denote the expected distribution over selected choice labels. For questions with more than answers we evaluate a maximum of 5000 permutations. For OpenAl's models we evaluate up to 50 permutations due to the costs of querying the OpenAI API. This distinction serves to decouple model's 'ss tendency towards picking particular answer from its tendency towards picking particular choice label. In the following we refer to the expected survey response R& under uniformly distributed choice ordering as the adjusted survey response. We will come back to this in Section
prompt models with different permutations of
Systematic biases in models' survey responses
We start by surveying the base pre-trained models. We present survey questions independently of one another, showing the answer choices the same order as the ACS.
For first investigation, we consider the normalized entropy of models' responses to the "SEX", "HICOV", and "FER" questions. The SEX question inquiries about the person's sex, encoded as male female, the HICOV question inquiries whether the person is currently covered by any health insurance plan, and the FER question inquires whether the person has given birth in the past months. When surveying the U.S. population, these three questions elicit responses with very different entropy: responses to the SEX question are almost uniformly distributed, whereas most people answer "No" to the FER question. In contrast, shown in Figure 2(a) the entropy of models' responses to these three questions are surprisingly similar. In particular, we find that the entropy models' responses tends to increase log-linearly with model size, independent of the question asked. This trend is consistent across all ACS survey questions, see Figure 8 in Appendix B.I
For a broader picture, we illustrate models' response entropy across all survey questions in Figure 2(b) The blue dots represent models' responses individual questions, and the green dots represent the entropy of the responses of the U.S. census. We order models by size. We observe that the entropy of responses the U.S. census greatly varies across questions. In contrast, for any given model, the entropy of its responses varies substantially less
5

---

Survey question
A-bias
0.0
-0.5
Alway GPT GPT Neo 12
11OM
GPT SS GPT Noo GPT
SM
T7MM
Neode Nee
mag inst
122
Liama CS Reo Cham 2.
Llame Chat 138 208
130
Chat
00
Nett daviin
Tee inst
17758
-ext- davinci- Turpo
cuinc- -000
GPT Turbe 01 0125
0022
GPT-4 Never
0613
Neoxi Liamy
Lama
Liamia
Figure 3: A-bias of in model responses across ACS questions. Each dot corresponds to one of the 25 questions. Models are ordered by size. As reference, the extreme points illustrate A-bias for model that always answers 'A' and a model that never answers A'' All models suffer from substantial A-bias.
Overall, we find that models' response distributions seem to be widely independent of the survey question asked, and variations across models are much larger than variations across questions. This lead us to suspect that observed differences across models might arise mostly due to systematic biases.
3.1 Testing for systematic biases: A-bias
It is well-known that language models' most likely answer to multiple-choice questions can change depending on seemingly minor factors such as the ordering few-shot examples (Zhao et al.] 2021 Lu et al. 2022] or the ordering of answer choices [Robinson and Wingate| [023a] We are interested in the extent to which changes choice ordering affect a model's output distribution over answers.
We start by measuring A-bias: the tendency of a model towards picking the answer choice labeled "A". In particular, seek to study the extent to which the strength of this bias explains the differences in responses observed across models. For an unbiased model that outputs the same answer distribution irrespective of choice ordering. the expected choice distribution OTr under randomized choice ordering would match precisely the uniform distribution (c.g., P("A")= P("B"") 0.5). We define a model's s A-bias as its absolute deviation from this unbiased baseline:
Abias"" t2 == P(O 4 = "A") 1/kg
We measure A-bias for each question and model m. Results are illustrated in Figure 3 We again sort models by their size. We observe all models exhibit substantial A-bias. However, models the order of few billion parameters or fewer consistently exhibit particularly strong A-bias, and tend towards mono answers. We additionally observe that the strength of A-bias in instruction or RLHF tuned models is similar to that of base models, see Appendix B.2 A plausible cxplanation for small models exhibiting strong A-bias is that the ability to answer MMLU-style multiple-choice questions only emerges for models of sufficient scale Dominguez-Olmedo et al. 2024
We investigate other types of labeling and position bias (e.g., last-choice bias) in Appendix C Overall, we find a strong tendency of LLMs to pick up spurious signals in the way that answers are ordered and labeled, rather than their semantic meaning. Notably, contrast to the primacy bias observed in humans [Groves et al. 2009), we find that models exhibit substantial A-bias even when randomizing the position of the "A" choice. Our findings are consistent with the concurrent work of ofujuat et al.] [203]. which similarly finds that models' response biases to multiple-choice survey questions are generally not human-like. The orthogonal work of Wang et al. (2024) additionally shows that models' responses to multiple-choice survey questions may not consistently reflect their free-form outputs.
In summary, we find that systematic biases confound models' answer distributions. This makes challenging to draw robust conclusions about inherent properties of LLMs, such as the opinions or populations they best represent. For example, simply reversing the order of answers to the "SEX" question could lead to GPT-2 seemingly representing population where females are significantly over-represented, whereas a reverse conclusion would be drawn when using the standard answer order. While much research went into designing the ACS to elicit faithful answers and eliminate systematic biases when surveying human populations, simply using the same question framing does not protect against the systematic response biases that language models exhibit.

---

Choice ordering
Response
Question:
months.
person given birth any children?
A. Yes
Adjusted response
B. No
Answer:
P("A")
0.82
P("8")
0.11
Choice ordering
Response
Question: In the past 12 months, has this person given birth children?
B. Yes
Answer:
A. No Ye
P("A")
0.80
P("B)
0.15
Response entropy
Base models
instruction-tuned and RLHF models
1.0
0.5
Census
78
B
vicuna 38
TUTBO SUIIO 2613
U.S. Census
Pythis Lamn
Pychi
MPT inst
Chat
Laama Sannnn
th anding
Liana
Liaman Neoxt
GF
Figure 4: Entropy of model responses after adjustment. (top) Illustration of how adjustment is performed. We average models' responses over all possible answer orderings. (bottom) Entropy of models' responses after adjustment. Entropy of base models' responses is close (i.e., uniform). Instruction tuned- models exhibit substantially higher variations in entropy across questions.
4 Inspecting adjusted responses
To eliminate confounding due to labeling and ordering biases, we survey models under randomized choice ordering, borrowing an established methodology to adjust for ordering biases of all kinds in survey research [Groves et al. 2009]. Also a recent work in LLM research adopts this methodol- ogy [Robinson and Wingate 2023b]. the following, we refer the expected response after answer choice randomization as the adjusted response.
In Figure we plot the normalized entropy of models' adjusted responses for the ACS questions considered. First focusing on base models, and comparing the results to Figure 2 we find that after adjustment, 1) the variations in responses' entropy across survey questions are very small, we no longer observe the trend of the entropy of model responses increasing log- ligeiinerly with model size. fact, models' survey responses have a normalized entropy of approximately irrespective of model size or survey question asked. This validates our initial hypothesis that, without adjustment, variations responses across base models arise predominantly due systematic biases such as A-bias, rather than the content of the survey questions asked.
4.1 Effect of instruction tuning
We now evaluate language models that have been fine-tuned with instructions and/or human prefer- ences, henceforth "instruction-tumee models". In the right plot of Figure 4 we show the normalized entropy of instruction- -uned models' ACS survey responses after adjustment. We observe that instruc- tion guned-modehl to base models. But in general, the entropy of their responses remains higher than the entropy of the census responses. Interestingly, as we will see, although deviating more from uniform, model responses do not tend to be closer to the U.S. census responses.
exhibit substantially higher variations entropy across questions compared
7

---

Base models
Instruction-tuned and RLHF models
KL(model, Ref.)
1958
Dally 12
Inst
Vicuna c0
taxt text davinng- ving--022
108 Inst
00
Text-daying saynes 003
T turns 25
furno
GPT-4 0813
Llang
Inst
Lia Liama Cha
Kosia
Liama
Lian "spar
Liama
text text
Figure 5: Divergence between adjusted model responses and different baselines: the overall U.S.
census (x), individual U.S. states (o), and a uniform baseline (*). Smaller means more similar. Model responses are by far more similar to the uniform baseline than to any human reference population.
4.2 Comparing model responses to the U.S. census
We now investigate the similarity of language models' adjusted responses to the census data. To do so, we consider the overall U.S. census population, as well as 50 census subgroups corresponding to every state in the United States. This leads to different human reference populations.
Inspired by the alignment measures proposed by Santurkar et al. [2023 and Durmus et al. 2023 we investigate the similarity of model responses to the census data by evaluating the average divergence across questions between model responses and the census statistics As we focus on categorical questions, we evaluate average KI. divergence between each language model m and each reference population Ref, as follows:
1 Σ KL (Refg)) [Q1
KL Ref)
gEQ
Results are depicted in Figure For each model we plot the divergence to the census in black, the divergence to the different subgroups in blue, and the divergence a uniform baseline with balanced responses in red. We observe that models are strikingly more similar to the uniform baseline than to any of the populations considered. For base models, this result unsurprising, since the previous section we established that base models' responses are essentially uniform after adjustment.
Looking at Figure 5 we find consistent trend that instruction-tunin would move responses closer to the census, despite the increased deviation from uniform and the larger variations in entropy (recall Figure A Only for larger models the divergence seems clearly decrease with instruction-tuning However, all models' responses still remain significantly closer to the uniform baseline than to the U.S. census. For instance, for the GPT-4 model whose answers exhibit the highest similarity the human reference populations, only 6 out of 25 questions (24%) are closer to the U.S. census than to the uniform baseline. Given these results, drawing conclusions about the relative alignment of models with subgroups is prone to resulting in brittle conclusions.
Implications for survey-based alignment metrics
Our findings add important context to previous works studying the alignment of language models with different human subpopulations. In particular, we highlighted the tendency of models towards balanced answers. Due varying entropy in the responses of subgroups this leads to strong correlation between model aligoment and the reference population's 's entropy. The linear trend in Figure 6 visualizes this. For any given model, it consistently appears to be more "aligned" with the subpopulations exhibiting high entropy in their answers. Interestingly, we find that this trend also holds pre-adjustment, suggesting that the transformation of the response through randomized choice ordering is orthogonal to differentiating aspects of any specific population. In contrast, when comparing different models Figure 6 we can see how adjustment has large influence on their relative order. Differences across models that we see under naive prompting disappear after adjustment, which means that they should largely be attributed to systematic biases, rather than inherent properties of the model.
'Whereas Santurkar et al. 2023 use the Wasserstein distance to compare answer distributions, we use KL divergence since questions in the ACS are predominantly nominal, rather than ordinal.
8

---

Unadjusted responses
Adjusted responses
Unadjusted responses
Adjusted respanses
KLimodel, subgroup)
2.5
KL[model subgroup)
2.0
1.5
1.5
1.0
1.0
0.5
0.50
0.55
0.60
0.50
0.55
0.50
0.30
0.55
0.60
0.50
0.55
0.60
Entropy
subgroup (U.S. state) responses
Entropy of subgroup (U.S. state)
responses
Subgroup
GFT3 138
text davinci-02 003
Subgroup
GPT NEO 125M
GPT NEO 1:38
Trendlnne
GPT
0613
Trendine
353M
GPT2 1.58
GPT3 2.78
text dal-ddvidddoo 001
GPT-4 Turbo 0125
GPT2 110M
GPT2 774M
GPT NEO 2.78
GPT3 6.78
text-davinci-e 002
GPT-4 0613
(a) GPT-2 and GPT-Neo.
(b) OpenAl's API models.
Unadjusted responses
Adjusted responses
Unadjusted responses
Adjusted responses
Klmmdel, subgroup)
2.00
(Lmodee, subgroup)
1.75
1.50
1.50
1.25
1.25
1.00
1.00
0.75
0.75
0.55
0.50
0.55
0.60
0.50
0.55
0.60
0.50
0.55
0.60
Entropy of subgroup (U.S, state) responses
Entropy of subgraup (U.S. state) respanses
Subgroup
Dolly GPT NeoX 208
Subgroup Trnndine
Liama 3 8B
Uamma 708
Trendine
Pythia 79
Liama 3 8B Inst
Llama Chat 708
Pythia 120
NeoXT
Uama
Liama
MPT Instruct 78
Uama Chat 78
Liama 2 Chat 13B
Liama 70B Inst
(c) MPT, Pythia, GPT-NeoX and its instruction variants.
(d) Llama and its instruction and chat variants.
Figure 6: Alignment of models with different census subgroups. All models tend exhibit similar relative alignment, and the divergence metric decreases with the entropy of the subgroups" responses.
Taken together our findings imply that the survey-derived alignment measure is more informative of differences in the reference populations rather than the language models is aims to evaluate. Model particularities, such as the pre-training data used, instruction tuning or the use of reinforcement learning with human feedback, seem to have little impact which population is best represented.
5.1 Beyond the ACS
To inspect whether this trend changes with the content of the questions asked, we reproduce our experiments with additional surveys. We use the American Trends Panel (ATP) opinion surveys considered by Santurkar et al. 2023]. and the Pew Research's Global Attitudes Surveys (GAS) and World Values Surveys (WVS) considered by Durmus et al. [2023]. These surveys encompass around 1500 questions and 60 U.S. demographic subgroups, and around 2300 questions and 60 national populations, respectively. We adopt the alignment metries considered by the aforementioned works. We find that our insights gained from the ACS also hold for the ATP and GAS/WVS surveys. In particular, similarly find linear trend between the alignment metrics and subgroups' entropy of responses, in particular after adjustment, see Figure] 7 Note here that alignment and divergence are negatively correlated by definition. Interestingly, this observation explains some of the findings in prior works. For example. Santurkar et al. [2023) find that "all the base models share striking similarities-e.g.. being most aligned with lower income, moderate, and Protestant or Roman Catholic groups" and "our analysis [...] surfaces groups whose opinions are poorly reflected by current LLMs (e.g., 65+ and widowed individuals)". For the ATP surveys considered, low income, moderate, and Protestant/Cathol
are precisely the demographic subgroups with responses closest to uniformly random among the income, political ideology, and religion demographic subgroups; whereas age 65+ and widowed are the demographic subgroups with responses furthest from uniform among the age and marital status demographic subgroups. Further, [Santurkar et al. [2023) observe that RI.FF can result in "substantial shift [...] towards more liberal, educated, and wealthy [demographic

---

Unadjusled responses
Adjusted responses
Unadjusted responses
Adjusted responses
Alignment (model, subgroue)
0.85
(madel, subgroupp
0.90
Alignment
0.85
0.80
0.80
0.75
0.70
0.75
0.80
0.75
0.63
0.75
0.65
Entropy of subgroup's responses
respoones
Entropy of subgroup's responses
Subgroup
Liama s8 Inst
Lama 708
Subgroup
Liama
Uama 708
Trendline
Liama 2 700
Uama 708 Inst
Trendline
Liama 708
Uama 708 Inst
Llema 8B
Llama 2 Chat 706
Liama
Liama Chat 708
(a) ATP surveys.
(b) GAS/WVS surveys.
Figure 7: Alignment beyond ACS for selected models. We adopt the measures of Santurkar al. [2023) and Durmus et al. [2023] on ATP and GAS/VVS opinion surveys. Again, the alignment be- tween models and given subpopulation correlates with the entropy of the subpopulations' responses.
groups]". Our results suggest that this could be artifact of systematic biases. For the ATP surveys, we observe three outliers for which its alignment before adjustment is not correlated with the entropy of subgroup's 's responses: Llama 70B Chat and the two Llama 3 Instruct models. These models with largest pre-training compute considered. However, after adjustment, the alignment trends of Llama 70B Chat and the Llama Instruct models are remarkably similar to that of their corresponding base models and all other LLMs.
the
Conclusion
We used popular methodology elicit LLMs' answer distributions to survey questions and closely examined the responses on the basis of the prime US demographic survey. We found that model responses are dominated by systematic ordering biases and do not exhibit the natural variations in entropy found in the human reference data collected by the US census. Even after adjusting for ordering biases. LLMs' responses still do not resemble those of human populations. Instead, they exhibit consistently high entropy, independent of the question asked. This holds true irrespective of model size or fine-tuning with human preferences.
These findings have important implications for insights gained from survey-derived alignment metrics. In particular, explains why models of varying size all exhibit the same trend: they are most aligned with subgroups who happen to have balanced answers for the survey questions under consideration. For all models and surveys considered, alignment appears to be a proxy for the entropy of subgroups, rather than an inherent property of the model, or its training data.
We want to reiterate that our focus lies questioning popular methodology of eliciting survey responses from large language models using multiple choice prompting. At the example of this methodology results highlight important pitfall and suggest caution expect robust insights when comparing such responses against those of human populations. The robustness and quality of established survey does not seamlessly translate from the results obtained by surveying human populations to the logits output by LLMs. More research is urgently needed to design methodologies for getting insights into the inherent biases of LLMs and the population they might represent. Here public surveys and their accompanying data offer exciting potential and the could play an important role as benchmarking tool for systematic evaluations of LLMs, see [Cruz et al., 2024] an example. Although the use of survey data for LLM research has recently gained popularity, it still remains a widely under explored data source.
Acknowledgements
The authors would like to thank Frauke Kreuter and the Social Data Science and AI Lab at Ludwig- -anmiiiann-Uuiversiini Munich for inspiring discussions on an earlier version of this manuscript. Celestine Mendler-Dünner acknowledges financial support from the Hector Foundation.
10

---

References
Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. In AAAI/ACM Conference on Al, Ethics, and Society, pages 298-306, 2021.
Gati V Aher, Rosa Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337-371, 2023.
Badr AlKhamissi, Muhammad EINokrashy, Mai Alkhamissi, and Mona Diab. Investigating cultural alignment of large language models. In Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 12404-12444 12422, 2024.
Lisa P Argyle, Ethan C Busby, Naney Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337-351, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: a suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, 2023.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensor,low, 2021.
James Brand, Ayelet Israeli, and Donald Ngwe. Using GPT for Market Research. Harvard Business School Marketing Unit Working Paper No. 23-062, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877-1901, 2020.
André Cruz, Moritz Hardt, and Celestine Mendler-Dünner, Evaluating language models as risk scores, ArXiv preprint arXiv:2407.14614, 2024.
Databricks. Dolly 12b, 2023. URL htttpp///g.ttubb..cm/attttarrttt
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Can Al language models replace human participants? Trends in Cognitive Sciences, 2023.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 2021.
Ricardo Olminoo Florian E Dorner, and Moritz Hardt. Training the test task confounds evaluation and emergence. ArXiv preprint 2407.07900 2024.
Florian Dorner, Tom Sühr, Samira Samadi, and Augustin Kelava. Do personality tests generalize to large language models? In NeurIPS Workshop on Socially Responsible Language Modelling Research, 2023.
Abhimanyu Dubey. Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. ArXiv preprint arXiv:2407.21788 2024.
Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodd Danny Hernandez, Nicholas Joseph, et al. Towards measuring the representation of subjective global opinions in language models, ArXiv preprint arXiv:2306. 16388, 2023.
11

---

Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From Pretraining Data to Language Models Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. Findings of the Association for Computational Linguistics, 2023.
R.M. Groves, F.J. Fowler, M.P. Couper, J.M. Lepkowski, E. Singer, and R. Tourangeau. Survey Methodology. Wiley, 2009.
Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental. left-libertarian orientation. ArXiv preprint arXiv:2301.01766 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
John J Horton. Large language models simulated economic agents: What learn from homo silicus? NBER Working Paper, 2023.
Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. CommunityLM: Probing Partisan Worldviews from Language Models. In International Conference on Computational Linguistics, 2022.
Junsol Kim and Byungkyu Lee. Al-Augmented Surveys: Leveraging Large Language Models for Opinion Prediction in Nationally Representative Surveys. ArXiv preprint arxiv:2305.09633 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019.
Sanguk Lee, Tai- Tai-Quan Peng, Matthew H Goldberg, Seth A Rosenthal, John E Kotcher, Edward W Maibach, and Anthony Leiserowitz. Can large language models capture public opinion about global warming? an empirical assessment of algorithmic fidelity and bias. ArXiv preprint arXiv:2311.002/7, 2023.
Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. Unqovering stereo- typing biases via underspecified questions. In Findings of the Association for Computational Linguistics, pages 3475 3489,,48922220 2020.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak. Frieda Rong, Hongyu Ren. Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel Orr, Lucia Zheng. Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. ArXiv preprint
1.09110 2022.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Annual Meeting of the Association for Computational Linguistics, volume 1, pages 8086-8086 2022.
Andrew Mao, Naveen Raman, Matthew Shu, Eric Li, Franklin Yang, and Jordan Boyd-Graber. Eliciting bias in question answering models through ambiguity. In Workshop on Machine Reading for Question Answering, pages 92-99. 2021.
Jonathan Mellon, Jack Bailey, Ralph Scott, James Breckwoldr, Marta Miori, and Phillip Schmedeman. Do ais know what the most important issue is? using language models to code open-text social survey responses scale. SSRN Electronic Journal, 2022.
12

---

Todor Mihaylov. Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, 2018.
MosaicML. Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs, 2023. URL mwwaicmiiiccccm wwt.coo///ccce wwtg:mpt-gooo/p/o//o//
Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. More human than human: Measuring chatgpt political bias. Available SSRV 4372349, 2023.
Laura K Nelson, Derek Burk, Marcel Knudsen, and Leslie McCall. The future of coding: A com- parison of hand-coding and three types of computer-asisted text analysis methods. Sociological Methods & Research, 50(1):202-235 2021.
OpenAL. Gpt-4 technical report. ArXiv preprint arXiv:2303.08774 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022.
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez. Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics, pages 13387-13434, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing, 2016.
Joshua Robinson and David Wingate. Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations, 2023a.
Joshua Robinson and David Wingate. Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations, 2023b.
Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, and Markus Pauly. The Self-
Perception and Political Biases of ChatGPT. ArXiv preprint arXiv:2304.07333, 2023.
Nathan E Sanders, Alex Ulinich, and Bruce Schneier. Demonstrations of the potential of bi-based political issue polling. ArXiv preprint (xXX::2307.04811, 2023.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? International Conference on Machine Learning. 2023.
Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. Evaluating the moral beliefs encoded in 1lms. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQ: A question answering challenge targeting commonsense knowledge. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages $149-4158, 2019.
Lindia Tjuatja. Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. Do Ilms exhibit human-like response biases? study in survey design. ArXiv preprint arXiv:231..04076 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint arXiv:2307.09288. 2023.
Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber- Genzr, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. "my answer is C": First-token probabilities do not match text answers in instruction-uun tuned language models. arXiv preprint arXiv:2402. 14499, 2024.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706, 2021.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguistics, 50(1): 237-291, 2024.
14

---

A Experimental details
We use the American Community Survey (ACS) Public Use Microdata Sample (PUMS) files made available by the U.S. Census Bureau The data itself is governed by the terms of use provided by the Census Bureau We download the data directly from the U.S. Census using the Folktables Python package [Ding et al. 2021]. We download the files corresponding to the year 2019.
We downloaded the publicly available language model weights from their respective official Hugging- Face repositories. We run the models in an internal cluster. The total number of GPU hours needed to complete all experiments is approximately 1500 (NVIDIA A100). The budget spent querying the OpenAl models was approximately $200.
We open source the code replicate all experiments
In addition, the repository contains notebooks visualize the results of our investigations under different prompt ablations.
A.1 Survey questionnaire used
The exact questionnaire used in our experiments can be retrieved from our Github repository. We consider 25 questions from the 2019 ACS questionnaire corresponding the following variables in the Public Use Microdata Sample: SEX, AGEP, HISP, RACIP, NATIVITY, CIT. SCH, SCHL, LANX, ENG. HICOV, DEAR, DEYE, MAR, FER, GCL, MIL, WRK, ESR, JWTRNS, WKL, WKWN, WKHP, COW, PINCP. We take all questions as they appear in the ACS, with the exceptions:
HISP:
The ACS contains answer choices corresponding different Hispanic, Latino, and Spanish origins, and respondents are instructed to write down their origin if their origin is not among the choices provided. We instead provide two choices: "Yes" and "No".
The ACS contains 15 answer choices, allows for selecting multiple choices, and respondents are instructed to write down their race if not among those in the multiple choice. The PUMS then provides up 170 race codes (RAC2P and RAC3P). We instead present choices, corresponding to the race codes of the RACIP varible in the PUMS data dictionary.
RACIP:
Additionally, the variables ESR and COW are not directly associated with any single question in the ACS, but rather aggregate employment information. We formulate them questions by taking the PUMS data dictionary's s variable and codes descriptions. Lastly, for the questions corresponding to the variables AGE, WKWN, WKHP, and PINCP, respondents are asked to write down an integer number. We convert such questions to multiple-choice via binning.
B Detailed experimental results
B.1 Model responses across questions before and after adjusting for A-bias
The results in this section complement Section 3, and pertain non-instruction-tue tuned language models. When surveying models without choice order randomization, we observe that the entropy of model responses tends to increase log-linearly with model size, often matching the entropy of the uniform distribution for the larger models. This trend is consistent across survey questions, irrespective the question's 's distribution responses observed in the U.S. census (Figure g
B.2 A-bias of instruction-tun models
The results in this section complement Section 3.1 and pertain instruction- language models well as language models fine-tuned with reinforcement learning with human feedback (RLHF). We observe that the strength of A-bias for these models, plotted in Figure 9 is comparable to that of base pre-trained models, plotted in Figure 3 This motivates the of choice-order randomization in order to eliminate confounding due to labeling biases in models' responses.
http://www.ccaaassso///ddooeeess otwp://wwwoooooomoooooeeoooeeeee html
http::/tttha.ooomccaalllomiiiinn models models
15

---

SEX
AGER
HISPR
RACIPR
NATIVITY
CIT
SCH
Entropy
SCHLR
LANX
ENG
HICOV
DEAR
DEYE
MAR
Entropy
FER
GCL
MIL
WRK
ESR
JWTRNS
WKL
Entropy
WKWN
WKHPR
COWR
PINCPR
10t0
10a 10th Model size
10a 10th Model size
Entropy
Model size
Model m
U.S. census
10l"10 Model size
10a 10 10t010"1 10t1
10 10
10f 10th 10 10tt
Uniform distribution
Model size
Model size
Model size
Figure 8: Normalized entropy of survey responses for individual questions (without adjustment).
Survey question
0.4
A-bios
0.2
MPT inst. MPT 78
Kapia 78
Vicuna
Chat
78
Daily 128
Kogta 138
Unma Chat 138
Vicuna 138
NeoX7 Chat 208
Liama a that 708
laxt: dadddinn davin:i-on vinc--001
taxt 2avincc-002 davidd 1evinc--00
GPT Turbo 0613
GPF-4 lurbe 0125
GPT-4 0613
Liama
Figure 9: A-bias of instruction tuned models.
B.3 Relative alignment across demographic subgroups
The results presented here complement those of Section 5 We plot the average KI divergence between each language model and each demopgrahic subpopulation (U.S, state) against the average entropy of the subgroup's responses. For readability, we split models into GPT-2 and GPT-Neo (Figure 6(a)
OpenAl's API models (Figure 6(b). MPT, Pythia, GPT-NeoX and its instruction variants (Figure ((C) and LLaMA, Llama 2 and its instruction and chat variants (Figure 6(d).
C Ordering bias: further experiments
We conduct additional randomization experiments pertaining to answer choice position and labeling bias, complimenting Section 3. We consider the GPT-2, GPT Neo, MPT, Pythia, and LLaMA models. The experiments follow consistent setup:
1. We randomize both the order in which choices are presented and the label (i.c., letter) as- signed to each answer choice. For example, for the "sex" question, the possible combinations are "A. Male B. Female". "A. Female B. Male". "B. Male Female", and "B. Female A. Male". Note that in the experiments presented in Section 3.1 we only randomized over the order in which choices are presented (i.e.. the "A" choice was always presented first).
2. We compute the output distribution over responses for choice position (the probability assigned to the first, second, etc., answer choice presented) and letter assignment (the probability assigned to the answer choice assigned "A", "B", etc.).
For each model and survey question, we estimate the expected distribution over responses for both choice position and letter assignment by collecting 3,000 responses (step 2) under different
16

---

randomizations of choice position and letter assignment (step 1). A model with no position and labeling biases would assign the same probability distribution answer choices (e.g.. "male" and "female") regardless of position or letter assignment, and therefore the expected distributions over position (e.g., selecting the first choice) and letter assignment (e.g., selecting "A") would be uniform.
C.1 Disentangling ordering bias into positioning bias and labeling bias
We perform chi-square tests to determine whether language models' output responses distributions position and letter assignment significantly deviate from the uniform distribution (i.e., if there exists statistically significant bias in position or letter assignment). Since we collect 3,000 response distributions under randomized choice position and letter assignment, we ensure high test power (2 0.98) in detecting small effect sizes (0.1) at significance level of 0.05.
We find that models exhibit significant positioning and labelling for most survey questions, Figure TO] We observe that labelling is more prevalent that positioning bias. While both tend to decrease with model size, order bias decreases more significantly with model size, whereas labeling bias tends to be very prevalent across all model sizes. In Figure [ we plot both the strength of A-bias and first-choice bias across survey questions. The strength of A-bias tends to be greater than that of first-choice bias, particularly for the smaller models.
% question with siginificant blas
% questions with
100
75
50
50
10a
10
10
10a
10"
1010
Model size
Model size
(a) Significant ordering bias.
(b) Significant label bias.
Figure 10: questions.
All models exhibit statistically significant letter and ordering bias for most survey
Unblased sampling
Survey questions
Unblased sampling
Survey questions
bias
0.4
0.2
0.2
GPT2 110M
NEO
5T1 SM.MM.
GPT2 NEO 30
GPT NED
58
"MPT 10
Pyrto 78
LLaMA 73
Pysbia SMA 138
128
LLAMA 306
310M MED 1MM
GPTS 355M
OF
GPT NEO 38
OPT 58
MEO
MOT 78
Pychis
LLaMAA
78
79
2A Neol
(L
(a) A-bias
(b) First choice bias
Figure Models, particularly those with less than few billion parameters, tend exhibit stronger A-bias than first-choice bias.
C.2 I-bias
We hypothesize that A-bias is prevalent because the single character "A" is relatively frequent as the starting word of sentence in written English. We test this hypothesis by replacing the character "B" with "T" when presenting the survey questions, since the character "T" is even more frequent as the starting word of sentence in written English. We randomize over choice ordering and label assignment as in the previous evaluation. We find that, when presenting both "A" and "T", small models then exhibit I-bias rather than A-bias (Figure 12), supporting initial hypothesis.
C.3 Using letters with similar frequency in written English
Motivated by the I-bias experiment, we now examine whether labeling bias can mitigated by using letters that have similar frequency in written English. Therefore, instead of assigning to choices the
17

---

Unalased sampling
Survey questions
Unblased sampling
Survey questions
I-bias
0.4
0.4
0.2
110M
GPT2 353H
30
1.50
78
Pynbor
70
78
78
12
138
LIaNA
208
308
GP12 11M
0.0
NED 225M
GF 335M
GPT 714M
NEO
GPT RED
78
78
71
78
MAAAA 139
120
MOOX
203 MA
6P
GPT
GPT
(a) A-bias in the "A", "T" randomization experiment.
(b) I-bias the "A", "T" randomization experiment.
Figure 12: When both "A" and "T" are present, small models exhibit I-bias rather than A- A-bias.
100
100
Letter bias
Order bias
75
75
109
100
1033
10a
10?
10th
Model size
Model size
(a) Significant labelling bias.
(b) Significant ordering
bias.
Figure 13: "R", "S", "N", etc. randomization experiment. All models, irrespective of size, exhibit statistically significant letter and positioning bias for most survey questions.
labels "A", etc. we assign the following labels: "R", "S", "N", "L", "O", "O","""""" "M", "P", "W", "U", "Y", "V". We find that, compared to the "A", "B", etc. randomization experiment, the percentage of questions for which models exhibit significant labeling bias somewhat decreases (Figure (3). However, models tend to exhibit substantially more position bias. This indicates that, in the absence of a label that provides a strong signal (e.g., "A" or "["), models tend to exhibit significantly higher choice-ordering bias, irrespective of model size.
etc.
D Prompt ablations
We reproduce experiments using different prompts query the model. Due to the cost of querying OpenAl's models, we only perform these ablations for models with publicly available weights. The notebooks with all figures can be retrieved from our Github repository.
Overall, the prompt ablation results are very consistent with the findings presented the main text of the paper. In the following we provide an overview over the different ablations performed.
D.1 System rompt used for GPT-3.5 and GPT-4
When querying GPT-3.5, GPT-4, and GPT-4 Turbo, we use the system prompt Please respond with single letter., as otherwise for most questions none of the top-5 logits correspond to answer choice labels (e.g., "A", "B""). Note that this problematic arises due to the fact that the OpenAl API only allows access to the top 5 logits. We adapt the system prompt used by Dorner et al. 2023 in the context of surveying GPT-4 with standarized personality tests.
D.2 Individual survey questions
First, we use different styles prompt individual survey questions. We enumerate the prompt styles as (PI)-(P8).
Additional context. We first explore whether including additional context signaling that the questions presented are from the American Community Survey, or that they are to be answered by
18

---

U.S. households. Keeping identical survey questions, we append at the start of the prompt one of the following sentences:
(P1) Bellow is question from the American Community Survey.
(P2) Answer the following queation from the American Community Survey.
(P3) Answer the following question as you lived household United States.
Asking questions in the second person. We change the framing of the questions.
(P4) We modify the survey questionnaire such that questions are formulated in the second person rather than the third person (e.g.. "What your sex?" instead of "What this person's sex?").
Including instructions. Following the prompt ablation of Santurkar et 2023 we append the start of the prompt one of the following instructions:
(P5) Please read the following saltiple-choice question carefully and select ONE the listed options.
(P6) Please read the multiple-choice question below carefully and select ONE of the listed options. Here is example of the format: : AnQuestion: Question 1\nA. Option 1\nB. Option 2
C. Option 3'AnAnswr:
Chat-style prompt.
We consider the prompt used by Durmus et al. 2023
(P7) Human:
{question] \nHere are the options: \n(options) \n Assistant: If had to select one of the options, my answer
would be
Interview-style prompt.
We consider the prompt used by Argyle et al 2023
(P8) Interviewer:
(questionn/hnoopptooonn)))
D.3 Sequential generation
We use different prompts to integrate a model's previous responses when prompting subsequent survey questions. Instead of summarizing previous responses using bullet points in Section keep previous questions and answers in-context.
Question answering. Keeping questions and answers in-context resembles the typical few-shot Q&A setting. For instance, prompting for the third question the questionnaire corresponds to
Question: {question 1)Anfoptions : (answer 1)An
Question: {question 2} 2nfopptions 2} AAAnnsswe: (answer 2} An
(answer
Question: {question 33 \nfooptions 33 \nAnwwe::
Interview-style prompt. We consider the prompting style used by Argyle al. (2023) For instance, prompting for the third question in the questionnaire corresponds
Interviewer: {question 1)An(options 1) fanswer 13
Interviewer: {question 2) 2)nnoptio \M\ (answer 2)
Interviewer: {question 3)Anfoptions \nMe:
19

---

Survey question
1.0
Response and
8
Abeis
Gez 110
Nep Ii5u
GPXx 355M
7au (PT Nee
1.30 MPr
Pytda
Lamm To
Pyttte 120
Liona tn
Naut cou
Littt sos
Uamaas 2 you
Canaus
GPTZ 110M
Neo 10SM
GPT2 15.MM
GF12 774M
GPT Roo 1.39
GTT2
1.5
MPT
Py
Utmo 2.78
Pytnis 178
Uhima 118
GPT Neot 209
U.aM 508
Llama 708
Rusponce antoppy
Survey sucstion
entragy
Survey quest
05
0.
GPI 125M
130M
GP2 355M
Pytha
Lima 28
GPT Seeck
1.00
Lt LAMA
Liama
U.S. Canses
MIT
sMFt chat 78
Kosa
Vizon
Lian Chat
Duily 128
Kaado 138
Lieiss Chat 138
x38
Neaer 208
Liena Chat U.S. Contin
(made)) Rel))
Alggmur
Altymart
02712
23.0
NED
1239
6772 3534
GP7 77M
1.30
150
MPT
Pytha
27
Liina
128
136
LUCHA 300
Liam 10s
MPI Instract 78
Chat
Kasis 78
Vicuns 78
Ukne chat 78
Dwly 120
1.0
vicuno 130
Liaaz 136
HealT 200
Ulems cher
oF1
Figure 14: Reproduction of the experiments in Sections 3 and 4 for the ATP surveys.
E Results for ATP, GAS, WVS, and ANES surveys
We reproduce the experiments of Sections 3 and using the ATP, and GAS/WVS used by Santurkar et al. 20223 and Durmus et al. [2023). where questions are presented individually of one another. We additionally reproduce the experiments of Section 5 using the 2016 ANES questionnaire considered by Argyle et al. 2023 where questions are presented in sequence. We do not consider OpenAl's models as the cost to reproduce the experiments via the OpenAI API exceeds our budget. We obtain very similar results to those of the ACS presented in the main text of the paper. The notebooks with all figures can be retrieved from our Github repository.
E.1 ATP surveys
We obtain the ATP survey questions and their corresponding human responses from the OpinionsQA We present all answer choices when querying the models, but exclude the answer choices
repository 10
corresponding to refusals from our analysis similarly to (Santurkar et al. [202]] When comparing the similarity of models' responses to different demographic subgroups, we use the demographic
subgroups and the alignment metric considered by [Santurkar al. 2003 For such metric, higher
values of alignment indicate that models' responses are more similar to the reference demographic
group. We find that all models are more "aligned" with the uniformly random baseline than with any of the demographic subgroups, see Figure
E.2 GAS and WVS surveys
We obtain the ATP survey questions and their corresponding human responses from the GlobalOpin-
ionsQA repository When comparing the similarity of models' responses to the population-level survey responses of different countries, we use the countries and the similarity metric considered by
Durmus al. 2023). We find that all models produce survey responses that are more similar to those of the uniformly random baseline than to those of any of the demographic subgroups, see Figure [5]
20

---

Survey question
suestion
Resperon entrewy
1.
0.8
03
0.4
GPT2 11:M
8.
GPT 12M
503M
GPT Neo
GPTZ 1
hybbia 18
2 78
Pymti
Lama
Ger Aoot
LEatt
Liama
G072 110M
0.0
Get Neo 125M
6PT2 355M
GPT2 7M
GPT Nio
GPTZ
1.28
Pythia 18
1.58
Lama 2
Pychia 128
Ulmm 138
Neat 208
(LaMA 508
Ulems 768
Response entropy
Survey question
Response entropy
0.5
0.5
0PT2 LsG
GPT Neo 125M
GPd 355M
G972 TMM
1.30
1.50
Pydhit
Uima 2788
syess 128
Loame 2 the
GPY zue
LLaMA y01
Liama 708
0.
Vicuna
Laaa Ghat
Guily 128
Koala 119
Vicuna 1
Uama Chat 139
AnarT ctal 200
Liams Chat 708
off
(madell Heft 0
Aligment Rett
Allament
solt
5912
350M
0P22 17M8
MDD
130
155
Pytso
278
Pylice
tlons
aPF Nost
LLaMA 300
Lioms 100
Koala
Vicen
Ulme 2 Cou
Deiy 120
Koais 236
Wore 1.39
Liena chat 1.3
Neatt Clat 268
Lion Chat 708
Figure 15: Reproduction of the experiments in Sections and for the GAS/WVS surveys.
Accuracy in discriminating model-generated data ANES 2016, interview prompting
Accuracy (86
100
98
96
Pygra 78
78
Koala 138
138
Liama 138
13
139
Chat 308
208
208
LA 658
Liama 108
Chat 708
GPT2
NEO GPT
GPF NEO OPT GPT NEO
MPT Instruc
MPT
Chat
LLaMA
Koala
Liama Liama
Liama
GPT Neox!
Char
LLa
Liama
Figure 16: The discriminator test performed on datasets generated using the 2016 ANES survey questionnaire (with choice randomization).
F,3 Relative alignment for ATP and GAS/WVS surveys
We consider the alignment measures proposed by Santurkar et al. 2023 and Durmus et al. (2023 ATP and GAS/VVS opinion surveys for the largest base insttrcct models considered. We find that, similarly to observations for the ACS, the alignment between models and given subpopulation is highly correlated with the entropy of the subpopulations' responses.
E.4
ANES survey
We present questions in the multiple-choice format described in Section 2, using the Interviewer: Me: prompt style described by Argyle et al. [2023]. We retrieve the 2016 ANES data from the official website 12 and process it such that it matches in form the questionnaire designed by Argyle et al. 22023. We find that the trained classifiers can discriminate between the -adde--geeerted data and the ANES data with very high accuracy (299%), Figure 16
Sequential sampling of responses
Motivated by recent findings of Argyle et al. 2023 we conducted an additional investigation where seek to fill entire ACS questionnaires in sequential manner, order to generate for each language model a synthetic dataset of responses. This data emulates in form the ACS dataset collected by the
21

---

Quastiore What person's
Information about person:
Information about this person:
SCX
AGER
HISPR
Male
Parus
Oven
Via
Female
Question: this person's age? Under years
Age 66 years and over
Answer:
person Hispanic,
Repeat to obtain a tabular dataset of responses
Medel's = Sample
Spanish origin?
PIA) P(S)
G. 68 years and over
No
SEx
HooPP
Models reeporee >
Sample
Answer:
Femral
Cvar
PVA) - P(G) 0.15
Model's s response = Sample
Male
411::00
P(A)= 0.55 PJB) 0.45
Paeusi
Vee..
Figure 17: Methodology and prompt template used to sequentially sample models' responses to entire survey questionnaires. We provide the answers to previous question context when prompting subsequent questions. The output is a tabular dataset of responses.
U.S. Census Bureau. We then study the extent to which such synthetic datasets resemble the ACS dataset.
F1
Methodology
We present survey questions in the same order as in the ACS questionnaire. When querying a model to answer survey question g, we include a summary of the q- previously sampled answers in context. We then sample from the model's output probability distribution over answers, and continue the next question. We illustrate this sequential process in Figure [17, refer to Appendix DD3 for results collected with different variations of how model's 's previous answers are integrated into the prompt. We find our results to be robust to these prompt variations.
13
For each language model we sample N=100,000 model-generated responses to the ACS. Due to the cost of querying OpenAl's models, we only survey GPT-4 and sample N 500 responses. As a result, we generate for each language model tabular dataset similar in form to the ACS data, with N rows corresponding to each filled questionnaire and 25 columns corresponding to each question.
F.2 The discriminator test
We investigate whether the model-generated datasets resemble the U.S. census data by constructing binary prediction task aiming to discriminate synthetic responses from census responses. Intuitively, if the two datasets were very dissimilar, then classifier would be able to achieve high accuracy. Formally, let F be class of binary prediction functions mapping each data point (i.e., row in the tabular dataset) (0, 1). then the accuracy of the best E F on the discriminator task provides lower bound on the total variation (TV) distance between the two empirical data distributions.
Hence, we train predictor f to discriminate between the model-generated data and the census data in order to obtain an empirical lower bound on the distance between the two datasets. Specifically, we concatenate each model-generated dataset a random sample of N individuals from the ACS census data, and introduce a binary label indicating whether each row of the concatenated dataset not. We then train XGBoost classifier in this binary prediction task. As an additional point of reference, we also consider the accuracy in discriminating between the census data of any given U.S. state and equally-sized sample of the ACS data of all other U.S. states.
was
We report mean test accuracy Figure 18. We consider 100 different random seeds. We find that the trained classifiers can differentiate between model-generated data and census data with very high accuracy (> 90%) in all cases. Therefore, the empirical distributions corresponding to the model- generated data and the census data have TV distance larger than 0.9. These stark results indicate that data generated by sequentially prompting language models with the ACS survey questionnaire bears little similarity with the data collected by surveying the U.S. population.
F.3 Contrast with silicon samples
Argyle et al. 22023 propose "silicon sampling". a methodology to produce synthetic survey re- spondents using LLMs by conditioning on actual survey respondents. They focus on a subset of
The maximum number of tokens in filled questionnaire was less than 1024 tokens in all cases, thus fitting entirely within the context window of all surveyed models.
22

---

Accuracy in discriminating model-generated data
With choice randomization Without choice randomization Discriminating between U.S. state and rest ACS
100
Was
90
Accuracy
80
70
60
GPT NEO 125M
50
110M
OPT2 355M
GPT2 774M
GPT RED 1.35
GPT2
GPT NED 2.78
1.58
Pythia
LLaMA 7B
Koala
Vicuna
Liama Liama
Chat 78
Pythia 128
Dolly 128
LLaMA
Koala 138
138
Vicuna 138
Liama 2 Char 138
Usma 13B
Neok? char 20g
GPT Neox 208
LiaMA 308
LLaMA 658
Uama 2 Chat 708
Uama 708
GPT4 0613
Figure 18: Accuracy of the discriminator test. For all language models, it is possible to discriminate with very high accuracy between the ACS census data and model-generater data, (x) before adjust- ment and (x) after adjustment. We contrast this against the accuracy value of discriminating between the ACS data of any given U.S. state and the rest of the ACS census data (-).
12 questions from the 2016 American National Election Studies (ANES) survey. For every human respondent, they construct a corresponding "silicon individual" by querying GPT-3 to predict the ANES respondent's s answer each survey question given the respondent's answers to all other questions. Their results indicate that, for the 2016 ANES survey, GPT-3 can be a fairly calibrated predictor of individual's answer some survey question conditioned the respondent's answers to all other survey questions [1]
However, Argyle et al. [2023] emphasize that important insights can be gained by emulating the
survey responses of human populations "prior to or in the absence of human data". this work have considered precisely the setting where models' responses
obtained in the absence of
To investigate how our findings transfer to the ANES, we reproduce the experiments of Section using the 2016 ANES survey questionnaire considered by (Agyle al. [2023] and their "interview-style" prompt. We apply the discriminator test, and find that the trained classifiers discriminate between the model-generated data and the ANES data with accuracy 99 % (see Appendix E), indicating that models' responses are markedly different to those in the ANES data.
human data [5]
Thus, the fact that models may perform reasonably well at feature imputation tasks (e.g.. predicting an individual's ss answer to some question given their answers to all other questions) does not imply that models can generate synthetic respondents that resemble the responses obtained by surveying human populations. This suggests caution when using LLMs to emulate human populations at present time, in particular in the absence of human data.
2023 and Sanders et al. 2023 study imputation tasks similar to those of le et al. 2023 LLe et al.
and find that LLMs are not calibrated predictors for variety of such tasks.
"Conceptually, to generate each synthetic individual, we start with blank survey questionnaire and prompt the LLM to sequentially fill the entire questionnaire. Whereas we only prompt LLMs with their own responses
(i.e., to previous survey questions), Argyle et al. 2023 prompt the model only with actual human responses from the 2016 ANES data.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction directly refer the experiments and results detailed in the paper's 's main body.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, see Section 2 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc///
blic/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
httppsscccpbbbc/pppeeecccc guwdes/dde/eeee Coodeeooddmsssseeee /ttc: wwwssnoiccciyiiicccocooociiicipi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Appendix and the code release.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The main figures contain exact measures. We conduct significance tests Section B.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, see Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We confirm that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification; We argue throughout the paper that current evaluation practices might result in misleading claims regarding what subgroups current models best represent.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any models. We release models' survey responses, which pose no risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit the authors of the models considered, well as the sources of the surveys considered.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswii thcode..thcoe....m com/ datasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "1mxAUFLdv1", "reviewer": "Reviewer_SkTL", "summary": "This paper conducts experiments to verify the alignment between human and LLM responses to the ACS survey. Particularly, the paper questions existing literature suggesting that LLMs can be used as proxies for measuring responses to survey questions, suggesting instead that LLM choices are biased by the ordering of the questions, and when order choice is randomized, models tend to present uniformly random survey responses, thereby closely modeling the behavioural characteristics of sub-groups whose aggregate statistics are close to a gaussian. The paper suggests that using LLMs as human-proxies for multiple choice surveys is a questionable strategy.", "questions": "1. How was the subset of 25 questions selected from the ACS? \n2. Do the authors have any comments on the training data and its influence on survey responses, beyond the frequency of appearance of certain letters in English as noted in C?\n3. Does skewed randomization (as opposed to simple randomization) of the responses, present a similar skew in the model response distributions?\n4. Instruction-tuned models show higher variance in entropies. What could be causing this?\n5. What form of instruction tuning was tried? \n6. Were the effects of question-ordering investigated?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors test 42 different LLMs in their experiment, testing base, instruction-tuned, and RLHF-tuned models. This is comprehensive and substantive, and a lot of work. The results they find agree across model size and type barring one outlier in instruction-tuned models over one survey.\n2. The authors test the use of randomized choice order and the original choice order, finding that randomizing the response order results in a uniform distribution of responses.\n3. The authors investigate the effect of using instruction-tuning to train models.\n4. The authors also test surveys besides the ACS, and find that the results persist.\n5. The authors interpret findings in earlier papers and provide explanations for why the LLM responses more closely resembles responses from certain demographic sub-groups, i.e that these distributions are Gaussian\n6. The paper is well-written and presents a simple and elegant experiment, and clear and consistent takeaways.", "weaknesses": "1. Order choice bias is a well-known phenomenon in LLMs (Lu, Yao, et al. \"Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.\" arXiv preprint arXiv:2104.08786 (2021)). Therefore the big finding in this paper is that LLM responses model a Gaussian when order choices are randomized using a gaussian distribution.\n2. Please use a different color for the sub-group dots in fig. 5 -- it is confusing because the same color is used for survey responses in earlier figures.\n3. Takeaways are harder to gauge from Fig. 3. Please use simpler aggregate statistics like means, confidence intervals, variance and save this figure for the appendix.\n4. Please plot the log linear scale as a dashed line in Fig. 2 for easy comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We will implement the suggested changes to improve the figures. Let us address your questions in the following.\n\n**Selection of questions.** We chose 25 representative questions to achieve diversity over topics (e.g., educational attainment, healthcare status, employment status, etc.), while keeping figures readable. In the rebuttal pdf we include additional results for all ACS survey questions for comparison. We provide the A-bias and response entropy of models with publicly available weights. They show identical trends.\n\n**The role of training data.** Regarding the pre-training data, we find that all base models exhibit similar response distributions, despite being trained on substantially different pre-training data. While we find little difference across base models, beyond the ordering effects identified in Appendix C for smaller language models, we do observe substantial differences between the response distributions of different fine-tuned models. This suggests that fine-tuning data can have a larger effect on models’ distributions. This is a positive result for future work seeking to fine-tune models to alter their survey responses (e.g., emulate those of certain populations).\n\n**Why do instruction-tuned models have higher variance in entropy?** We generally find that, compared to base models, instruction-tuned models tend to have higher confidence in their responses for at least some of the survey questions. This causes instruction-tuned models to have higher variance in their response entropy compared to base models, as any deviation from balanced responses is more amplified. Note that we used publicly available instruction-tuned models, we did not perform instruction tuning ourselves. For some models (e.g., the Llama models), these instruction-tuning datasets are not publicly available.\n\n**Effects of question-ordering** We follow the predominant methodology of asking questions independent of each other. Therefore, there are no question-ordering effects. If questions were to be asked in sequence, putting the answer to previous questions in context, then we would expect to observe substantial question-ordering effects. But this was not the focus of our work.\n\n**Skewed randomization** Yes, for models that exhibit choice ordering biases, skewed randomization would change the response distribution. This is because we would no longer uniformly average across each of the possible choice orderings, but perform some weighted average. However, uniform is the only principled approach here to adjust for it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wh2vgCB71W", "reviewer": "Reviewer_gFgE", "summary": "This paper critically examines possible pitfalls of using the responses of LLMs to survey queries to study the model alignment. They found substantial bias, e.g., with respect to the order of response option,", "questions": "-", "limitations": "The limitations are mostly described well in the paper. For an exception, see weaknesses.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper examines a very important methodological topic that has gained significant attention also beyond computer science: measuring the values and opinions, in which LLMs are rooted. As such, the topic is very relevant, interesting, timely, and certainly fitting for the conference.\n\nThe paper is well written and well motivated. The provided material (i.e., code and documentation) are exemplary. Results are presented in a clear and concise way.", "weaknesses": "The paper is motivated by surveys on the \"demographics, political opinions, or values best represented by current language model\". For this, the paper mostly relies on the ACS dataset. However, this questionnaire mostly covers demographic information, for which naturally the LLM cannot have a \"correct\" answer. It would be criticial to see for which type of question the high entropy responses actually hold. From my own experience, I would not expect at all that similar uniform distribution also would occur for (political) opinion or value-based questions.\n\nMinor note: I would recommend to add \"forward mentioning\" in Section 2 that other datasets will also be covered in Section 5\n\nAlthough a lot of LLMs have been included in the study, only the large-scale models of OpenAI have been studied. To see if the observations hold also for other, similary large models, including also other commercial providers (i.e., Anthropic or Google) would be nice. This is not required for the key results of the paper, however.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive assessment and the feedback. Please note that we discuss in Appendix E how our findings for the ACS transfer to opinion surveys. We agree that our observations regarding models’ survey responses may be partially attributable to survey questions not having a “correct” answer. This is in stark contrast with the multiple-choice questions that are typically used to evaluate LLMs (e.g., MMLU), and reveals interesting new insights for model evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mxRspO1Efl", "reviewer": "Reviewer_gDVK", "summary": "This paper prompts LLMs with 25 multiple choice questions (on basic demographic information, education attainment, healthcare coverage, disability status, family status, veteran status, employment status, and income) from the 2019 ACS. The authors use eight kinds of prompts which vary in additional context, instructions, and asking in the second person. However, each time, the next-token probabilities are used to determine the immediate reply by the LLM to the multiple choice question. To evaluate the responses and the differences between LLM and human generated responses, the authors compute the normalized entropy and use KL divergence. They find that \"smaller\" LLMs are vulnerable to ordering and labeling biases, and that after correcting for these through randomized answer ordering, the LLMs trend towards uniform distributions in their responses (~high entropy). Instruction-tuning seems to increase the variance in the entropy measure for LLMs, but nonetheless the entropy remains higher overall compared to the human generated responses. The authors state that the main takeaway from their paper questions the popular methodology of using survey responses from LLMs using multiple choice questions. They challenge prior work and give the explanation that models consistently appear to better represent subgroups whose aggregate statistics are closest to uniform.", "questions": "Minor comments:\n- There is a discrepancy between the 42 models mentioned in the abstract vs 39 in the main text.\n- Typo line 561: rompt instead of prompt (in title)", "limitations": "The paper does not explicitly state the limitations of the experiments but carefully reassesses its findings in the conclusion. The checklist points to Section 2 where, for example, the authors point to the prompt ablations.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper has a clear goal to challenge previous work on survey-derived alignement measures by offering the explanation that models consistently appear to better represent subgroups whose aggregate statistics are closest to uniform.\n- The paper shows that LLMs should not be used out-of-the-box to replace human responses in census data.\n- The paper is generally well-written and clear which makes it easy to follow.", "weaknesses": "While I find the paper enjoyable to read and it challenges important earlier findings on LLMs, I am not convinced that the current experiment fully supports the claims:\n- The 2019 ACS uses stratified sampling. Therefore, I believe that the variance is already being increases to obtain a representative sample. Since the additional context given to the LLM is very limited + the draws are independent, it seems like a very hard task to generate a matching distribution by the LLM. This is less important when assessing, for example, the political view of an LLM.\n- First-token probabilities may be a biased measure to obtain the replies (see Wang et al., 2024).\n- Additionally to the first-token probabilities, more advanced prompting techniques, such as Chain-of-Thought (Wei et al., 2022a), could improve the coherence and dependencies in the responses, especially for the sequential generation.\n- I believe that the sensitivity to ordering and labeling biases is known (Wei et al. (2022b) and Wei et al. (2023)).\n\nOverall, the authors show that independent draws from an LLM with limited context generates a uniform distribution, which questions earlier findings made used by such as methodology. While I agree with the authors on that statement, I believe that their experiment adds little value to further support their claim on \"better represent subgroups whose aggregate statistics are closest to uniform.\" I believe additional, more fine-grained analysis is required for this.\n\nReferences\n-------------\nWei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022a). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.\n\nWei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., ... & Fedus, W. (2022b). Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.\n\nWei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., ... & Ma, T. (2023). Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846.\n\nWang, X., Ma, B., Hu, C., Weber-Genzel, L., Röttger, P., Kreuter, F., ... & Plank, B. (2024). \"My Answer is C\": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models. arXiv preprint arXiv:2402.14499.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback. We hope to address your concerns and clarify some misunderstanding below.\n\n> their experiment adds little value to further support their claim on \"better represent subgroups whose aggregate statistics are closest to uniform.\"\n\nWe believe this to be a misunderstanding. We do not claim that models better represent subgroups whose aggregate statistics are closest to uniform. Let us clarify.\n\nOur experiments show that, using the de-facto standard multiple-choice methodology, models strongly trend towards uniformly random responses (Figures 4 and 5). This results in a very strong correlation between subgroup entropy and alignment (Figure 6). Such correlation consistently holds across surveys, subgroups, and models. Therefore, models **appear to** better represent subgroups whose aggregate statistics are closest to uniform. \n\nOur experiments explain the findings of earlier work (i.e., Santurkar et al. 2023, see the discussion in Section 5, “Beyond the ACS”), and why these findings may be misleading. Models appear to better represent younger demographics not because of the pre-training data, but because younger demographics happen to have more uniform responses for the ACS. We are not claiming that language models actually better represent certain populations intrinsically. We are arguing that the de-facto standard methodology to survey language models has strong limitations, and it can potentially lead to misleading insights. Our claims are not about whether language models better represent certain populations, but rather about the limitations of the dominant survey methodology itself.\n\n>The 2019 ACS uses stratified sampling. Therefore, I believe that the variance is already being increased to obtain a representative sample. Since the additional context given to the LLM is very limited + the draws are independent, it seems like a very hard task to generate a matching distribution by the LLM.\n\nIf we understand correctly, your point is that the aggregate responses of the U.S. census (the reference population used in our work) appear to be more entropic than they actually are, due to the U.S. census not representing households uniformly at random. Our claim is that, when using the dominant multiple-choice methodology to survey models, model responses are *qualitatively* different from those of the U.S. census. Model responses strongly trend towards uniformly random, irrespective of the survey question being asked. The responses of the U.S. census do not – they are heterogeneous. If stratified sampling were to have a small effect on the U.S. census responses, it is still the case that model responses (e.g., the blue dots in Figure 4a) look nothing like those of the U.S. population (green dots).\n\nPlease note that there are no “draws” – for each survey question, we extract the models’ survey response analytically by extracting its next-token probabilities over each of the answer choices (e.g., “A”, “B”, …). This is the standard methodology introduced by Santurkar et al. for surveying language models using multiple-choice questionnaires. Our contribution is to shed light on the properties of these output distributions.\n\n> First-token probabilities may be a biassed measure to obtain the replies [...] more advanced prompting techniques, such as Chain-of-Thought (Wei et al., 2022a), could improve the coherence and dependencies in the responses, especially for the sequential generation.\n\nWe agree with your points. They support the overall conclusion of our work that the current multiple-choice methodology used to survey language models has strong limitations, and we should move towards a more holistic evaluation of LLMs (e.g., open ended survey responses rather than multiple choice) in order to elicit more faithful representations of the population a language model might represent.\n\n> I believe that the sensitivity to ordering and labelling biases is known (Wei et al. (2022b) and Wei et al. (2023)).\n\nOrdering bias has been observed in various works. We cite Robinson and Wingate (2023a).  We are happy to include more references in the final version. Our work is different from prior work in studying the effects of ordering biases for models’ survey responses. We show that models’ survey responses can substantially change after adjusting for their ordering biases, leading to fundamentally different insights regarding the populations that models’ best represent.\n\nWe hope the additional explanations helped address your concern. We are happy to answer further questions that you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LpGR95BmTM", "reviewer": "Reviewer_Pjc3", "summary": "The paper focuses on evaluating 42 different language models on the American Community Survey, highlighting how model responses are governed by ordering and label biases and in general the fact that any demographic correlation with specific subgroups is actually due to the fact that those subgroups aggregated statistics are closest to uniform.", "questions": "Do humans as well have A-bias? \n\nHave you considered exploring whether RLHF would introduce a slight steer for the model towards certain sub-groups (reflected in the way people give feedbacks) which was not present in the base model?", "limitations": "I think authors have addressed the main limitation of this work (which would be the focus on US survey) by examining other surveys.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The study in the paper is well-designed and discussed, considering a very large variety of LLMs and in particular by considering both base and instruct-tuned models. The presented results are very relevant for any study that plans to either examine LLMs for understanding underlying characteristics and, most importantly, for researchers who plan to use LLMs as a proxy to study human sub-groups. The strong A-bias showed in the paper is a very important take-away.", "weaknesses": "There are two main weaknesses that I have encountered while reading the paper, namely:\n1) It is not clear what the authors would expect the models should do based on their training data. It is true that we don't know exactly on what these models have been trained, but if we consider for instance the Common Crawl as the main corpus, would we expect that models trained on it would somehow produce answers correlating with certain sub-groups? I think a larger discussion in this paper on what we should reasonably expect models answers to be is needed.\n\n2) given the emphasis on sub-groups, I would have expected the authors to explore impersonation of LLMs as a way of seeing whether that would steer responses to surveys in the direction of what we would expect those sub-groups typical responses. This would clarify whether LLMs have the ability of producing answers relevant to specific sub-groups, if instructed to do so.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback.\n\n**What would you expect the models’ answers to be?** It is unclear what to expect models’ responses to be. Prior work has hypothesised that models may trend towards certain demographics; for example, younger demographics, which tend to be more present on the internet. Another candidate could be models trending towards the responses of particularly populous U.S. states, simply because they may produce larger volumes of data. However, we observe that base models' ACS responses are *qualitatively* different from those of human populations. Because of these qualitative differences, we argue that the quantitative analysis of prior work may be misleading (Section 5). This is not to say that base models do not have biases, or do not represent certain populations better than others. Rather, our findings signal the need to move beyond multiple-choice prompting towards a more holistic evaluation of LLMs (e.g., open ended survey responses) to elicit more faithful representations of the population a language model might represent.\n\n**Impersonation** We focus on evaluating whether the survey responses of LLMs are representative of certain U.S. demographic subgroups. In this setting, it is standard to prompt the model without any added context. Assessing whether models have the ability of producing answers relevant to specific sub-groups if instructed to do so is beyond the scope of our work. \n\n**Do humans have A bias?** There is evidence for ordering bias in humans in the context of opinion surveys, and a tendency not to pick extreme values. However, in the context of the ACS demographic survey, it is well-understood that ordering effects play a very minor role in the distribution of responses collected by the U.S. census. The recent work of Tjuatja et al., 2023 finds that the response biases of language models (e.g., A-bias) are generally not human-like.\n\nTjuatja, Lindia, et al. \"Do LLMs exhibit human-like response biases? A case study in survey design.\" arXiv preprint arXiv:2311.04076 (2023).\n\n**Does RLHF introduce steering?** We evaluate models that have undergone RLHF, particularly the Llama 2 Instruct models, text-davinci-003, GPT-4, and GPT-4 Turbo. However, these models have undergone both standard supervised fine-tuning (i.e., instruction tuning) as well as RLHF. Overall, we observe that the responses of fine-tuned models vary more across questions (e.g., are not as balanced as those of base models). We, however, find no evidence that the responses of RLHF models better represent those of human populations. This is not to say that RLHF introduces no steer, but rather that the multiple-choice survey methodology that has recently gained traction in the community may not be appropriate to study this phenomena.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their feedback. 

We hope to have addressed your concerns, and we are happy to answer any further questions you may have.

Thank you,
Authors

Meta-review: This paper conducted large scale experiments to understand LLM's answer to survey questions that's designed for human. It found out that the models suffers strong positiional bias (which is known to the community in general) and models' answer would represent aggregated uniform if the positional bias is properly controlled.

The paper studies LLMs from a novel perspective as a survey participant. The conclusion may not surprise a LLM researcher too much, but have good value to other researchers that could be convinced  by the model's human-like behavior and try to use them in studies about social topics, which could be dangerously misleading.

The paper is well received by the reviewers and is in good writing quality.
Decision: Accept (oral)