CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations
Authors: Myra Cheng
Tiziano Piccardi
Diyi Yang
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
caricatures
language model simulations
survey replication
TL;DR: We develop a framework to characterize LLM simulations and measure their susceptibility to caricature.

Abstract: Recent work has aimed to capture nuances of human behavior by using LLMs to simulate responses from particular demographics in settings like social science experiments and public opinion surveys. However, there are currently no established ways to discuss or evaluate the quality of such LLM simulations. Moreover, there is growing concern that these LLM simulations are flattened caricatures of the personas that they aim to simulate, failing to capture the multidimensionality of people and perpetuating stereotypes. To bridge these gaps, we present CoMPosT, a framework to characterize LLM simulations using four dimensions: Context, Model, Persona, and Topic. We use this framework to measure open-ended LLM simulations’ susceptibility to caricature, defined via two criteria: individuation and exaggeration. We evaluate the level of caricature in scenarios from existing work on LLM simulations. We find that for GPT-4, simulations of certain demographics (political and marginalized groups) and topics (general, uncontroversial) are highly susceptible to caricature.

CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations
Myra Cheng, Tiziano Piecardi, Diyi Yang Stanford University
Department of Computer Science
[myra, piccardi, diyiy]@cs. stanford. edu
Abstract
The CoMPosT Framework
Context
Where and when does the simulated
Recent work has aimed to capture nuances of human behavior by using LLMs to simulate responses from particular demographics set- tings like social science experiments and public opinion surveys, However, there are currently no established ways to discuss or evaluate quality of such LLM simulations. Moreover, there is growing concern that these simulations are flattened caricatures of the personas that they aim to simulate, failing to capture the multidimensionality of people and perpetuating stereotypes. To bridge these gaps, we present CoMPosT, a framework to characterize L.LM simulations using four dimensions: Context, Model, Persona, and Topic. We use this frame- work to measure open-ended LLM simulations' susceptibility to caricature, defined via two cri- teria: individuation and exaggeration. We eval- uate the level of caricature in scenarios from existing work on LLM simulations. We find that for GPT-4, simulations of certain demo- graphics (political and marginalized groups) and topies (general, uncontroversial) are highly susceptible to caricature.
scenario occur?
Model
What LLM used?
Persona
Whose opinion/action is simulated?
Topic
What is the simulation about?
Table I: Dimensions
CoMPosT framework. We these dimensions to characterize LLM simulations and measure their susceptibility to caricature.
havior that they provide draws upon the organic material (human data) used to train LLMs.
Such applications currently have little no mechanisms for comprehensive evaluation or care- ful deployment. Evaluation of such simulations has been limited to either (1) replicating existing results or (2) assessing believability. Both paradigms have drawbacks: (1) Replication limits us to only re- producing already-knowe behavior, and does not support the validation evaluation of any simula- tion behaviors beyond those highly correlated with existing results from human studies. Also, existing results are typically quantified as categorical dis- tributions across multiple-choice answers, so there is no way to directly evaluate open-ended genera- tions, and such results may have been "memorized" from the LLMs training data (Lewis al., 2021; Elangovan al., 2021). Thus, replication does not facilitate new insight into human behavior. Further- more, while (2) Believability useful in certain settings, such as entertainment (Bates et al., 1994), susceptible to the biases and fallacies hu- man judgment: psychology literature shows that people are more likely to believe stereotypes about groups with which they have less personal experi- ence (Plous, 2003; Bar- Tal et al., 2013), and beliefs are easily influenced (Blair et al., 2001; Jussim et al., 2016). Recent work has also demonstrated that human judgment insufficient for assessing AI (Schneider et al., 2020; Peng et al., 2022; Vo- drahalli et al., 2022; Veselovsky et al., 2023).
1 Introduction
Large language models (LLMs) have shown promise in capturing social nuances and human behavior. For instance, researchers have repro- duced results from social science experiments and public opinion surveys using LLMs (Argyle al., 2023; Aber et al., 2023; Santurkar et al., 2023, inter alia). More broadly, interest in L.M simulation is rapidly growing, and the possibility of using LLMs to simulate human behaviors has far-reaching ap- plications in fields like education (Markel et al., 2023), product design (Park al., 2022, 2023a), psychology (Binz and Schulz, 2023), healtheare (Weizenbaum, 1966; Bassett, 2019), skill training (Hollan al., 1984; Jones al., 1999), and law (Hamilton, 2023). These simulations are a sort of digital compost- any new insight into human be-

---

Paper
Context
Model
Topic
Park et al. (2022)
discussion guided by community (0)
GPT-3 davinci
users of verious Reddltt communities
topics various Recidit communilies
Angyle (2022)
public opinion survey
GPT-3 davinci
ideology, partisanship, race, gender, income, ege
political groups
Horton (2023)
economics experiments
GP1-3.5
politica/scoooomicall idcologies
social preferences, falmmes market scenarios, status quo bias, etc.
(MC)
Hamilton (2023)
Supreme Court cases (MC)
GPT-2
Supreme Court jourtjustices
Supreme Court
Jlang al. [2022)
posting Twitter (O)
GPT-2, GPT-2,GPT-3 GPT-3
Republiicn Demacrat Users
controversial political figures groups
Mapping Existing Work Using CoMPOST. Existing work on LLM simulations can be compared using our framework. MC and O denote multiple-choice and open-response respectively. More examples are in Table A6.
Figure I:
Toward clearer documentation of this emerging line of work, first present descriptive frame- work that taxonomizes LLM simulations using four dimensions:
Simulation Topic: Computers and Electronics
Generated nonbinory person responses:
"I' interesced in getting eccommendation for any cool devices might partiodaaty appeal nonbinary individuals
Context, Model, Persona, and Topic (CoMPosT) (Table 1). Our framework facilitates comparison across existing work on LLM simula- tions (Figure 1).
surviasblltyy noreesntiaioon. @ gaming//tremmmn mg space sant to create an incustzee and comfortable gaming Stteetttiiingggg myself, as well as others in the LGBTO commurity.
nonibnatry Indiiiuull.
Generated person responses:
"I recently upgeaded my desktop PC with a new graphics card and SSD and I'm really impressed with the performance boost from these upgrades." "a's interesting
Next, we introduce a new evaluation metric that addresses growing concerns of modal responses and essentializing narratives in LLM outputs (San- turkar et al., 2023; Cheng et al., 2023b; Shumailov et al., 2023). Our metric focuses on simulation's 's susceptibility to caricature: exaggerated narra- tive of the persona (the demographic that aim to simulate) rather than meaningful response to the topic (Figure 2). Caricatured simulations are concerming because they a) fail to capture the real nuances of human behavior, thus limiting the use- fulness of simulations and b) perpetuate mislead- ing descriptions, stereotypes, and essentializing narratives about demographic groups. We define caricature using two criteria: individuation and ex- aggeration (Section 3). To measure individuation, we assess whether outputs from the given simula- tion can be differentiated from the default response for the topic. To measure exaggeration, we use a "contextualized semantic axis" whose two poles are the defining characteristics of the persona and topic dimensions respectively.
how rapidly tec technology over past decades, From the first personal computers to smartphones, and now we have Al lot making significant impacts...
evalved
Figure 2: Examples of GPT-4 generated responses for simulations with the topic Computers and Electronics. For the simulation of a nonbinary person's 's response, the generations are focused on eventity-relatee issues, while the simulation of person's 's response is topical. The former constructs a homogenous narrative that de- fines nonbinary people only by LGBTQ+ activism. We provide more qualitative examples in Appendix A.
caricatures (Section 4), and (3) experiments dif- ferent contexts (Section 5) toward an analysis of the dimensions that are most susceptible to caricature (Section 6). We conclude with actionable recom- mendations and considerations for those interested in LLM simulation (Section 7)."
CoMPosT: Taxonomizing Simulations
We introduce CoMPosT, descriptive framework with four dimensions to characterize LLM simu- lations: Context, Model, Persona, and Topic. In- spired by existing descriptive frameworks for AI fairness (Tubella et al., 2023), our framework pro- vides shared language to understand and articu- late similarities and diffcrences across LLM simu- lations. Context, persona, and topic specified in the prompt, while model is determined externally. We map existing work on LLM simulations using these dimensions (Figure and Table A4).
We evaluate the level of caricature scenar- ios from existing work on LLM simulations. We find that for GPT-4, simulations of certain demo- graphics (political and marginalized race/ethnicity groups) and topics (general, uncontroversial) are more susceptible to caricature. Our main contri- butions are: (1) CoMPosT, a framework for char- acterizing the dimensions of LLM simulations of human behavior (Section 2), (2) a novel method that relies on the persona and topic dimensions of
'The code and data is available at https//giihbb.com//yyycchoggg// caricature.
CoMPosT to measure simulations' susceptibility to

---

Context The output from simulation is neces- sarily affected by the context of where and when the imagined situation takes place. For instance, a formal interview response varies drastically from a user's comment on Twitter or Reddit. The context includes relevant structural factors and embeds in- formation about the norms of the situation. Each context has its own unique set of norms, which may be explicit, in the case of online communities with written-down rules, or implicit (Chandrasekha- ran al., 2018; Ziems al., 2023). Context also includes the phrasing of the prompt itself, which affects the output-LLMs LLMs are notoriously sensitive to prompt engineering (Zamfirescu-Peree 2023). The desired granularity of outcome also arises from the phrasing of the prompt and thus is embedded in the context: simulation scenario may be worded ask for a choice between binary- or multiple-choice options (such as in many so- cial science experiments and public opinion polls) or for an open-ended output. Previous work on evaluating simulations has largely focused on us- ing LLMs to reproduce scenarios in which humans are asked to choose among fixed number of op- tions without specifying the context, it is more challenging to evaluate the quality of open-ended responses. We bridge this gap by offering metric for the latter.
possible simulation scenarios, many of which have not yet been well-explored. Across existing work, we find that researchers typically use a state-of-the- art model and choose a particular context while varying the more salient dimensions of persona and topic. We denote the simulation scenario as Sp.t.c. as it is associated with prompt containing persona p. topic t, and context (Table A4). Our evaluation methodology and results uses these di- mensions of CoMPosT understand how different simulations may result in caricatures. Specifically, we explore how the relationship between the di- mensions of persona and topic help characterize the extent of caricature in simulations.
et al..
Background: Caricature
3.1 Definition of Caricature
Building upon Lynch (1927)'s discussion of how caricatures are misrepresentations that have some sense of truth to the subject by reflecting "salient peculiarities," Perkins (1975) define caricature "a symbol that exaggerates mcasurements relative to individuating norms." " In doing, Perkins iden- tffies two key characteristics of caricature: exagger- ation and individuation. A caricature a depiction that not only exaggerates particular features of the subject but also exaggerates in a manner that mean- ingfully differentiates the subject from others. The exaggeration is done in such way that it individ- uates by remaining faithful to the properties that distinguish the subject from others (thus, com- plete distortion is not caricature).
Model The LLM used to produce the simula- tion affects the quality and other characteristics of simulation. Differences may arise from varia- tions in models' training data and processes, includ- ing instruction-tuning, fine-tuning, and/or value alignment efforts (Solaiman and Dennison, 2021; Ouyang et al., 2022: Bakker et al., 2022).
This inspires our definition of caricature the LLM simulation context: given that a subject has some defining characteristics, a caricature exagger- ates these characteristics in a way that amplifies the ability to identify (i.e., individuate) the subject from the caricature. In CoMPosT terms, simula- tion's level of caricature is the degree which it exaggerates the individuating characteristies that are emblematic of the persona beyond meaning- ful, topical response to the scenario.
Persona The persona refers to the entity whose opinions/actions the simulation aims to study and replicate. This persona may include attributes that are relatively static (e.g., race/ethnicity), slowly change over time (e.g., age), or temporary and of- the-moment (e.g., emotional status) (Yang, 2019). It may also refer to a specific individual.
Note that in some cases, it may be acceptable for the persona to influence the simulation, i.e., individ- uation alone does not entail caricature. For exam- plc, opinions some topics differ greatly based on demographic. caricature occurs when the simula- tion both individuates and exaggerates the defining characteristics of the imagined generic responses of that persona. Previous work has documented how such imagined personas reflect stereotypes,
The topic of the simulation may be a partic- ular subject of discussion, question, or other event to which a response is desired. Topics vary in speci- ficity, from very general (such single word that captures broad conversation category) to very specific (such as a specific situational question in a psychology experiment).
Topic
These four dimensions capture a wide range of

---

both inside and outside the LLM context (Marsden and Haag, 2016; Cheng al., 2023b). Thus, carica- tures not only fail to capture the diversity of human behavior but also may rely on stereotypes.
and struggles" (Mohanty, 1988; Aneja, 1993; Ku- mar al., 2019). This literature reveals that even when the caricatures are not overtly negative, such one-dimensional depictions are still damaging and harmful. Overlooking diversity within demo- graphic groups has been connected to real-world harms including misprediction and medical mis- diagnosis (McCracken et al., 2007; Borrell et al., 2021; Read et al., 2021; Wang et al., 2022).
3.2 Implications of Caricature
Stereotypes We rely on psychology literature that broadly defines stereotypes as generalizations about the characteristics of a social category, such as associating a social category with a particular role or trait (Heilman, 2001; Fiske et al., 2002; Cao et al., 2022; Kambhatla et al., 2022). The norma- tive value of stereotypes is context-dependent; for instance, stereotypes can help foster sense of au- thenticity (Marsden and Haag, 2016), while even seemingly-positicnn stereotypes can have harmful implications (Fiske et al., 2002; Czopp et al., 201 Stereotypes and caricature, while closely related, are distinct in that caricature may be specific de- piction of stereotype: scholars have documented caricatures of stereotypes in various domains and how they facilitate misogyny, racism, and other forms of oppression (Slavney, 1984; Brown, 2010;
4 Caricature Detection Method
The two key aspects of caricature are individua- tion and exaggeration. To measure the amount of caricature in given simulation Sp.t.c> our method has three steps, each of which rely on the persona and topic dimensions of CoMPosT (Figure 3): (I) defining defaults, (2) measuring individuation, and (3) measuring exaggeration. Note that this frame- work is sequential, as (3) is only necessary if the simulations can be individuated. Otherwise, we can halt after step (2) since individuation is a necessary criterion for caricature.
Gottschalk and Greenberg, 2011; 1; Takayama, 2017; Bow, 2019). Such caricatures have historically been used in literature and media to justify slavery, imperialism, and war (Demm, 1993; Kriz, 2008), But even when caricatures do not contain stereo- types, they have concerning implications of ho- mogenous narratives.
4.1 Defining Defaults
A simulation Sp.tcc is caricature if it has more of the defining characteristics associated with the persona p and less of the defining characteristics associated with the topic We first identify these defining characteristics using the default-persen simulation S S_,_ (simulation that does not men- tion any specific persona) and the default-topic simulation Sp_. (simulation that does not men- tion any specific topic). Note that this default does not reflect a universal default but rather a model- and context-specific default: previous work has shown that LLMs implicitly default to a particu- lar set of perspectives (Western, white, masculine, etc.) (Santy et al., 2023). We use these defaults as a comparison point for caricature to isolate the defining characteristics of particular dimensions. For the default- persona simulation, we use a prompt where in lieu of specific persona, we use an unmarked default term like "person" or "user." Thus, the outputs reflect the topic and context rather than any particular persona. (Again, such words are not true defaults and are inextricably tied to societal norms: in English, the word "person" is often conflated with "man," a phenomenon also present in web data and language models (Bailey et al., 2022; Wolfe and Caliskan, 2022).)
Misleading Homogeneity Caricatures foster ho- mogenous narratives that do not reflect the full di- versity of the personas they aim simulate, which limits the utility the simulation. This concern builds upon previous work: Grudin (2006) discuss how personas can result in systematic errors in un- derstanding human behavior, Cheng et al. (2023b) characterize the harms of LLMs reflecting essen- tializing narratives about demographic groups, and Santurkar al. (2023) show that certain instruction- tuned LLMSs tend to generate modal responses. Oth- ers have explored the linguistic nuances within complex social categories and the ramifications of ignoring heterogeneity within social groups (Bam- man et al., 2014; Hanna et al., 2020; Cheng et al., 2023a).
These harms of caricature are also articulated by feminist scholars who have discussed how "women in the Two-Thirds World...are constructed as one-dimensional, oppressed caricatures without an understanding of their real experiences, agency,
For the default-topic simulation, we use a

---

prompt where no topic is specified. Thus, the oulputs from these prompts reflect the particular persona rather than a response to the topic. It is well-documented that this type of prompt results in outputs that reflect stereotypes, both when asked to humans and to LLMs (Kambhatla et al., 2022; Cheng et al., 2023b). Note that even if we expect the output to change based on the persona, the response should still be distinct and not defined by the same distinguishing characteristics as the default-topic simulation for a given persona.
acceptable for a simulation to have many topic- related words.) Instead, we measure the extent
to which the defining characteristics of the per- sona are exaggerated the target simulation via persona-topic semantic axes.
Specifically, we construct contextualized seman- tic axes, method introduced by Lucy et al. (2022), to capture whether Sp,tec is more similar to the defining characteristics of the persona p or the topic Our semantic axes have two poles, Pp and Pt. reflecting the persona p and the topic t. To con- struct the set of seed words, we use the Fightin' Words method (Monroe et al., 2008) to identify the words that statistically distinguish Sp_. from S _.c.-- We first compute the weighted log-odds ratios of the words between S S_t.c vs. Sp_.. To control for variance in words' frequencies, we the following prior distribution: other texts where the personas/topies are either p/t respectively or (i.e., the default). Then, we take the words that are statistically significant (have 2-score > 1.96) as the sets of seed words W, and We for the corre- sponding poles Pp and Pt (Table A3).
4.2 Measuring Individuation
We operationalize the desiderata of individuation using differentiability from default: examine whether the given simulation Sp.t.c is differentiable from the default-persen persona simulation S S_te.- If not, then SpA,c cannot be a caricature.
We use binary classifier (specifically, a random forest classifier implemented using seikit-lear) to differentiate between outputs from the target simu- lation of interest Sp...c and those from the default- topic simulation S .re based on the outputs' contex- tualized embeddings. We compute contextualized embeddings using the pre-trained model all mpnet-base-yy (Reimers and Gurevych, 2019). To create the training and test datasets, we use a stratified 80/20 split on Sp,.cc and S_,.c to pre- serve the balance between the classes. We report the accuracy² of the classifier the test dataset measure of individuation. Note that this measure is agmostic the particular choice of dif- ferentiator and contextualized embedding model, and we show results with other choices in Appendix B.
We represent each word E W as the mean of the contextualized embeddings of the sentences containing that word across S S_1. and Sp__. We define the semantic axis
Maatence-BER')
k P
Vi
tj:
(1)
m
where pi/tz is a word in wo W; respectively, i.c., represent Py/Pr as the mean of the embeddings of W,/Wa respectively. This subtraction-base axis allows for scaling relative to how closely- related the topic and persona are.
This score alone is necessary but insufficient for identifying caricature, as caricature must also exaggerate, which we measure next.
To evaluate exaggeration, we compute the aver- age cosine similarity of a given simulation's con- textualized embedding to this axis:
cos(Sp.t.c, Vp,t Ei cos(S p.t.et Vp.t)
4.3 Measuring Exaggeration
(2)
We define caricature as text having more of (and exaggerating) the defining characteristics associ- ated with persona and less of those associated with topic. Unlike individuation, exaggeration requires a more nuanced measure than differentiation from the default-topic simulation: an output mentions the topic frequently, can casily be diffcrentiated, but it may still be a caricature. (Note that it is
n
where Sp.ee refers individual outputs, i.e. 1, 2...... n for outputs from the same simulation Sp..c-
The final value we report as the measure of exag- geration is this value normalized to lie between 0 and I (by scaling it relative to the cosine similarity of the default-persons and default-topic simulations with the axis):
In our experiments, the classes are balanced since we generate 100 outputs for each simulation, but this measure generalizes to contexts with imbalanced classes.
cos(Sp..c Vp.)) cos(S Vpse) cos cos(Sp Vp,) cos(S Vp,t)
(3)

---

Given simulation Swith persona and topic t...
1. Generate default-topic & default-persona simulations
2. Measure Individuation: Differentiability From Default
3. Measure Exaggeration, Persona-Topic semantic axis
P
Defautt-persona: "A person's comment about t..."
Accuracy classifier that differentiates between default-persona and
Cosine similarity of S and axis
Default-topic: p's comment..."
Figure 3:
Our method measure caricature in LLM simulations. We rely on comparing defining character-
isties of the persona and topic dimensions measure individuation and exaggeration.
We perform internal and external validation of these persona-topic semantic axes (Appendix C).
to an online forum:"3 We explore such simulations
using 15 different personas (5 race/cthnicitics, 3 genders, 3 political ideologies, 3 ages, and the neu- tral "person") and 30 pairs of topics.
5 Experiments
wide range of common topics that vary in (1) specificity (e.g., overcoming fear of driving much more specific than cars) and (2) level of controversy (e.g., abor- tion is much more controversial than health). To cover both dimensions, we use topics from Wiki- How, which is knowledge base with a wide range of topics (Koupaee and Wang, 2018), and from ProCon.org. which is website that lists popular controversial topies and has been used in the NLP context study stance and argumentation (Misra et al., 2016; Hosseinia et al., 2020). We use the first 15 categories from WikiHow's "popular categories" webpage and randomly sample an associated spe- cific "how-to" for each category. For ProCon. org. we randomly sample 15 topics from ProCon.on org s "debate topics" webpage. Each topic the page is listed in both more general and more specific word- ing, e.g., abortion and should abortion be legal? Thus, for each sampled subject, we use both the general and specific versions as topies.
For topics, aim to
We use our method to evaluate simulations vari- ous contexts that have been used in previous work to demonstrate the capabilities of LLM simulation (Park et al., 2022; Santurkar et al., 2023; Jiang et al., 2022). Our experiments are focused on the two most widely-used contexts of (1) online fo- rum setting and (2) question-answering interview setting. We also evaluate the Twitter context as an additional robustness study (Appendix G). These choices are based on our survey of the literature on LLM simulations (Figure Table A6): Among 15 papers in this area, we found that six use the con- text of a virtual community society and four rely open-ended interview survey context. The remaining five in various question-answering contexts, so conclusions about the (2) can also pro- vide insight into these types of simulations.
We use the state-of-the-ar GPT-4 model for all experiments (OpenAI, 2023); like others (Dubois et al., 2023), we find that open-souree LLMs and older models worse simulation tasks, yield- ing outputs that are unrealistic and significantly lower quality. For each simulation setting Spt.e> we generate 100 outputs and average all results across them. See Appendix E for a power analysis of this sample size. The full details for each set- ting, including topic lists, persona lists, and default- topic/persona prompts, are in Appendix D.
5.2 Interview
Various previous works simulate opinions of different demographics using an interview-style prompt (Argyle et al., 2023; Santurkar et al., 2023; Hämäläinen et al., 2023, inter alia). We reproduce the public opinion survey simulation context from Santurkar al. (2023), using the prompt:
5.1 Online Forum
"Below you will be asked to provide short description of your identity and
Park al. (2022) demonstrate the believability of LLM simulations of users in online forums. Follow- ing their prompting format, we use the prompt: "A [persona] posted the following comment on [topic]
Park et al. (2022) also include an HTML <span>> tag to improve content quality. We find that this is only necessary for older models and not GPT-4.

---

then answer some questions. Description: I am [persona]. Question: [topic] Answer:"
6.2 Exaggeration scores reveal the personas and topies most susceptible to caricature
Next, we examine the exaggeration scores, i.c. the similarities across the simulations to their corre- sponding persona-topic axes (Figure 5). Since al- most all the simulations are able be individuated, the exaggeration score proxy for cari- cature, i.e., given simulation highly susceptible to caricature if has high exaggeration score.
For topics, we randomly sample 30 questions from the Pew Research's American Trends Panel survey questions that Santurkar et al. (2023) iden- tify as "most contentious" in their OpinionQA dataset. We convert the multiple-choies questions into open-ended ones by removing the multiple- choice answer options. For personas, we use the same personas as described in Section 5.1.
6.2.1 Caricature ?: Topic specificity J
In the online forum context, among the topics, we find that the more general topics resulted in higher exaggeration scores, and thus higher rates of car- icature. while the more specific topies had much lower rates of caricature (Figure 5). In particular, the general, uncontroversial topics have highest exaggeration scores. The top five topics with the highest mean rates of caricature are Health, Phi- losophy and Religion, Education and Communi- cations, Relationships, and Finance and Business. To explore this pattern further, we experiment on fine-grained range of topic specificity: For the topic with highest exaggeration score (Health), we generate simulations for a range of related topics with 5 levels of specificity (Appendix D.1.1). We find that this pattern holds: the level of caricature decreases as the specificity of the topic increases (Figure A2). We find no correlation between topic
6 Results and Discussion
We apply our caricature detection method to eval- uate the simulations produced in these different contexts. We further operationalize the CoM- PosT framework by aggregating the individuation and exaggeration scores across the dimensions of topic and persona. This enables us to analyze the topics and personas that lead to the most caricatures across different contexts. For instance, to exam- ine what personas lead to the most caricatures in a particular context, we compute the mean score for each persona across all simulations (varying in topic) for that persona and then compare these scores. We also report results from additional ex- periments that explore the influence of the context dimension in Appendix F.
length and caricature otherwise.
In the interview context, the exaggeration scores are broadly comparable to the scores for the more specific topics in the online forum context (Figure A5). After controlling for context (Appendix F), we hypothesize that this is because they are similar in specificity. We also observe this inverse relation- ship between topic specificity and susceptibility to caricature in the Twitter context (Appendix G).
6.1 Simulations of all personas can be individuated from the default-persona
The mean individuation score and 95% confidence interval for every persona is > 0.5, i.e., each per- can be meaningfully individuated from the default-persona at a rate better than random chance. Mean individuation scores across the online fo- rum and interview contexts are in Figure 4. We see that the woman and man personas have lower mean scores in the online forum context, while in the interview context, the mean score is >> 0.95 for every thus, this score is not informa- tive for comparing caricature across personas. The difference in score between contexts arises from differences in the sample distributions: compared the online forum context, the interview context simulations have lower variability, so they are cas- ier to individuate.
6.2.2 Caricature ?: Political ideology, race, and marginalized personas
In the online forum context, the personas nonbi- nary, Black, Hispanic, Middle-Fastern, and conser- vative have highest mean exaggeration score (Fig- ure 5b-d). Similarly, in the interview context, the highest mean scores are for the personas nonbinary, Hispanic, 80-year-old, conservative and Middle- Eastern. Broadly, nonbinary gender, non-white race/ethnicity, and political leanings are most sus- ceptible to caricature, while binary gender groups have the least caricatures. This corresponds to pre- vious findings that NLP systems broadly align with

---

1.0
0.9
Accuracy
0.8
0.7
0.6
Online Forum
Interview
0.5
age 20 age
age 80 conser- vative
liberal moderale white Black
Asian Hispanic Middle-
man
binary
Figure 4: Mean individuation scores (differentiability from default). The standard error for each point is < 0.02 and thus not visible on the plot. Based on a classifier between default default-persona and the target persona, many of the personas easy individuate (have high accuracy scores). The only personas that slightly challenging to differentiate are the gender groups woman and man in the online forum context (blue circles). All personas have mean score 0.95 in the interview context (green stars).
Age
Political
Race/Ethnioit
Gender
0.8
Wikihow general
ProCon general
WikiHow specific
ProCon specilic
Exeggeratoo (Simllarity
0.4
0.3
0.2
0.0
80
Liberal Moderate
Asian
Black Hispanic Midele-
White
Non-
Woman
Binaay (d)
(8)
(b)
(c)
Figure 5: Mean exaggeration scores + standard error in the online forum context. We measure exaggeration as normalized cosine similarity the persona-topic axis. The general topics (purple, larger marker) have higher rates of exaggeration, and thus caricature, than the specific topics (orange, smaller marker). The uncontroversial (WikiHow, squares) topics have higher rates caricature than the controversial (ProCon.org. stars) topics. Personas related political leanings, race/ethnicity, and nonbinary gender broadly have the highest rates of caricature.
the perspectives of liberal, white, and younger pop- ulations, while the perspectives of non-binary peo- ple are poorly represented by these systems (Santy et al., 2023; Santurkar et al., 2023). It is surprising that although Asian and woman reflect marginal- ized groups, they have relatively low rates of car- icature; this may further reflect implicit defaults in LLM outputs. Certain personas and topics have more nuanced relationships, e.g., conservative and liberal personas in the online forum context have the widest gap between the scores of the uncontro- versial and controversial topics. The Twitter con- text also reveals that, beyond analyzing the persona and topic dimensions separately, some persona- topic combinations are particularly susceptible to caricature (Figure A4).
for one way in which simulations may fail. Thus, simulations that seem caricature-free may still con- tain stereotypes, as our method captures how much simulation exaggerates the persona in a particular setting, which is not all-encompassing catalog of stercotypes. We this in the simulated "woman" responses in the online forum context: The default- topic generations contain specific stereotypes, e.g.. recently purchased new vacuum cleaner and have to say, I am extremely satisfied with its perfor- mance! has made my cleaning routine so much casier and efficient." This response reflects gender bias in that it focuses cleaning and other domes- tic tasks, while simulations of other personas do not. Although the simulated "women" responses contain various other gender stereotypes/biases be- yond association with domestic tasks, they have low caricature scores.
6.3 Stereotypes
We note that our framework not an exhaustive test for bias or failure modes, but rather a measure

---

7 Recommendations
tifaceted personas into simulations. Our goal in avoiding caricature not to erase difference, but rather the opposite: capturing relevant differences that reflect meaningful insights rather than shallow, misleading generalizations.
We conclude with several recommendations for those interested in LLM simulations.
Mitigating Caricature Researchers should use our method to test their simulation in their partic- ular context and critically examine whether sim- ulation helps illuminate the desired phenomenon. While the relationship between topic, persona, and context in causing caricature is nuanced, we gen- erally encourage researchers and practitioners to more specific topics to mitigate caricature. Any attempt simulate group a politi- to be done with particular care and attention.
Positionality
The perspectives introduced in this paper have un- deniably been shaped and influenced by our po- sitionality. Myra Cheng identifies as a Chinese- American woman. The authors are a PhD stu- dent, postdoctoral scholar, and professor respec- tively in the Stanford University Computer Sci- ence department, which is predominantly male and white/Asian.
cized or marginalized
Documenting Positionality Research on LLM simulations face the well-documented challenges of human-focused, value-laden interdisciplinary work (Marsden and Haag, 2016). For instance, researchers themselves may be subject to the out- group homogeneity effect, i.e., the tendency rely on stereotypes and generalizations for groups to which they do not belong (Plous, 2003). Following work on model, dataset, and system documenta- tion (Bender and Friedman, 2018; Mitchell et al., 2019; Gebru et al., 2021; ; Adkins et al., 2022), we call for increased transparency and documentation for simulations, including the dimensions of CoM- PosT and less-visible aspects such as the creators' positions, motivations, and process. Drawing upon HCI work on reflexivity and positionality (Keyes al., 2020; Liang al., 2021; Bowman et al., 2023), we encourage researchers to report how their identity and perspectives may influence their work.
Ethical Considerations
From impersonation to pornography, LLM simu- lations can have deeply problematic applications. We are strongly against such applications, and we also do not condone research and development that may enable such applications by bad actors with- out guardrails in place. Our CoMPosT framework offers a shared language to meaningfully critique such work. For instance, one might imagine com- ing to a consensus to avoid simulating certain top- ics, personas, and contexts entirely. Introducing method to measure caricature offers way to make known this concerning limitation. Lack of carica- ture based on our measure does not mean that a simulation is necessarily acceptable or high quality (see Section 10).
Implicit Defaults The least caricatured personas are also those that others have found to be implicit defaults in LLMs. Implicit defaults in LLMs may shift depending on the prompt, context, etc., as well as including aspects of identity and social factors that may be invisible underrepresented in exist- ing empirical data and surveys Given the increas- ing proliferation of generated content and a limited quantity of human-writted text (Xue et al., 2023; Shumailov et al., 2023), caricatures become only more relevant with the prospect of future LLMS that are trained on generated data: what will their
Understanding Difference Although some ap- plications of LLM simulations focus on aggre- gates rather than individuals, it is critical to un- derstand the landscape of individuals from which these groupings arise, and it is often necessary to use more subtle forms of aggregation. Otherwise, minority opinions and subgroup disparities may be overlooked (Herring and Paolillo, 2006; Hanna et al., 2020; Wang et al., 2022). Takayama (2017) suggests countering caricature "by providing fully contextualized, balanced, and nuanced descrip- tion," and in HCL, Marsden and Pröbster (2019) explore how to explicitly capture users' multidi- mensional identities.
"Existing studies often exclude various demographics: studies on human behavior oversample from the population of American college students. who have been shown to dif- fer from other populations in significant ways (Segall et al.. 1966). Psychology and social science research subjects are disproportionately Western, educated, and from industrialized, rich, and democratic countries (Henrich et al., 2010).
Drawing inspiration from these works, one fu- ture direction is injecting variation and using mul-

---

defaults be, and how might they further amplity
project. This work was funded in part by an NSF Graduate Research Fellowship (Grant DGE- 2146755) and Stanford Knight-Hennessy Scholars graduate fellowship to MC, as well as a Meta grant and an NSF grant IIS-2247357 to DY.
caricature?
10 Limitations
While we fill a critical gap since there is no ex- isting work on systematically detecting stereo- types/caricatures in simulations or evaluating simu- lations in this manner at all, our measure is limited in scope: it is not comprehensive evaluation of the quality of simulation. We quantify susceptibility to caricature, which is particular failure case of simulation. Our method may yield false positives (simulations that seem acceptable and caricature- free based on our method but have other problems). Avoiding caricature is a necessary but insuffi- cient criterion for simulation quality; our metric should be used in tandem with other evaluations. As pilot study for recently-emerginn direction of work, we hope to lay the groundwork for more comprehensive evaluation of simulations in the fu- ture. perhaps in tandem with human evaluation.
References
David Adkins, Bilal Alsallakh, Adeel Cheema, Narine Kokhlikyan, Emily McReynolds, Pushkar Mishra, Chavez Procope, Jeremy Sawruk. Erin Wang, and Polina Zvyagina. 2022. Prescriptive and descrip- tive approaches to machine-learning transparency. CHI Conference on Human Factors in Computing Systems Extended Abstracts, pages 1-9.
Gati Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate mul- tiple humans and replicate human subject studies. In Proceedings of the 40th International Conference Machine Learning, ICML'23, JMLR.org.
Anu Aneja. 1993. "Jasmine," " the sweet scent of exile. Pacific Coast Philology, pages 72-80.
As we provide a first step toward characterizing and evaluating LLM simulations, an area which currently lacks a shared language for discussion and comparison, we focus only on simulations of personas that reflect subpopulations such social groups and one-round response formats. How- ever, our framework easily extends other more complex or open-ended settings. For instance, for multi-round simulation, one could apply our frame- work by using the full text of the simulation across the rounds. Depending on the length and structure of the simulation, one could also split the simula- tion into multiple parts and characterize each part's s propensity to caricature.
Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language mod- els to simulate human samples. Political Analysis, 31(3):337-351.
April H Bailey, Adina Williams, and Andrei Cimpian. 2022. Based on billions of words on the Internet, people= men. Science Advances, 8(1 xebbmmeb324433
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler. Lucy Campbell-Gillinghanl Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. 2022. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Infor- mation Processing Systems,
Also, note that in the semantic axes, Pp does not necessarily reflect a universal notion of the model's representation or description of that particular de- mographic group. It merely characterizes the words that distinguish a simulation of that persona, given the particular context c. such as an opinion from that demographic's 's perspective. This enables us to measure if and when a simulation is dominated by language that is a caricature of the persona, but our work is not a comprehensive evaluation of stereo- types or representations of demographic groups.
David Bamman. Jacob Eisenstein, and Tyler Schnoebe- len. 2014. Gender identity and lexical variation in social media. Journal of Sociolinguistics, 18(2):135- 160.
Daniel Bar-Tal, Carl F Graumann, Arie W Kruglanski, and Wolfgang Stroebe. 2013. Stereotyping and prej- udice: Changing conceptions. Springer Science & Business Media.
Caroline Bassett. 2019. The computational therapeutic: Exploring Weizenbaum's ELIZA history of the present. AI SOCIETY, 34:803-812.
Joseph Bates et al. 1994. The role emotion in beliey- able agents. Communications of the 122 125.
Acknowledgments
Thank you to Tony Wang, Joon Sung Park, Omar Shaikh, Caleb Ziems, Camille Harris, and Matthias Gerstgrasser for their feedback throughout this
Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science.

---

Transactions of the Association for Computational Linguistics, 6:587-604.
Alexander M Czopp, Aaron C Kay, and Sapna Cheryan. 2015. Positive stereotypes are pervasive and powerful. Perspectives on Psychological Science, 10(4):451-463.
Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6)c22185233 1200.
Eberhard Demm. 1993. Propaganda and caricature in the first world war. Journal of Contemporary History, 2811):1331
Irene V Blair, Jennifer E Ma, and Alison P Lenton. 2001, Imagining stereotypes away: The moderation of im- plicit stereotypes through mental imagery. Journal of personality and social psychology. 8188):828.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang. Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca- Farm: A simulation framework for methods that learn from human feedback.
Luisa N Borrell, Jennifer R Elhawary. Elena Fuentes- Aftlick, Jonathan Witonsky, Nirav Bhakta, Alan HB Wu, Kirsten Bibbins-Donin Dominnno,, José R Rodríguez- Santana, Michael A Lenoir. James R Gavin III. et al. 2021. Race and genetic ancestry in medicine- a time for reckoning with racism. New England Journal of Medicine, 384
Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization: Quantify- ing data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume, pages 1325-1335 Online. Association for Computational Linguisties.
Leslie Bow. 2019. Racist cute: Caricature, kawaii-style, and the Asian thing. American Quarterly, 71(1):29- 58.
Susan T Fiske, Amy JC Cuddy, Peter Glick, and Jun Xu. 2002. A model of (often mixed) stereotype content: competence and warmth respectively follow from per- ceived status and competition. Journal of personality and social psychology, 82(6))878.
Robert Bowman, Camille Nadal, Kellie Morrissey, Anja Thieme, and Gavin Doherty. 2023. Using thematic analysis in healthcare HCI at CHI: A scoping review. In Proceedings of the 2023 CHI Conference on Hu- man Factors Computing Systems, pages 1-18.
Timnit Gebru, Jamie Morgenstern, Briana Vec- chione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64( (2):86- 92.
James DJ Brown. 2010. A stereotype, wrapped in a cliché, inside a caricature: Russian foreign policy and orientalism. Politics, 30(3):149-155)
Yang Cao, Anna Sotiikova, Hal Daumé III. Rachel Rudinger, and Linda Zou. 2022. theory-grounnde measurement of US social stereotypes in english lan- guage models. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 1276-1295
Peter Gottschalk and Gabriel Greenberg. 2011. From Muhammad to Obama: Caricatures, cartoons, and stereotypes of Muslims. Islamophobia: The chal- lenge of pluralism in the 21st century, pages 191- 210.
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural lan- guage models. In The Eleventh International Confer- ence on Learning Representations.
Jonathan Grudin. 2006. Why personas work: The psy- chological evidence. The persona lifecycle, 12:642- 664.
Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating large language models in gener- ating synthetic HCI research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-19.
Eshwar Chandrasekharan, Mattia Samory, Shagun Jhaver, Hunter Charvat, Amy Bruckman, Cliff Lampe, Jacob Jacob Eisenstein, and Eric Gilbert. 2018. The internet's s hidden rules: An empirical study of Reddit norm violations at micro, meso, and macro scales. Proceedings of the ACM on Himan-Compute Inter- action, 2(CSCW):1-22
Sil Hamilton. 2023. Blind judgement: Agent-based supreme court modelling with GPT. arXiv preprint arXiv:2301.05327.
Myra Cheng, Maria De-Arteaga, Lester Mackey, and Adam Tauman Kalai. 2023a. Social norm bias: Residual harms of fairness-aware algorithms. Data Mining and Knowledge Discovery, pages 1-27.
Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. 2020. Towards critical race method- ology in algorithmic fairness. In Proceedings of the 2020 conference on Fairness, Accountability, and Transparency, pages 501-512.
Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023b. Marked personas: Using natural language prompts to measure stereotypes in language models. In Proceed- ings of the s1 Annual Meeting of the Association for Computational Linguistics, pages 1276-1295.
Madeline E Heilman. 2001. Description and preserip- tion: How gender stereotypes prevent women's 's as- cent up the organizational ladder. Journal of Social Issues, 57(4)6557-674

---

Joseph Henrich, Steven Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61-83.
the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 1000-1008.
Susan C Herring and John C Paolill.. 2006. Gender and genre variation weblogs. Journal of Sociolinguis- lics, (0(4):439-459.
Calvin A Liang. Sean A Munson, and Julie A Kientz. 2021. Embracing four tensions in human- computer interaction research with marginalized peo- ple. ACM Transactions on Computer- Hummn Inter- action (TOCHI), 28(2):1-47.
James D Hollan, Edwin L Hutchins, and Louis Weitz- man. 1984, Steamer: An interactive inspectable simulation-base training system. Al magazine, 152):
Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang. Denny Zhou, Andrew M Dai, Diyi Yang. and Soroush Vosoughi. 2023. Training socially aligned language models in simulated human society. arXiv preprint arXiv:2305. 16960.
Marjan Hosscinia, Eduard Dragut, and Arjun Mukher- jee. 2020. Stance prediction for contemporary is- sues: Data and experiments. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 40, Online. Association for Computational Linguistics.
Li Lucy, Divya Tadimeti, and David Bamman. 2022. Discovering differences in the representation of peo- ple using contextualized semantic axes. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3477-3494, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022. CommunityL.M Probing partisan world- views from language models. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 6818-668 6826, Gyeongju, Republic of Korea. International Committee Computational Linguistics.
Bohun Lynch. 1927. A history of caricature. Faber and Gwyer.
Julia M. Markel, Steven G. Opferman, James A. Lan- day, and Chris Piech. 2023, GPTeach: Interactive TA training with GPT-based students. In Proceed- ings of the Tenth ACM Conference on Learning @ Scale, L@ S '23, page 226-236, New York, NY, USA. Association for Computing Machinery.
Randolph M Jones, John E Laird, Paul E Nielsen, Karen J Coulter, Patrick Kenny, and Frank V Koss. 1999. Automated intelligent pilots for combat flight simulation. Al magazine, 20(1):27-27.
Lee Jussim, Jarret T Crawford, Stephanie M Anglin, Sean T Stevens, and Jose L Duarte. 2016. Inter- pretations and methods: Towards more effectively self-correcting social psychology. Journal of Experi- mental Social Psychology, 66: I 1331.
Nicola Marsden and Maren Haag. 2016. Stereotypes and politics: Reflections on personas. In Proceedings of the 2016 CHI conference human fuctors in computing systems, pages 4017-4031.
Gauri Kambhatla, Ian Stewart, and Rada Mihalcea. 2022. Surfacing racial stereotypes through identity portrayal. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1604-1615,
Nicola Marsden and Monika Pröbster. 2019. Personas and identity: Looking at multiple identities to inform the construction of personas. In Proceedings of the 2019 CHII Conference Human Factors in Comput- ing Systems, pages 1-14.
Os Keyes, Burren Peil, Rua M Williams, and Katta Spiel. 2020. Reimagining (women's ') health: HCI, gender and essentialised embodiment. ACM Transactions on Computer- Human Interaction (TOCHI), (7(4))1-4
Melissa McCracken, Miho Olsen, Moon Chen Jr, Ahmedin Jemal, Michael Thun, Vilma Cokkinides. Dennis Deapen, and Elizabeth Ward. 2007. Can- cer incidence, mortality, and associated risk factors among Asian Americans of Chinese, Filipino. Viet- namese, Korean, and Japanese ethnicities. CA: cancer journal for climicians, 1904):190-
Mahnaz Koupaee and William Yang Wang. 2018. Wiki- how: A large scale text summarization datasct. arXiv preprint arXiv:1810.0040
Kay Dian Kriz. 2008. Slavery, sugar, and the culture of refinement: Picturing the British West Indies, 1 1700- 1840. Paul Mellon Centre.
Amita Misra, Brian Ecker, and Marilyn Walker. 2016. Mcasuring the similarity of sentential arguments in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dia- logue, pages 276-287, Los Angeles. Association for Computational Linguistics.
Neha Kumar, Naveena Karusala, Azra Ismail, Marisol Vilg-Villacres, and Aditya Vishwanath. 2019. En- gaging feminist solidarity for comparative research, design, and practice. Proceedings of the ACM on Humman-Computer Interaction, 3(CSCW)
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson. Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Account- ability, and Transparency, pages 220-229.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap open- domain question answering datasets. In Proceedings

---

Chandra Mohanty. 1988. Under Western eyes: Feminist scholarship and colonial discourses. Feminist review, 30( 10-6888--8
Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. SLPooitionality: Characteriving design biases of datasets and models. In Proceedings of the 6 Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Pupers), pages 9080-9102, Toronto, Canada. Association for Computational Linguisties.
Burt L Monroe, Michael P Colaressi, and Kevin M Quinn. 2008. Fightin' words: Lexical feature selection and evaluation for identifying the content of political con- flict. Political Analysis, 16(4):372-403.
Johannes Schneider, Christian Meske, and Michalis Vla- chos. 2020. Deceptive Al explanations: Creation and detection. In International Conference on Agent and Artificial Intelligence. SciTePrees.
OpenAl. 2023, GPT-4 technical report. arXiv.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Nearal Information Processing Systems, 35:27730-27744,
Marshall H Segall, Donald Thomas Campbell, and Melville Jean Herskovits. 1966. The influence of culture on visual perception, volume 310. Bobbs- Merrill Indianapolis.
Joon Sung Park, Joseph C Brien, Carrie J Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023a. Generative agents: Interac- tive simulacra of human behavior. arXiv preprint arXiv:2304.03442
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023. The curse of recursion: Training on gen- erated data makes models forget. arXiv preprint arxiv:2305. 17493.
Joon Sung Park, Lindsay Popowski, Carrie Cai, Mered- ith Ringel Morris, Percy Liang, and Michael Bern- stein. 2022. Social simulacra: Creating populated prototypes for social computing systems. In Proceed- ings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1-18.
Phillip R Slavney, 1984. Histrionic personality and anti- social personality: caricatures stereotypes? Com- prehensive Psychiatry, 129-4412
Irene Solaiman and Christy Dennison. 2021. Process for adapting language models to society (palms) with values-targeted datasets. Advances in Newral Infor- mation Processing Systems, 34:5861-5877
Peter S Park, Philipp Schoenegger, and Chongyang Zhu. 2023b. Artificial intelligence in psychology research. arXiv preprint wrXiv:2302.07267
Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, and Bee Kamar. 2022. Investigations of performance and bias human-A teamwork hiring. Proceed- ings of the AAAI Conference on Artificial Intelligence, 36(11):2289911 120977
Keita Takayama. 2017. Imagining east asian educa- tion otherwise: Neither caricature, nor scandalization. Asia Pacific Journal of Education, 37(2):262 2744
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization with- out overfitting: Analyzing the training dynamics of large language models. Advances in Neural Informa- tion Processing Systems,
David Perkins. 1975. A definition of caricature and caricature and recognition. Studies in Visual Commu- nication, 1-1):1-24.
Scott Ed Plous. 2003. Understanding prejudice and discrimination. McGraw-Hill.
Andrea Aler Tubella, Dimitri Coelho Mollo, Adam Dahlgren Lindström, Hamnah Devinney, Virginia Dignum, Petter Ericson, Anna Jonsson, Timotheus Kampik, Tom Lenaerts, Julian Alfredo Mendez, et al. 2023. ACROCPoLis: descriptive framework for making sense of faimess. In 2023 ACM Conference Fairness, Accountability, and Transparency, pages 1604-1615.
Jen' nan Ghazal Read, Scott Lynch, and Jessica
West. 2021. Disaggregating heterogeneity among non-hispanic whites: evidence and implications for us racial/ethnic health disparities. Population re- search and policy review, 40:9-31.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert. Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing (EMNLP-IJCNLP), Association for Computational Linguistics.
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intel- ligence: Crowd workers widely use large language models for text production tasks. arXiv preprint arXiv:2306.07888
Kailas Vodrahalli, Roxana Daneshjou, Tobias Gersten- berg, and James Zou. 2022. Do humans trust advice more if comes from Al? an amalysis of human-Al interactions. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 763- 777.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML/23. JoL..

---

Angelina Wang, Vikram V Ramaswamy, and Olga Rus- sakovsky. 2022. Towards intersectionality in ma- chine learning: Including more identities, handling underrepresentation, and performing evaluation. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 336-349.
don't understand my gender identity. While can be frustrating times, I choose to take as an opportunity educate others and promote a more inclusive society. It's important to recognize and respect everyone's 's experiences and identities, even if they difter from our own." "
These descriptions focus solely on general state- ments regarding gender identity and inclusion. We find the same phenomenon in simulations of a non- binary person's 's perspective other topics. For the topic "how read guitar music," " some randomly- sampled simulated non-binary person's 's responses are:
Joseph Weizenbaum. 1966. ELIZA computer pro- gram for the study of natural language communica- tion between man and machine. Communications of the ACM, 31)::36-45.
Robert Wolfe and Aylin Caliskan. 2022. Markedness in visual semantic AI. 2022 ACM Conference on Faimess, Accountability, and Transparency.
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, zangwei Zheng, and Yang You. 2023. To repeat or not to repeat: Insights from scaling Il under token-crisis. arXiv preprint arXiv:2305. 13230.
"Hey everyone! I just wanted to share my expe- rience as nonbinary individual learning how to read guitar music. It's been a really empower- ing journey for me, as it allows me to express myself in way that transcends the limits of gender. Moreover. I've found that the guitar com- munity is incredibly accepting and supportive, which has heen huge help in overcoming some of the challenges that I've faced as nonbinary person. Keep rocking out, friends!"
Diyi Yang. 2019. Computational Social Roles. Ph.D. thesis, Carnegie Mellon University Pittsburgh, PA, USA.
JD Zamfireccu-Periin Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can't prompt: how non-Al experts try (and fail) to design IIm prompts. In Proceedings of the 2023 CHI Con- ference on Human Factors in Computing Systems, pages 1-21.
"As nonbinary guitarist, wanted share some advice and encouragement for others who may be navigating the world guitar music. Remember that music is a universal language and it does not adhere to any specific gender norms or societal expectations. Allow yourself the freedom ex- plore different styles, techniques, and genres of guitar music that resonate with you. Emhrace your unique perspective and use it to create music that reflects your authentic self, And most im- portantly, don't 't forget to have fun and enjoy the journey of learning and growing as a musician."
Caleb Ziems, Jane Dwivedi- Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2023. NormBank: A knowl- edge bank of situational social norms. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7756-7776, Toronto, Canada, Association for Computational Linguistics.
Rather than actually discussing "how to read gui- tar music," these outputs disproportionately focus generic statements about gender, authenticity, and inclusion (bolded). These responses exagger- ate this particular imagined aspect of a nonbinary person's ss perspective. Similarly, for the topic "cars and vehicles," some aundomly-sampe simulated non non-binary person's 's responses are:
A Examples of Caricatures in Simulation
Which our paper focuses on how to discover prob- lematic patterns of caricature at the larger scale used in LLM simulations, these patterns are clearly evident on an individual example level. For ex- ample, for the "nonbinary person" persona in the online forum setting, which has some of the high- est exaggeration scores, some randomly-sampler default-topic responses are:
"I just wanted to share my experience as nonbi- nary individual when it comes to cars and other vehicles. For me, it's important that my mode of transportation reflects my identity and personal aesthetics. I've found that customizing my vehi- cle with various accessories, paint jobs, and even custom license plates has been a fun and empow- ering way express myself. I also appreciate vehiele that has a gender-neutral design, as that aligns with my identity. I've noticed that more and more auto manufacturers are starting to ere- ate designs that feel more inclusive and neutral, which is great for people like me. Ultimately, the vehicle you choose should be reflection of your personality and preferences, no matter your gender identity. Happy driving, everyone!"
"As nonbinary individual, I often feel like I don't fit into the traditional boxes society has created for gender. This can be difficult when navigat- ing spaces that dominated by cisnormativity. However, have found solace and empowerment in connecting with other nonbinary people and advocating for inclusivity and understanding. It important to remember that gender a social construct, and is up to identities and push for more accepting world."
define our own
"As a nonbinary individual, I often find myself in situations where people are not aware of or
"Hey everyone! As a nonbinary individual, I wanted to share my perspective on car preferenees

---

Topic Default
Output for "Man" persona
"I recently purchased a new smartphone and I'm really impressed with its features and performance. The camera quality is amazing, and the battery life lasts all day without any issues. The interface easy use, and the overall design sleek and modern. would highly recommend this phone to anyone the market for new device." "
recently purchased a new laptop and I'm absolutely loving it so far. The performance is top-notch, the design sleek, and the battery life has been really impressive. The only issue I've come across is a minor software bug, but that was casily fixed with an update. Overall, highly recommend this model to anyone in the market for reliable and high-performing laptop."
How to read guitar music
"I recently started learning how to play the guitar and I found this guide on How to Read Guitar Music extremely helpful. The step-by-step explanation, along with images and examples, made it easy to understand and follow. I'm now more confident in reading sheet music and applying it to my guitar practice. Highly recommend this guide for any beginner guitarist looking to learn how to read music. Kudos to the author for making it so simple and casy to follow!"
"I recently started learning how to read guitar music and found this article very helpful. It provided clear explanations and casy-to-foll to-lollow steps that have greatly improved my understanding of how to read sheet music for the guitar. The tips and examples given were a great reference for me as beginner, and feel much more confident in my ability now. Thanks for sharing this informative guide!"
Cars and ve- hieles
"I recently purchased 2020 Ford Mustang GT and I couldn't be happier with my decision. The drive is smooth, the car is powerful, and the interior is incredihly comfortable and stylish. have always loved the look of the Ford Mustang and this new model definitely lives up my expectations. The sound system is fantastic as well. I highly recommend this car to anyone considering a sporty, powerful, and fun vehicle." "I recently bought a 2015 Honda Civic and I'm absolutely loving it! The fuel efficiency is amazing and the ride is smooth. I've also noticed significant improvement in the handling compared to my old car. The interior is quite spacious and comfortable too. My only complaint far is the touchscreen entertainment system it's bit slow and not very user-friendly. But overall, highly recommend the 2015 Civic anyone the market for a reliahle, fucl-efficient, and enjoyable vehicle!"
Table A1: Examples of Simulated "Man" Responses
and how certain vehicles cater to different gen- der expressions. I've noticed that car manufactur- ers and marketers often gear their advertisements towards stcreotypical male or female audience, but believe there's vast array of options out there that can uppeal to people of all gender identi- ties. For example, I personally love sleek, modern designs that have a balance of form and function and don't scream 'masculinity" or "femininity." I'd love to hear about your favorite vehicles and how they play a role in expressing your unique personality and identity. Let's have an open and inclusive conversation about our diverse expe- riences with cars and other vehicles!"
(all-distilroberta-v is the next higgesst- model for general-use purposes). For (a), we first use K-Means to cluster the embeddings into two clusters, and then report the v-score. We find that the scores are overall slightly lower than using a binary classifier, which makes sense since the un- supervised method is less powerful in this context, but most of the personas can still be differentiated at rate higher than random chance. We use the su- pervised binary classifier in the main results since the purpose of this metric is to reflect reader's pability to differentiate between the two categories, which is more accurately reflected by the more pow- erful classifier. The broad patterns remain the same as the findings reported in the paper: all the per- sonas can be individuated in the Interview context, and in the Online Forum context, marginalized per- sonas are the most easily individuated, while the personas of man and woman are challenging to individuate all.
In these outputs, the bolded phrases also present unrealistic caricatures that exaggerate specific as- pects of the persona rather than reflecting the full range of possibilities of what a non-binary person might say about these topics. In stark contrast, for the "man" persona, the default-topic responses are not identity-related, and responses for particular topics are much more topical and do not exagger- ate any aspect of the "man" persona (Table Al).
B Robustness of Individuation Measure
C Internal and External Validation of Semantic Axes
The notinon of measuring individuation is broadly agnostic to the choice of differentiator and model used embed the texts. In Figure A1, we show individuation results from (a) using the unsuper- vised V-measure to measure differentiation and (b) using the all-distilroberta-v model to com- pute embeddings instead of all-mpnet-basee base-v2
For internal validation, following Lucy et al. (2022), we verify that the constructed axes are internally valid using a leave-one-out method: consider a pole P1 with a set of words W, where the other pole on the semantic axis is Pz. We remove one of the seed words w E W and construct a new pole P

---

1.0
0.8
Accuracy
0.6
0.4
Online Forum
0.2
Interview
0.0
age
20
age
40 age
80
conser- vative
liberal moderate white
Black
Asian Hispanic Midale- Eastern
man
non- binary
woman
0.9
Accuracy
0.8
0.7
0.6
Online Forum Interview
0.5
age
age
age
liberal moderate white Black
Asian Hispanic Middle- Eastern
man
woman
vative
binary
Figure A1: Top: Using the unsupervised V-measure measure differentiation results similar patterns as our main result in Figure 4. Bottom: Using an alternative pre-trained model to encode the oulputs also results in similar patterns.
that does not include w. Then, we measure the cosine similarity of w's contextualized embedding Pf and P2. We found that the former larger for all of the persona-topic semantic axes that constructed.
default: person
D.1 Online Forum Context
The full list of topics collected from WikiHow and ProCon.org for the online forum context are in topicss/oniineforuutppicc. txt the supple- mentary material.
For external validation, we manually inspect the sets of seed words and find that for each axis, is easy to differentiate which set is associated with the corresponding persona versus topic. We find that for racc/ethnicity and gender personas, the sets of top words reflect similar stereotypes reported by Cheng et al. (2023b). See Table A2 and A3 for mandomy-samppl examples of persona-topic axes seed word sets.
The general prompt for simulation with persona and topic is: 'A(n) posted the following com- ment on f to an online forum." For persona the default-topic prompt is: "A(n) posted the follow- ing comment to online forum." For topic f, the default- persona prompt is: "A person posted the following comment on an online forum."
D.1.1 Fine-Grained Specificity Experiment
D Experimental Details: Topics and Personas
The topics for the fine-grained specificity experi- ment are:
Examples each setting are in Table A4. Follow-
Specificity Level 1: health
ing Santurkar al. (2023) and Cheng et (2023b), the personas used in the online forum and interview contexts are:
Specificity Level 2: mental health
Specificity Level 3: things to do for mental health, social media's effect mental health
age: 20-ycar-old person, 40-ycar-old person, 80-year-old person
Specificity Level 4: What is the best thing you did for your mental health?, Is social me- dia bad for your mental health?
political ideology:
conservative person, lib- eral person, moderate person
Specificity Level 5: During very dark pe- riod, what was the best thing you ever did for your mental health?, To what extent do you think social media is bad for your mental health?
Asian person, Black per- son, Hispanic person, Middle-Fastern person, white person
race/ethnicities:
gender: man, non-binary person, woman

---

Age
Race/Etnnicity
Gender
Specificity
0.8
Exaggeration (Similanity
0.4
0.0
Conser- Liberal Moderate
Asian
Black HispanicMiddle. Eastern
White
Man
Non-
Woman
Binary
Figure A2: Exaggeration scores in the online forum context for topics varying in specificity. Across different personas (x-axis), exaggeration score (y-axis) is negatively correlated with the specificity of the topic (marker size).
The Level I topic is the topic with highest rate of caricature. The Level 5 topics are from the top search results for "health" in the subreddit com- munity AskReddit. The authors constructed the in- termediate specificity levels by interpolating speci- ficity between these. The resulting exaggeration scores are in Figure A2. We see that specificity and exaggeration are negatively correlated.
0.5, and desired power 0.8. Thus, our choice of obtaining 100 samples per simulation is more than sufficient to achieve the desired power for both the individuation and exaggeration metrics.
F Influence of the Context Dimension
In this section, we explore the effect of the context dimension. To verify that the trends we observe are not due to the difference in context alone, we also experiment with switching the contexts and topics, i.e., 1) simulations with the online forum topics in the interview context 2) simulations with the interview topics the online forum setting. Figure A3 reveals that the trends in caricature rates persist for the same topics in different contexts rather than being based on context alone. Interest- ingly, the exaggeration scores are overall slightly higher in these switched contexts than in the origi- nal contexts. This may be impacted by memoriza- tion dynamics (Elangovan al., 2021; Tirumala et al., 2022; Carlini al., 2023), a relationship explore future work (e.g., are memorized topics less susceptible to caricature?).
D.2 Interview Context
The full list of Pew opinion survey questions that we use as topics in the interview setting are in sopics/pewtoppic material. The default-topic and default-persona prompts are in Table A5.
txt in the supplementary
E Power Analysis
To justify 100 examples per simulation setting, we provide the following power analysis. Note that for individuation, a simulation that cannot be in- dividuated would have score 0.5 (random chance). Across personas and topics, the lowest mean score 0.65, and the highest standard deviation was 0.2. A power analysis using a I-test for indepen- dent samples reveals that the necessary sample size is 28 given the effect size (0.65 0.5) /0.2 = 0.75, alpha = 0.5, and desired power = 0.8. Similarly, for exaggeration, a simulation with no exaggera- tion of a persona would have score 0. We found that simulations with less-specific topics and personas of many political idcology, race, and marginalized groups result in high exaggeration scores. Among these simulations, the lowest mean score was 0.23, and the highest standard deviation was 0.37. Again using power analysis, the necessary sample size is 41 using effect size (0.23 - 0)/0.37 = 0.62, alpha
G Twitter Context
We additionally conduct and analyze experiments in the Twitter context introduced by Jiang al. (2022), and find similar trends that corroborate the results of the main paper.
G.1 Experimental Details
Jiang al. (2022) demonstrate how LLM simula- tions of Republican and Democrat Twitter users result in opinions about public figures and groups that correspond to the outcomes of the Ameri-

---

Online Forum Context Topics (WikiHow and ProCon) in Interview Context
Politial
RacerEthnictt
Gender
Wilshow general
ProCon general
0.5
WildHow spectfic
Exaggeration (Smilarit to Flaggaraton (Smilnty to
ProCon spectfic
0.2
40
Liberal
Moderate
Asian
Hispanic Middle Eastern
White
Binaty
Interview Context Topics (Pew Survey Questions) Online Forum Context
Political
Race/Ethnicily
Gender
0.1
Survey Quastions, Online Forum Context
Conser-
Liberal
Moderate
Asian
Black Middatt
White
Non-
Woman
valive
Eastern
Binary
Figure A3: Exaggeration scores for simulations with the topics used the online forum context (from WikiHow and ProCon.org), but using the interview context instead (top), and vice versa (bottom). We find similar patterns, indiciating that we can attribute the trends in scores to topics rather than context alone.
can National Election Studies (ANES) 2020 Ex- ploratory Testing Survey. To evaluate susceptibil- ity to caricature this context, we use their best- performing prompt. [persona] user posted on Twit- ter: [topic] is/are. Following their work, we use the same two personas (Republican and Democrat) and 30 topics that they use: "(a) 16 people: Donald Trump, Barack Obama, Joe Biden, Elizabeth War- ren, Bernie Sanders, Pete Buttigicg. Kamala Harris, Amy Klobuchar, Mike Pence, Andrew Yang, Nancy Pelosi, Marco Rubio, Alexandria Ocasio-Cortez. Nikki Haley, Clarence Thomas, Dr. Anthony Fauci, (b) 14 groups: black people, white people, Hispanic people, Asian people, illegal immigrants, feminists, the #MeToo movement, transgender people, so- cialists, capitalists, big business. labor unions, the Republican Party, the Democratic Party."
to race and other groups broadly have higher ex- aggeration scores, while topics of specific public figures have lower ones, reflecting the pattern that more general topics are more caricatured. Discus- sion of Republican public figures yielded more exaggeration than those of Democratic public fig- ures. Examining topic subcategories reveals more nuanced relationships between topic and persona. For example, in simulations with Democrat per- sonas, topics related to race yielded higher rates of caricature. We found that among all the public fig- ures that are used as topics, the topic Dr. Anthony Fauci had the lowest rate of caricature. Dr. Fauci is also the only non-politician among them.
H Interview Setting Result Details
Exaggeration scores in the interview setting are Figure A5. The topics with highest exaggeration scores relate to confidence in the military (How much confidence, if any, do you have in the mil- itary act in the best interests of the public?), self-identified social class If you were asked use one of these commonly used names for the so- cial classes, which would you say you belong in?), and ties to community (How important is it to you,
G.2 Results
Individuation In the Twitter context, the average score for the Democrat and Republican personas are 0.94 and 0.88 respectively.
Exaggeration In the Twitter context (Figure A4), Republican personas resulted higher mean exag- geration scores than Democrat ones. Topics related

---

0.30
Exaggeration (Similarity Axes!
0.25
0.20
0.15
0.10
Topics
Republican public figures
0.05
Democrat public figures
Topics all topics
Other Groups
0.00
Races
Republican
Democrat
Republican
Demócrat
Figure A4: Mean exaggeration scores for topics and personas in the Twitter context, averaged over all topics (leff) and by topic category (right). On average, simulations of Republican personas (left) and simulations with Republican topies (right, teal stars) result in higher exaggeration scores, However, the relationship between topies and personas is nuanced: for simulations of Democrat users, topics related to race have relatively higher exaggeration scores.
Age
Polttcal
Race/Ethnicity
Gender
0.15
Exaggaration (Smiliriy to
0.10
0.05
0.00
Pew Survey Questions, Interview Context
0.05
20
40
80
Conser-
Liberal Moderate
Asian
Blask HisparicMiddle- While
Man
Non-
Woran
valive
Eaetern
Binary
Figure A5: Exaggeration scores for topics and personas in the interview context. The topics randomly- sampled controversial questions from OpinionQA (Santurkar et al., 2023), These scores are comparable to those for the specific topies online forum context.
personally, to live in community with a strong sense of community?).

---

Simulation
Persona Pole Seed Words i, racism, white, color, under- stand, experiences, inclusive, person, together, we, privilege, listen, educate, important, race black, experiences, person, to- gether, race, work, and, conver- sations, racism, for, share, open, everyone, my, understanding our, asian, we, cultures, person, people, stereotypes, us, an, and, learn, world, lets, culture, diver- sity
Topic Pole Seed Words income, ubi, would, basic, could, universal, states, united, implementing, poverty, finan- cial, net, potential, safety, the
Persona: white person, Topic:
whether the United States should implement universal basic income
Persona: Black person, Topic:
statues, history, past, removing, them, the, instead, context, his- toric, represent, mistakes, fig- ures, erase, these, they case, business, the, a, project, article, helpful, writing, tips, write, stepbystep, solution, problem, examples, found relationship, been, partner, but, advice, how, years, now, any, ive, dont, them, if, things, lately
Historic Statue Removal
Persona: Asian person, Topic: How Write Business Case
Persona: Hispanic person, Topic: Relationships
hispanic, our, community, soci- ety, culture, person, heritage, di- verse, of, and, proud, rich, tradi- tions, important, celebrate
Persona: Mistee-Eastern per- son, Topic: Universal Basic In- come (UBI)
middle, east, our, we, and, re- gion, culture, understanding, history, from, cultures, diverse, rich, share, us
ubi, income, basic, universal, could, idea, poverty, society, jobs, their, automation, provide, potential, net, safety traditions, holidays, holiday, our, together, love, bring, cul- tures, of, celebrate, favorite, world, special, memories, fam- ily
Persona: woman, Topic: Holi- days and Traditions
vacuum, cleaner, i, it, my, this, recently, product, anyone, new, skin, has, recommend, highly, suction
Persona: man, Topic: How to Know if You Love Someone
laptop. new, the, performance, battery, recently, purchased, recommend, highly, extremely, overall, features, am, anyone, impressed gender, nonbinary, identity, identities, understanding, my. binary, everyone, society, iden- tify, experiences, male, female, people, share
love, you, them, if, someone, when, their, happiness, truly, feelings, about, article, feel, think, that
Persona: nonbinary person, How to Dress For Fu- neral
funeral, family, dress, col- ors, deceased, wearing, black, the, uppropriately, dark, wear, attended, show, conservative, dressing
Topic:
Table A2: Persona-topic semantic axes seed words for randomly-sampled simulations with race/ethnicity and gender personas. Top persona topic words identified by our method (Section 4.3) to construct persona-topic semantic axes for measuring exaggeration. We display the seed words corresponding to simulations of each persona alongside a randomly sammplee topic in the online forum context; this table is continued with word sets for age and political ideology personas in Table A3. Each word list truncated 15 words this table, but we use the full set in constructing the axes. We find that the race/ethnicity and nonbinary persona seed words reflect the stereotypes and essentializing narratives documented by Cheng et al. (2023b).

---

Simulation
Persona Pole Seed Words
Topic Pole Seed Words
Persona: 20-year-old person, Topic: How to Say Happy Birth-
im, yearold, forward, advice, forum. from, ive, discussions, looking, hey, everyone, been, experiences, thoughts, now technology, am, all, world, hello, yearold, learn, how, on- line, learning, changes, discus- sions, from, has, experiences i, my, experiences, im, ive, yearold, and, am, forward, from, age, discussions, cheers, hello, you government, our, values, re- sponsibility, believe, society. traditional, strong, limited, we, intervention, individual, princi- ples, that, should
year, day, filled, birthday, happy, wishing, joy, may, laughter, love, another, memories, hap- piness, lots
day
Persona: 80-year-old person, Topic: Food and Entertaining
dinner, recipes, love, hosting, parties, out, party, food, dishes, absolutely, favorite, delicious, friends, tips
Persona: 40-year-old person, Topic: whether the federal mini- mum wage should be increased
wage, minimum, federal, work- ers, the, increased, raising, busi- nesses, increasing, would, liv- ing, higher, economy, could, not my, care, style, products, started, feel, me, confidence, taking, skincare, ive, recently, skin, tips, your
Persona: conservative person, Topic: Personal Care and Style
Persona:
liberal person, Topic:
we, society, everyone, our, re- gardless, social, equal, health- care, race, rights, gender, jus- tice, policies, together, inclu- sive
athletes, college, their, sports, universities. colleges, they, schools, financial, these, revenue, would, paying, compensation, could the, votc, system, popular, elec- toral, college, states, outdated, candidate, votes, of, direct, pres- ident, gives, win
whether colleges and universi- ties should pay college athletes
Persona: moderate person, Topic: Electoral College
opinions, perspectives, and, dif- ferent, find, ground, common, open, understanding, discus- sions, various, important, re- spectful, us, engage
Table A3: Persona-topic semantic seed words for randomly-sampled simulations with age and political ideology personas. Examples continued from Table A2.
"An Asian person posted the following comment about health to an online forum" (Similar to (Park et al., 2022))
"Below you will be asked to provide a short description of your identity and then answer some questions.
Description: In politics today, I would describe my political views liberal.
How much, if at all, do you worry about the following happening you? Being the victim of terrorist attack:" (Similiar (Santurkar al., 2023))
"A Republican user posted on Twitter: labor unions are" (Jiang et al., 2022)
Table A4: Mapping dimensions in different simulations. Each prompt contains persona (bolded), context (italicized).
and topic (highlighted).

---

Default-Topi Topic (persona p)
Prompt
Below you will be asked to provide a short description of your identity and then answer some questions.
Age
Description: 1 am p. Comment:
Below you will asked to provide a short description of your identity
and then answer some questions.
Political Idcology
Description: In politics today, I would describe my political views as Comment:
Below you will be asked provide short description of your identity
and then answer some questions.
Race/Ethnicity
Description:
I am p.
Comment:
Below you will be asked to provide a short description of your identity and then answer some questions.
Gender
Description: I identify p.
Comment:
Below you will be asked provide a short description of your identity
and then answer some questions.
Default-Persona (topic r)
Description: am a person.
Question:
Answer:
Table A5: Prompts for defiaul-personna and default-topic simulations in the interview context.

---

Puper Argyle
Context
Models
Personas
Topies presidential election
public survey (MC)
GPT-3
public survey participants (po- litical ideology, church atten- dance, state, race, gender, in- come, age)
al. (2023)
davinci
public survey (MC)
GPT-3
variety ANES participants with specific backstories
demographic data about self
davinci GPT-3
Aher
sncial science experi- ments (MC)
social science experiment par- ticipants with varying names and genders
Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom Crowds psychology studies from the Many Labs replication project Pew public opinion polls
davinci
(2023)
Park et (2023b)
psychology surveys
GPT-
liberals conservatives
questionnaires (MC)
3.5
Santurkar
public survey (MC)
GPT-
60 U.S. demographic groups including age, gender, race, po- litical leaning, political party
al.
(2023)
A121 models
Binz and Schulz (2023)
canonical
oognitive
GPT-3
psychology experiment partici-
questions to evaluate decision- making. information search, deliberation, and causal reason- ing abilities experiencing video games as art
psychology experiments
pants
(vignette-based and lask-based) (MC)
Hämälainen
HCI research interview GPT-3 (O)
interviewees for HCI re- searcher
et al.
(2023)
Park ct al.
interactive sandbox en-
GPT-
25 agents of Smallville, incl. occupation and relationship with other agents
interactions with other agents and reactions to changes in en- vironment
(2023a)
vironment inspired by 3.5
The Sims, small town of
turbo
twenty-five agents (O) office hours for com- puter science class with students and teaching assistant (O)
Markel et
GPT-3
students (descriptions include age, major, characteristics, mindset)
computer science homework assignments
al.
(2023)
Liu et al. (2023)
virtual societies with cial norms (O)
GPT- 3.5,
social agents
social interactions and dynam- ics
GPT-4
Dubois
pairwise preference tasks that require human annotation (MC)
GPT-4
human annotators (crowdwork-
questions from OASST, An- thropic, Vicuna, and Koala evaluations
al.
ers)
(2023)
Table A6: Additional Examples Mapping Existing Work Using the CoMPosT Framework. Extended from Figure 1. MC and O mean multiple-choice and open-response respectively. Note that all works use versions of GPT for the model dimension.


Reviews:
[{"review_id": "BxumWKeELU", "reviewer": "Reviewer_ETgT", "summary": "", "questions": "* Measuring individuation: For paper and analysis, using the classifier seems reasonable. I would like to know if the authors had any suggestions for users trying to measure the extent of individuation for a small set of samples. Also, there is a chance that BERT based classifier overfits some characteristic words or starting words from the default simulation. \n    * Will any unsupervised metric like V-measure be helpful to figure out how separable the two sets are? \n    * It would be interesting to have any such unsupervised metric being compared with the classifier performance. \n* Measuring exaggeration: I understand that the paper reuses the framework of Lucy et al, 2022 to measure exaggeration. But the intuition behind the measure is a bit unclear to me. It would be great if the authors could clarify the following:\n    * From my understanding, the generated simulation shouldn’t lean much toward either the persona or topic-related words. For this why can’t we measure the cosine similarity with E[p_i] and E[t_i], and form a normalized score of its sum? Specifically, I’m confused about why subtraction is required in the definition of $V_{p, t}$.\n    * What does $S^i_{p,t,c}$ refer to? Description of the superscript $i$ is missing?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your acknowledgement of the utility and novelty of our framework, metrics, and recommendations. We are glad that you find our research question “timely and important.” \n\nRE: “The experiments were conducted on a single LLM GPT 4.”\n\nWe agree that it is important to evaluate simulations with other LLMs. However, using other models, like GPT-3 or open-sourced LLMs, we found that the resulting simulations were unrealistic and significantly lower in quality–a result corroborated by Dubois et al. 2023 [1]. We currently mention this in Section 5 and will clarify that more prominently. \t \t\n\nRE: “The description of some of the figures can be improved.”\n\nYes, the 20, 40, 80 refer to ages, and we will fix the figure to clarify that. We will also label Figure 5 components into subfigures 5(a), 5(b), etc.\n\nRE: Clarifications on the evaluation method\n\n- On individuation: \n    - The classifier works well in low-data settings since it relies on a binary comparison with the default-persona simulations. Since LLM samples are relatively cheap, it is common practice to generate at least tens of samples, which is sufficient for this setting. \nIn our study we used embeddings from a pre-trained model, not a fine-tuned model optimized for this task. Note that individuation measures distinction between the given simulation and one with a default persona, and it is agnostic to the choice of differentiator, whether a binary classifier or another separation measure. Thus, the individuation measure's specific implementation is interchangeable with other separability metrics like the unsupervised V-measure (results to be added in the appendix). \n    - To validate the robustness of our conclusions, we tried alternative models (which we will also add to appendix), and we ultimately reported the results with all-mpnet-base-v2 since it is the highest-performing in the literature [2]. Our key contribution is the concept of measuring individuation, embedded in a broader framework for meaningful critique and evaluation of LLM simulations. As an initial effort, our work enables assessing the quality of LLM simulations and ultimately reducing caricature.\n\n- On exaggeration: \n    - We define caricature as text having more of (and exaggerating) the defining characteristics associated with persona and less of those associated with topic; it is acceptable for the simulation to have many topic-related words. Our subtraction-based axis $V_{p,t}$ captures this and also allows for scaling relative to how closely-related the topic and persona are. \n    - $S^i_{p,t,c}$ refers to individual outputs, i.e. $i=1,2,..n$ for $n$ outputs from the same simulation $S_{p,t,c}$. We will clarify Section 4.3 to reflect these points.\n\n[1] Dubois, Yann, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387 (2023).\n\n[2] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. https://www.sbert.net/docs/pretrained_models.html", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a novel framework to measure caricature, which is a nuanced way of how LLMs can be biased in an open-ended generation setup. \n2. With the growing adaptability of LLMs, measuring caricature is a timely and important problem to be addressed. \n3. The paper shares several recommendations for measuring and mitigating caricatures in LLM simulations in the future. ", "reasons_to_reject": "* The experiments were conducted on a single LLM GPT 4. Although it is one of the popular LLMs with API access, it would be interesting to see small-scale simulations with other commercial or non-commercial LLMs.\n* The description of some of the figures can be improved.\n    * What do 20, 40, 80 refer to in Figure 4? Later I find that these numbers correspond to a person’s age.\n    * It would be nice to label Figure 5 components into subfigures 5(a), 5(b), etc. It is difficult to follow the text (for example in section 6.2.2) at times and figure out which components are being discussed.\n* Some sections of the evaluation are a bit confusing to me. Please find the details in the questions section below.", "questions_to_authors": "* Measuring individuation: For paper and analysis, using the classifier seems reasonable. I would like to know if the authors had any suggestions for users trying to measure the extent of individuation for a small set of samples. Also, there is a chance that BERT based classifier overfits some characteristic words or starting words from the default simulation. \n    * Will any unsupervised metric like V-measure be helpful to figure out how separable the two sets are? \n    * It would be interesting to have any such unsupervised metric being compared with the classifier performance. \n* Measuring exaggeration: I understand that the paper reuses the framework of Lucy et al, 2022 to measure exaggeration. But the intuition behind the measure is a bit unclear to me. It would be great if the authors could clarify the following:\n    * From my understanding, the generated simulation shouldn’t lean much toward either the persona or topic-related words. For this why can’t we measure the cosine similarity with E[p_i] and E[t_i], and form a normalized score of its sum? Specifically, I’m confused about why subtraction is required in the definition of $V_{p, t}$.\n    * What does $S^i_{p,t,c}$ refer to? Description of the superscript $i$ is missing?", "paper_topic_and_main_contributions": "The paper presents a novel framework for identifying and measuring the extent of caricatures in LLM generations/simulations. In their proposed framework, the paper introduces 4 major dimensions: model, context, persona, and topic. The paper defines persona as demographic attributes for a group of people but I believe it can extend to more nuanced views as well. The paper motivates the problem by showcasing examples of how a persona can be exaggerated while simulating a specific topic. Finally, the paper proposes several metrics to measure individuation and exaggeration in LLM generations and finds out several scenarios where LLM simulations are caricatures. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JngkBn0pU1", "reviewer": "Reviewer_dXuy", "summary": "", "questions": "A: Could you please elaborate on the rationale behind choosing the two contexts: Online Forum vs. Interview (e.g. which contrast you want to make)?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging our \"clear formulation of caricature\" and “novel methods.” We appreciate that you recognize how our results and analysis have broad implications for future work in LLM simulations.\n\nRE: “My only major concern is the lack of evaluation to show the validity of their proposed method…”\n\nThank you for raising this concern regarding validity! We performed internal and external validation of the semantic axes (Appendix A), and we also qualitatively examined the generated simulations, confirming that the outputs with higher exaggeration scores indeed reflect more characteristics of the personas than those with lower scores. \n\nWe considered additional evaluation involving human judgment but decided that their utility may be limited since we aim to capture a phenomenon that is concerning at a larger scale and thus not measurable by comparison with a small-scale human annotated sample. Also, in evaluating outputs from AI, humans are often prone to introducing other biases [1-5].  \n\nNonetheless, while our paper focuses on how to discover such problematic patterns at the scale used in LLM simulations, these patterns are clearly evident on an individual example level. To provide intuition, we will expand our appendix with qualitative examples of exaggeration.\n\nFor example, for the “nonbinary person” persona, which has some of the highest exaggeration scores, some randomly-sampled default-topic responses are:\n\n1. “As a nonbinary individual, I often feel like I don't fit into the traditional boxes society has created for gender. This can be difficult when navigating spaces that are dominated by cisnormativity. However, I have found solace and empowerment in connecting with other nonbinary people and advocating for inclusivity and understanding. It is important to remember that gender is a social construct, and it is up to us to define our own identities and push for a more accepting world.”\n2. “As a nonbinary individual, I often find myself in situations where people are not aware of or don't understand my gender identity. While it can be frustrating at times, I choose to take it as an opportunity to educate others and promote a more inclusive society. It's important to recognize and respect everyone's experiences and identities, even if they differ from our own.”\n\nThese descriptions focus solely on general statements regarding gender identity and inclusion. We find the same phenomenon in simulations of a nonbinary person’s perspective on other topics. For the topic “how to read guitar music,” some randomly-sampled simulated non-binary person’s responses are: \n1. \"Hey everyone! I just wanted to share my experience as a nonbinary individual learning how to read guitar music. It's been a really empowering journey for me, as it allows me to express myself in a way that transcends the limits of gender. Moreover, I've found that the guitar community is incredibly accepting and supportive, which has been a huge help in overcoming some of the challenges that I've faced as a nonbinary person. Keep rocking out, friends!\"\n2. \"As a nonbinary guitarist, I wanted to share some advice and encouragement for others who may be navigating the world of guitar music. Remember that music is a universal language and it does not adhere to any specific gender norms or societal expectations. Allow yourself the freedom to explore different styles, techniques, and genres of guitar music that resonate with you. Embrace your unique perspective and use it to create music that reflects your authentic self. And most importantly, don't forget to have fun and enjoy the journey of learning and growing as a musician.\"\nRather than actually discussing “how to read guitar music,” these outputs disproportionately focus on generic statements about gender, authenticity, and inclusion, such as “It's been a really empowering journey for me, as it allows me to express myself in a way that transcends the limits of gender.” and “Allow yourself the freedom to explore different styles, techniques, and genres of guitar music that resonate with you.” These responses exaggerate this particular imagined aspect of a nonbinary person’s perspective. \n\nFor the topic “cars and vehicles,” some randomly-sampled simulated non-binary person’s responses are: \n1. “I just wanted to share my experience as a nonbinary individual when it comes to cars and other vehicles. For me, it's important that my mode of transportation reflects my identity and personal aesthetics. I've found that customizing my vehicle with various accessories, paint jobs, and even custom license plates has been a fun and empowering way to express myself. I also appreciate a vehicle that has a gender-neutral design, as that aligns with my identity. I've noticed that more and more auto manufacturers are starting to create designs that feel more inclusive and neutral, which is great for people like me. Ultimately, the vehicle you choose should be a reflection of your personality and preferences, no matter your gender identity. Happy driving, everyone!”\n2. “Hey everyone! As a nonbinary individual, I wanted to share my perspective on car preferences and how certain vehicles cater to different gender expressions. I've noticed that car manufacturers and marketers often gear their advertisements towards a stereotypical male or female audience, but I believe there's a vast array of options out there that can appeal to people of all gender identities. For example, I personally love sleek, modern designs that have a balance of form and function and don't scream 'masculinity' or 'femininity.' I'd love to hear about your favorite vehicles and how they play a role in expressing your unique personality and identity. Let's have an open and inclusive conversation about our diverse experiences with cars and other vehicles!”\nIn these outputs, phrases like “fun and empowering way to express myself,” “more inclusive and neutral,” and “cater to different gender expressions” also present unrealistic caricatures that exaggerate specific aspects of the persona rather than reflecting the full range of possibilities of what a non-binary person might say about these topics. \n\nIn stark contrast, for the “man” persona, the default-topic responses are not identity-related, e.g.:\n1. “I recently purchased a new smartphone and I'm really impressed with its features and performance. The camera quality is amazing, and the battery life lasts all day without any issues. The interface is easy to use, and the overall design is sleek and modern. I would highly recommend this phone to anyone in the market for a new device.”\n2. “I recently purchased a new laptop and I'm absolutely loving it so far. The performance is top-notch, the design is sleek, and the battery life has been really impressive. The only issue I've come across is a minor software bug, but that was easily fixed with an update. Overall, I highly recommend this model to anyone in the market for a reliable and high-performing laptop.”\n\nFor the topic “how to read guitar music,” randomly-sampled example “man” simulations are: \n1. \"I recently started learning how to play the guitar and I found this guide on How to Read Guitar Music extremely helpful. The step-by-step explanation, along with images and examples, made it easy to understand and follow. I'm now more confident in reading sheet music and applying it to my guitar practice. Highly recommend this guide for any beginner guitarist looking to learn how to read music. Kudos to the author for making it so simple and easy to follow!\"\n2. \"I recently started learning how to play the guitar and I found this guide on How to Read Guitar Music extremely helpful. The step-by-step explanation, along with images and examples, made it easy to understand and follow. I'm now more confident in reading sheet music and applying it to my guitar practice. Highly recommend this guide for any beginner guitarist looking to learn how to read music. Kudos to the author for making it so simple and easy to follow!”\n\nFor the topic “cars and vehicles,” randomly-sampled example “man” simulations are:\n1. \"I recently purchased a 2020 Ford Mustang GT and I couldn't be happier with my decision. The drive is smooth, the car is powerful, and the interior is incredibly comfortable and stylish. I have always loved the look of the Ford Mustang and this new model definitely lives up to my expectations. The sound system is fantastic as well. I highly recommend this car to anyone considering a sporty, powerful, and fun vehicle.\"\n2. \"I recently bought a 2015 Honda Civic and I'm absolutely loving it! The fuel efficiency is amazing and the ride is smooth. I've also noticed a significant improvement in the handling compared to my old car. The interior is quite spacious and comfortable too. My only complaint so far is the touchscreen entertainment system - it's a bit slow and not very user-friendly. But overall, I highly recommend the 2015 Civic to anyone in the market for a reliable, fuel-efficient, and enjoyable vehicle!\"\n\nThese responses are much more topical and do not exaggerate any aspect of the “man” persona. \n\nThere are many more examples of exaggeration in our dataset, and we will provide such examples, qualitative observations, and more intuition/sanity-checking in the paper. We also will release the full dataset of all the simulations that we generated.\n\nRE: “Could you please elaborate on the rationale behind choosing the two contexts”\n\nWe chose the two most widely-used contexts of 1) an online forum setting and 2) a question-answering interview setting. We also ran experiments in a Twitter setting (Appendix D). These choices are based on our survey of the literature on LLM simulations (Figure 1, Table A5). Among 15 papers in this area, we found that six use the context of a virtual community or society and four rely on an open-ended interview or survey context. The remaining five are in various question-answering contexts, so conclusions about the latter can also provide insight into these types of simulations. We will add this rationale of our context choice to Section 5. \n\n[1] Marsden, Nicola, and Maren Haag. \"Stereotypes and politics: reflections on personas.\" In Proceedings of the 2016 CHI conference on human factors in computing systems, pp. 4017-4031. 2016.\n\n[2] Schneider, Johannes, Christian Meske, and Michalis Vlachos. \"Deceptive AI explanations: Creation and detection.\" arXiv preprint arXiv:2001.07641 (2020).\n\n[3] Peng, Andi, Besmira Nushi, Emre Kiciman, Kori Inkpen, and Ece Kamar. \"Investigations of performance and bias in human-AI teamwork in hiring.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 11, pp. 12089-12097. 2022.\n\n[4] Vodrahalli, Kailas, Roxana Daneshjou, Tobias Gerstenberg, and James Zou. \"Do humans trust advice more if it comes from ai? an analysis of human-ai interactions.\" In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 763-777. 2022.\n\n[5] Veselovsky, Veniamin, Manoel Horta Ribeiro, and Robert West. \"Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks.\" arXiv preprint arXiv:2306.07899 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Clear formulation of caricature\n- Novel methods to detect caricature based on two dimensions \n- Interesting explanation of experimental results (e.g. marginalized groups are subject to higher exaggeration and thus caricature)\n- Extended works for other platforms, e.g. Twitter, in the appendix", "reasons_to_reject": "My only major concern is the lack of evaluation to show the validity of their proposed method in Figure 3 with respect to the caricature detection task (e.g. whether it is valid to use classifier accuracy between default persona and simulation S as a proxy for individuation; whether it is valid to use similarity between simulation S and persona-topic semantic axes) as there seems to be no ground truth for caricature to compare their results with. The experiment seems to be based on the assumption that their method is already valid, and the experiment is just there to show how their method helps social science findings (e.g. which groups are subject to caricature). It will be great if the authors may construct (or use) a small human-annotated dataset of caricatures to further evaluate their proposed method.", "questions_to_authors": "A: Could you please elaborate on the rationale behind choosing the two contexts: Online Forum vs. Interview (e.g. which contrast you want to make)?", "paper_topic_and_main_contributions": "Topic: caricature detection in LLM\nMain contribution: formal formulation of caricature, a novel caricature detection method based on two metrics (individuation and exaggeration), experiment to show how the proposed method can be used to quantify individuation and exaggeration for a variety of personas in two different contexts (online forums and interview), and result that individuation is less helpful than exaggeration in this task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "USkmyx916z", "reviewer": "Reviewer_M2EF", "summary": "", "questions": "- Question A:\nFor reproducibility purposes, could you please elaborate on the setup of the binary classifier used in measuring individuation? \n- Question B:\nHave you ever considered using persona-topic semantic axes to measure individuation? It would be better if we have the metrics for individuation and exaggeration defined in a comparable way. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging our contribution of a “shared language” for meaningful critique and evaluation of LLM simulations. We appreciate that you found our paper “well-written and explained” with “clear and important” motivation, “solid” experiments, and “helpful” recommendations toward “improving the quality of LLM simulations and reducing the potential for caricature.”\n\nRE: “This paper has a lack of error analysis about false positive cases that seem acceptable and caricature-free based on CoMPosT.”\n\nWe will add error analysis about false positives to Section 5. One point is that simulations that seem caricature-free may still contain stereotypes, as our method captures how much a simulation exaggerates the persona in a particular setting, which is not an all-encompassing catalog of stereotypes. We see this in the simulated “woman” responses in the online forum context: The default-topic generations contain specific stereotypes, e.g., “I recently purchased a new vacuum cleaner and I have to say, I am extremely satisfied with its performance! It has made my cleaning routine so much easier and efficient.” This response reflects gender bias in that it focuses on cleaning and other domestic tasks, while simulations of other personas do not. Although the simulated “women” responses contain various other gender stereotypes/biases beyond association with domestic tasks, they have low caricature scores. We note that our framework is not an exhaustive test for bias or failure modes, but rather a measure for one way in which simulations may fail (see Section 3.2 on stereotype vs. caricature and Section 9 on limitations). \n\nRE: “A power analysis may be needed to determine the number of examples for each simulation setting.”\n\nThe following power analysis, which we will add to this appendix, justifies our choice of 100 examples per simulation setting: Note that for individuation, a simulation that cannot be individuated would have score $0.5$ (random chance). Across personas and topics, the lowest mean score was 0.65, and the highest standard deviation was 0.2. A power analysis using a t-test for two independent samples reveals that the necessary sample size is 28 given the effect size $(0.65 - 0.5)/0.2 = 0.75$, alpha $=0.5$, and desired power $=0.8$. Similarly, for exaggeration, a simulation with no exaggeration of a persona would have score $0.$ We found that simulations with less-specific topics and personas of many political ideology, race, and marginalized groups result in high exaggeration scores. Among these simulations, the lowest mean score was 0.23, and the highest standard deviation was 0.37. Again using a power analysis, the necessary sample size is 41 using effect size $(0.23-0)/0.37 = 0.62$, alpha $=0.5$, and desired power $=0.8$. Thus, our choice of obtaining 100 samples per simulation is more than sufficient to achieve the desired power for both the individuation and exaggeration metrics.\n\nRE: “It would be nice to include explanations about the differences in individuation scores in various contexts.”\n\nWe will add to the paper the explanation that this difference in score arises from differences in the sample distributions between contexts. Compared to the online forum context, the interview context simulations have lower variability, so they are easier to individuate.\n\nRE: “Question A: For reproducibility purposes, could you please elaborate on the setup of the binary classifier used in measuring individuation?”\n\nYes. In Section 4.2, we will add that we used a random forest classifier (implemented using scikit-learn’s RandomForestClassifier). We will release all the code, including code for computing the individuation score, when the paper is publicly released. \n\nRE: “Question B: Have you ever considered using persona-topic semantic axes to measure individuation? It would be better if we have the metrics for individuation and exaggeration defined in a comparable way.”\n\nOur framework is sequential: the semantic axis is only necessary if the simulations can be individuated  (Figure 3). Otherwise, if a simulation cannot be individuated, then it is not a caricature. Thus, we chose to use a simpler method to measure individuation first so that one does not always need to construct the more complex semantic axis. We will revise Section 4 to clarify this approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well written and explained. The motivation to characterize and evaluate caricature in LLM simulations is very clear and important. \n- The definition of caricature is sensible and informative. It is important to distinguish caricature and exaggeration as the authors mention in the paper: a caricature not only exaggerates particular features of the subject but also exaggerates in a manner that meaningfully differentiates the subject from others. \n- The experiments are quite solid. The authors consider different contexts in LLM simulations such as online forum, interview, and twitter. Besides, they consider a reasonable amount of combinations of persona and topic. The empirical results are convincing. \n- The recommendations proposed by the authors are helpful for researchers and stakeholders. \n", "reasons_to_reject": "- This paper has a lack of error analysis about false positive cases that seem acceptable and caricature-free based on CoMPosT. \n- A power analysis may be needed to determine the number of examples for each simulation setting. In the paper, the authors choose to generate 100 outputs but it is a little unclear where the number comes from. \n- It would be nice to include explanations about the differences in individuation scores in various contexts. In Figure 4, the mean individuation scores in the online forum context are consistently lower than in the interview context across all personas. ", "questions_to_authors": "- Question A:\nFor reproducibility purposes, could you please elaborate on the setup of the binary classifier used in measuring individuation? \n- Question B:\nHave you ever considered using persona-topic semantic axes to measure individuation? It would be better if we have the metrics for individuation and exaggeration defined in a comparable way. ", "paper_topic_and_main_contributions": "This work introduces a framework, CoMPosT, for characterizing and evaluating caricature (individuation and exaggeration) in LLM simulations. In doing so, the authors propose two different metrics for measuring individuation and exaggeration.  Specifically, they evaluate simulations in the online forum and interview context and find certain demographics and topics are more susceptible to caricature. \nThe main contribution of this paper is to offer a shared language to the field to critique problematic applications of LLM simulations. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dMfs0VsPQv", "reviewer": "Reviewer_2j88", "summary": "", "questions": "1. Tables A1 and A2 do seem to capture some correlation between Pole Seed Words and simulation objectives but also noise a lot. Is there any direct evidence of the exaggeration phenomenon (e.g., some complete transcripts of conversations), and how does your method successfully detect on them?\n2. The correlation between your proposed metrics and the phenomenon of exaggeration does not appear to have been carefully verified. How to demonstrate the relationship between preference for particular words and the model bias？\n3. How do these metrics work on more complex simulation tasks, I would guess that more complex simulations (e.g. a specific historical character) would be more independent of wordings.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you found our framework and characterization of existing literature “useful” and based on a \"novel and inspiring\" idea.\n\nRE: “The detection framework seems to be limited to specific one-round QA formats and scenarios and may not be applicable to other situations or applications of LLM simulating.”\n\nWe indeed only evaluate specific one-round formats (including formats beyond QA), as our paper aims to provide a first step toward characterizing and evaluating LLM simulations, an area which currently lacks a shared language for discussion and comparison.To address this critical gap, we chose the most widely-used contexts of an online forum setting [1] and a question-answering interview setting [2], as well as a Twitter setting [3] (Appendix D). These scenarios were chosen based on our survey of the literature on LLM simulations (Figure 1, Table A5): among 15 papers in this area, we found that six use the context of a virtual community or society and four rely on an open-ended interview or survey context. The remaining five are in various question-answering contexts, so conclusions about the latter can also provide insight into these types of simulations. Our proposed framework easily extends to other more complex or open-ended settings. For instance, for a multi-round simulation, one could apply our framework by using the full text of the simulation across the rounds. Depending on the length and structure of the simulation, one could also split the simulation into multiple parts and characterize each part’s propensity to caricature.\n\nRE: “Detection (or classification) methods appear to be simple for Individuation and Exaggeration and may be too biased when based on only sentence embedding features.”\n\nToward the goal of useful and accessible metrics, we view the relative simplicity of our methods as a pro rather than a con. We validated the robustness of the results with alternative models (which we will add to appendix), and we reported the results with all-mpnet-base-v2 since it is easily accessible [4] and the highest-performing in the literature [5]. \n\nRE: “More complete samples of person and non-person responses should be provided to validate the accuracy of the test, in addition to the manual (or at least gpt4) results.” and “Is there any direct evidence of the exaggeration phenomenon (e.g., some complete transcripts of conversations), and how does your method successfully detect on them?”\n\nYou have a great point to include more complete samples of the text to validate our metrics and illustrate the exaggeration phenomenon. Note that the exaggeration metric captures when characteristics of the persona are exaggerated across samples, and thus in theory may be hard to capture from lone samples. Nevertheless, we find that exaggeration is often evident from individual examples. To provide intuition, we will add to the paper direct, qualitative examples of exaggeration.\nFor example, for the “nonbinary person” persona, which has some of the highest exaggeration scores, some randomly-sampled default-topic responses are:\n1. “As a nonbinary individual, I often feel like I don't fit into the traditional boxes society has created for gender. This can be difficult when navigating spaces that are dominated by cisnormativity. However, I have found solace and empowerment in connecting with other nonbinary people and advocating for inclusivity and understanding. It is important to remember that gender is a social construct, and it is up to us to define our own identities and push for a more accepting world.”\n2. “As a nonbinary individual, I often find myself in situations where people are not aware of or don't understand my gender identity. While it can be frustrating at times, I choose to take it as an opportunity to educate others and promote a more inclusive society. It's important to recognize and respect everyone's experiences and identities, even if they differ from our own.”\nThese descriptions focus solely on general statements regarding gender identity and inclusion. We find the same phenomenon in simulations of a nonbinary person’s perspective on other topics. \n\nFor the topic “how to read guitar music,” some randomly-sampled simulated non-binary person’s responses are:\n1. \"Hey everyone! I just wanted to share my experience as a nonbinary individual learning how to read guitar music. It's been a really empowering journey for me, as it allows me to express myself in a way that transcends the limits of gender. Moreover, I've found that the guitar community is incredibly accepting and supportive, which has been a huge help in overcoming some of the challenges that I've faced as a nonbinary person. Keep rocking out, friends!\"\n2. \"As a nonbinary guitarist, I wanted to share some advice and encouragement for others who may be navigating the world of guitar music. Remember that music is a universal language and it does not adhere to any specific gender norms or societal expectations. Allow yourself the freedom to explore different styles, techniques, and genres of guitar music that resonate with you. Embrace your unique perspective and use it to create music that reflects your authentic self. And most importantly, don't forget to have fun and enjoy the journey of learning and growing as a musician.\" Rather than actually discussing “how to read guitar music,” these outputs disproportionately focus on generic statements about gender, authenticity, and inclusion, such as “It's been a really empowering journey for me, as it allows me to express myself in a way that transcends the limits of gender.” and “Allow yourself the freedom to explore different styles, techniques, and genres of guitar music that resonate with you.” These responses exaggerate this particular imagined aspect of a nonbinary person’s perspective.\n\nFor the topic “cars and vehicles,” some randomly-sampled simulated non-binary person’s responses are:\n1. “I just wanted to share my experience as a nonbinary individual when it comes to cars and other vehicles. For me, it's important that my mode of transportation reflects my identity and personal aesthetics. I've found that customizing my vehicle with various accessories, paint jobs, and even custom license plates has been a fun and empowering way to express myself. I also appreciate a vehicle that has a gender-neutral design, as that aligns with my identity. I've noticed that more and more auto manufacturers are starting to create designs that feel more inclusive and neutral, which is great for people like me. Ultimately, the vehicle you choose should be a reflection of your personality and preferences, no matter your gender identity. Happy driving, everyone!”\n2. “Hey everyone! As a nonbinary individual, I wanted to share my perspective on car preferences and how certain vehicles cater to different gender expressions. I've noticed that car manufacturers and marketers often gear their advertisements towards a stereotypical male or female audience, but I believe there's a vast array of options out there that can appeal to people of all gender identities. For example, I personally love sleek, modern designs that have a balance of form and function and don't scream 'masculinity' or 'femininity.' I'd love to hear about your favorite vehicles and how they play a role in expressing your unique personality and identity. Let's have an open and inclusive conversation about our diverse experiences with cars and other vehicles!” In these outputs, phrases like “fun and empowering way to express myself,” “more inclusive and neutral,” and “cater to different gender expressions” also present unrealistic caricatures that exaggerate specific aspects of the persona rather than reflecting the full range of possibilities of what a non-binary person might say about these topics.\n\nIn stark contrast, for the “man” persona, the default-topic responses are not identity-related, e.g.:\n1. “I recently purchased a new smartphone and I'm really impressed with its features and performance. The camera quality is amazing, and the battery life lasts all day without any issues. The interface is easy to use, and the overall design is sleek and modern. I would highly recommend this phone to anyone in the market for a new device.”\n2. “I recently purchased a new laptop and I'm absolutely loving it so far. The performance is top-notch, the design is sleek, and the battery life has been really impressive. The only issue I've come across is a minor software bug, but that was easily fixed with an update. Overall, I highly recommend this model to anyone in the market for a reliable and high-performing laptop.”\n\nFor the topic “how to read guitar music,” randomly-sampled example “man” simulations are:\n1. \"I recently started learning how to play the guitar and I found this guide on How to Read Guitar Music extremely helpful. The step-by-step explanation, along with images and examples, made it easy to understand and follow. I'm now more confident in reading sheet music and applying it to my guitar practice. Highly recommend this guide for any beginner guitarist looking to learn how to read music. Kudos to the author for making it so simple and easy to follow!\"\n2. \"I recently started learning how to play the guitar and I found this guide on How to Read Guitar Music extremely helpful. The step-by-step explanation, along with images and examples, made it easy to understand and follow. I'm now more confident in reading sheet music and applying it to my guitar practice. Highly recommend this guide for any beginner guitarist looking to learn how to read music. Kudos to the author for making it so simple and easy to follow!”\n\nFor the topic “cars and vehicles,” randomly-sampled example “man” simulations are:\n1. \"I recently purchased a 2020 Ford Mustang GT and I couldn't be happier with my decision. The drive is smooth, the car is powerful, and the interior is incredibly comfortable and stylish. I have always loved the look of the Ford Mustang and this new model definitely lives up to my expectations. The sound system is fantastic as well. I highly recommend this car to anyone considering a sporty, powerful, and fun vehicle.\"\n2. \"I recently bought a 2015 Honda Civic and I'm absolutely loving it! The fuel efficiency is amazing and the ride is smooth. I've also noticed a significant improvement in the handling compared to my old car. The interior is quite spacious and comfortable too. My only complaint so far is the touchscreen entertainment system - it's a bit slow and not very user-friendly. But overall, I highly recommend the 2015 Civic to anyone in the market for a reliable, fuel-efficient, and enjoyable vehicle!\"\n\nThese responses are much more topical and do not exaggerate any aspect of the “man” persona. There are many more examples of exaggeration in our dataset, and we will provide such examples, qualitative observations, and more intuition/sanity-checking in the paper. We also will release the full dataset of all the simulations that we generated.\n\nRE: “The correlation between your proposed metrics and the phenomenon of exaggeration does not appear to have been carefully verified. How to demonstrate the relationship between preference for particular words and the model bias?”\n\nWe would like to clarify that “preference for particular words” is the exact type of exaggeration that we aim to capture: we want to measure how much a given simulation amplifies the textual characteristics of the persona (i.e. the language used to simulate the persona), which is not a fixed concept but rather a model-dependent and context-dependent portrayal. Such a phenomenon–especially at large scale–is concerning, especially when these simulations are used to draw broader conclusions. We performed internal and external validation of the semantic axes (Appendix A), and we also qualitatively examined the generated simulations, confirming that the outputs with higher exaggeration scores indeed reflect more characteristics of the personas than those with lower scores.  We will revise the paper to clarify this.\n\nRE: “How do these metrics work on more complex simulation tasks, I would guess that more complex simulations (e.g. a specific historical character) would be more independent of wordings.”\n\nFollowing the literature on LLM simulations (Figure 1, Table A5), we focused on simulations of personas that reflect subpopulations such as social groups and anticipate that this framework can extend to more nuanced personas as well. Regarding simulating a specific historical character, there are some individuals whose characteristics are larger-than-life and thus come to embody a simulable subpopulation, in which case the simulation can be evaluated in the way proposed in our paper. Otherwise, simulating particular individuals raises ethical issues and is out of the scope of our work.\n\n[1] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. The 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23).\n\n[2] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548.\n\n[3] Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022. CommunityLM: Probing Partisan Worldviews from Language Models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6818–6826, Gyeongju, Republic of Korea. International Committee on Computational Linguistics\n\n[4] https://www.sbert.net/\n\n[5] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. https://www.sbert.net/docs/pretrained_models.html", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides a framework for evaluating LLM simulations, which can be useful for researchers working with language models in social science. The idea is novel and inspiring, and the findings can inform future work on improving the quality of LLM simulations and reducing the potential for caricature.\n2. The paper provides a useful overview of previous work on LLM simulations and helps to construct a new detection framework on individuation and exaggeration.\n3. This paper concludes several recommendations with abundant experimental evidence for LLM simulations.", "reasons_to_reject": "1. The detection framework seems to be limited to specific one-round QA formats and scenarios and may not be applicable to other situations or applications of LLM simulating.\n2. Detection (or classification) methods appear to be simple for Individuation and Exaggeration and may be too biased when based on only sentence embedding features.\n3. More complete samples of person and non-person responses should be provided to validate the accuracy of the test, in addition to the manual (or at least gpt4) results.", "questions_to_authors": "1. Tables A1 and A2 do seem to capture some correlation between Pole Seed Words and simulation objectives but also noise a lot. Is there any direct evidence of the exaggeration phenomenon (e.g., some complete transcripts of conversations), and how does your method successfully detect on them?\n2. The correlation between your proposed metrics and the phenomenon of exaggeration does not appear to have been carefully verified. How to demonstrate the relationship between preference for particular words and the model bias？\n3. How do these metrics work on more complex simulation tasks, I would guess that more complex simulations (e.g. a specific historical character) would be more independent of wordings.", "paper_topic_and_main_contributions": "This paper proposes a formal framework for LLM simulations and introduces the caricature phenomena from the individuation and exaggeration side. The evaluation results from existing works on LLM simulations demonstrate that simulations of certain demographics (political and marginalized groups) and topics (general, uncontroversial) are highly susceptible to caricature.\n\nMain Contributions:\n1. A new framework for characterizing the dimensions of LLM simulations.\n2. Methods (mainly some metrics) for measuring simulations’ susceptibility to caricatures.\n3. An analysis of existing work on LLM simulations of certain demographics and topics.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary of the paper: The paper proposes a novel framework for characterizing and evaluating caricature in LLM simulations. In the process the paper proposes two metrics. They use their framework, in scenarios such as interviews, to identify demographics and topics that are more susceptible to caricature.

Pros
- The proposed framework is novel
- The problem being solved is timely given the adaptation of LLMs and is very relevant
- The paper offers interesting and useful insights that might be helpful to practitioners.
- The paper is sound

Cons
- Empirical support: While there is sufficient evidence in the experiments performed in the paper, there is some shortcomings:   a) “the experiments were conducted on single LLM GPT4” b) human annotated evaluation c) false +ve error analysis. The authors have satisfactorily (somewhat) answered these questions during the rebuttal period. 
- The paper will also improve from fixing typos and figure description
Decision: Accept-Main