In-Context Impersonation Reveals Large Language Models' Strengths and Biases
Authors: Leonard Salewski
Stephan Alaniz
Isabel Rio-Torto
Eric Schulz
Zeynep Akata
Venue: NeurIPS 2023 spotlight (2023)
Keywords: large language models
impersonation
vision language models
reasoning
TL;DR: Large language models can impersonate different personas and it affects their performance in bandit, reasoning and vision and language tasks.

Abstract: In everyday conversations, humans can take on different roles and adapt their vocabulary to their chosen roles. We explore whether LLMs can take on, that is impersonate, different roles when they generate text in-context. We ask LLMs to assume different personas before solving vision and language tasks. We do this by prefixing the prompt with a persona that is associated either with a social identity or domain expertise. In a multi-armed bandit task, we find that LLMs pretending to be children of different ages recover human-like developmental stages of exploration. In a language-based reasoning task, we find that LLMs impersonating domain experts perform better than LLMs impersonating non-domain experts. Finally, we test whether LLMs' impersonations are complementary to visual information when describing different categories. We find that impersonation can improve performance: an LLM prompted to be a bird expert describes birds better than one prompted to be a car expert. However, impersonation can also uncover LLMs' biases: an LLM prompted to be a man describes cars better than one prompted to be a woman. These findings demonstrate that LLMs are capable of taking on diverse roles and that this in-context impersonation can be used to uncover their strengths and hidden biases. Our code is available at https://github.com/ExplainableML/in-context-impersonation.

In-Context Impersonation Reveals Large Language Models' Strengths and Biases
Leonard Salewski 1,2
Stephan Alaniz 1.2
Isabel 3,4+
Eric Schulz
Zeynep Akata 12
University of Tübingen
2 Tübingen AI Center University of Porto Max Planck Institute for Biological Cybernetics
INESC TEC
Abstract
everyday conversations, humans can take on different roles and adapt their vocabulary to their chosen roles, We explore whether LLMs can take on, that is impersonate, different roles when they generate text in-context. We ask LLMs to assume different personas before solving vision and language tasks. We do this by prefixing the prompt with persona that is associated either with a social identity or domain expertise. In multi-armed bandit task, we find that LLMs pretending to be children of different ages recover human-like developmental stages of exploration. In language- based reasoning task, find that LLMs impersonating domain experts perform better than LLMs impersonating non-domain experts. Finally, we test whether LLMs' impersonations are complementary visual information when describing different categories. We find that impersonation can improve performance: LLM prompted be bird expert describes birds better than one prompted to be car expert. However, impersonation can also uncover LLMs' biases: an LLM prompted to be a man describes cars better than one prompted to be a woman. These findings demonstrate that LLMs are capable of taking on diverse roles and that this in-context impersonation can be used uncover their strengths and hidden biases. Our code is available at sttpps//// thub. com/ ExplainableML/Iie iewiiiii--ot
contextttttttt-tttttoooooooooeee context- impersonation -
1 Introduction
Large Language Models (LLMs) can not only summarize documents and converse on large range of topics [T) but they have also shown other emergent abilities 2 Because their impressive abilities, LLMs are permeating into many applications 5).). This means that there is societal need understand how these models "tick" [6] 7) Traditionally. LLMs are provided with context as textual prompt and are asked to provide answers via text completion, thereby solving variety of choice-based (8) description based [9, and reasoning tasks [00 Yet how in-context learning works is not fully understood. When Min et al. [1] prompted LLMSs with random labels, they found that this did not drastically degrade performance, suggesting that the role of in-context demonstrations to prime the model for particular task. This is in line with other results suggesting that LLMs internally infer latent variables to make better predictions [2]. It has been suggested that LLMs, and other large models, can change their behavior when asked to respond Deshpande et al. [3] asked LLMs to respond as a hateful person, their toxicity score increased. When Wang and colleagues [[4] asked LLMs to imagine being expert systematic reviewers. the
particular persona. When
WWrk done during research visit the University of Tübingen
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

quality of their literature search queries increased. That LLMs can impersonate specific people also known; they can, for example, pretend to be Oscar Wilde, Carrie Bradshaw from Sex and the City, or Donald Trump 15 But how does in-context impersonation affect LLMSs behavior in language-based and other downstream tasks?
In the current work, we let LLMs impersonate, that is taking on different roles, in context. We do this by prefixing the prompt with you were a [persona]" where persona replaced with the persona that the LLM is asked to impersonate. These personas are associated either with a social identity or a domain of expertise. In a first simulation using a multi-armes bandit task [16]. we find that LLMs impersonating children of different ages can recover the developmental stages of human- like exploration strategies. In language-barr based reasoning tasks, we find that LLMs impersonating domain experts perform better than LLMs impersonating non-domain experts. Finally, we ask LLMs to describe different classes of either birds or cars and then use their descriptions in downstream, visual classification task. The results of this experiment corroborate our earlier results: LLMs become better as they pretend to be older, and they are also better when they pretend to be domain experts. However, we also see how impersonating LLMs reproduce biases: LLMs impersonating black person or a male describe cars better, while LLMs impersonating a white person or a female describe birds better. These results expand our understanding of in-context learning in LLMs and open up new research directions investigating role-taking and pretense in LLMs and beyond.
2 Related Work
In-context learning refers to an LLM's ability to improve at a given task after being provided with a number of task-relevant demonstrations [I]. This ability sets LLMs apart from traditional models and has led to a totally new paradigm- one which does not require fine-tumin tuning of weights on task-specific data but instead relies entirely on contextual information [17 [8).
This contextual information is normally delivered as textual prompts [9), where a task or scenario is described and model is asked to solve the task or reason about the scenario by generating the next words of the provided text. Due to its flexibility, prompting has been widely used as a generic method for natural language tasks 22 (2). Importantly, the resulting in-context learning does not only work after LLMs have seen some examples, i.c. in the few-shot regime [22], but also without any examples, i.e. in the zero-shot regime (23). LLMS reasonably proficient at solving arithmetic [24] or reasoning tasks (25 without having been prompted with example solutions but only after being asked to provide answer to given problem. LLMs can require careful engineering of the provided prompts, either manually 26] or automatically (27]. Indeed, whole books have been written to provide guidelines on how to best perform prompt engineering (2 especially because
engineering prompts can require a great amount of expertise (29].
One method known to influence LLMs behavior is to ask them to respond as particular person B0 31], an effect which is also described as tole-taking [32]. LLMs can take in the text of one famous author, c.g. Oscar Wilde, and rewrite it in the style of another famous author, e.g. James Joyce 333 This is not only true for L.Mss but for any large model that provides results based on prompts, such as text-to-image known to boost the quality (29 or to substantially affect the style 37 of the generated images. To make LLMs respond more truthfully, Lin and colleagues introduced scenarios from the perspective of a fictional persona called "Professor Smith" (40). Conversely, to make LLMs act maliciously, Wolf et al. [41) prompt LLMs adversarially overcome alignment techniques. LLMS can also be used to simulate multiple humans which changes how they cooperate in economic games
models 36). For example, using the artist's 's name for generative art prompting is
LLMs also have their own "personalities" which can be evoked in-context 43 Although LLMs frequently behave like the average person 44), their personality profiles can be tinkered with 45). e.g. by changing the context to be more or less emotional [46]. This has led researchers to use LLMs to simulate survey responses [47] of subpopulations by conditioning them on socio-demographic
descriptions 48 to ask them to respond in persona when writing about fictitious childhood Additionally, non-deterministic tasks such as open-ended questions have also been
events 49
explored (0
Semantics derived automatically from language corpora can contain human-like biases [5I]. Thus, LLMs do not only reproduce human-like text but also replicate biases present in the training

---

data 7
Importantly, these biases can get exacerbated if LLMs are asked to provide answers in
persona 46 13
LLMs are naturally combined with large vasion-language models (VLMs) 55 such as CLIP [56] due to their versatility in wide range of visual recognition tasks. Menon al. [57 used GPT-3 to generate diverse set of short descriptions of class that improve zero-shot classification when their CLIP scores are combined. Similarly, Yang al. (58) used GPT-3 descriptions of classes as concept bottlenecks for interpretable image classification. LLMs can also be used as knowledge base for visual question-answwr answering (VQA) tasks [59]
3 In-context Impersonation Methodology
Our methodology is composed of two steps. First, we prompt and query the LLM. Second, we evalu- ate the resulting text queries in three tasks, i.e. two-armed bandit, reasoning, and visual classification.
3.1 Prompting and Querying the Large Language Model with Personas
LLMs are trained to predict the most probable next token i4 given previous tokens t t.1 by maximizing the likelihood function PLLM
fk=-)) In this work, we use pre-trained LLMs without further finetuning them. Depending on the task, we generate one or more tokens given a task-specific context e that describes the task to the language model and prompts it for an answer. The context includes the instruction to impersonate using the phrase "If you were a [persona]" where persona p is replaced by the persona name. Thus, we obtain generated tokens by sampling from
PLLM(t]c(P))
PLLM(LE
....t...,.......... tR-1)
We refer this type of contextualization as in-context impersonation.
Personas Considered. The first interesting question to look at was if LLMs could impersonate the behavior of differently aged people. For this, ask the LLM to imagine it is either 2, 4, 7, 13, or 20-year-old. We also evaluate whether the LLM is able to impersonate different fields of expertise. Depending the task considered, the expertise profiles differ (more details below). Finally, evaluate whether LLMs have biases regarding gender and skin color. For this, we asked LLMs to imagine that they were either man woman black person white person.
Large Language Models Considered. In this work, we evaluate two LLMs. For all of our tasks, we used the Vicuna-13B language model [60] which has 13 billion parameters and was trained to follow natural language instructions. Vicuna a fine-tuned version of the LLAMA language model [61] using ShareGPT (62] conversational data. We use instruction fine-tuned model because it was optimized to follow user prompts. Its weights are publicly available, allowing us to run the model locally. Vicuna is competitive with proprietary services such ChatGPT in some domains [63 In addition to Vicuna, we use the OpenAl API of ChatGPT 644 with the gpt-3. 3. 5-turbo model for the reasoning and vision tasks. For the bandit task, however, running 12k games with 10 trials each is infeasible.
We do not further train the models, nor do we provide sample solutions in-context; thus, all experi- ments are conducted in zero-shot fashion. By providing minimal guidance to perform the task, we avoid pre-conditioning the model such that answers can better reflect the internalized language of the LLM instead of relying on few-shot examples. When sampling full sentences, we use temperature of 0.7; to obtain the answer as single symbol (token), we set it to unless otherwise stated. These different temperatures were chosen based on the recommended default values each LLM.
3.2 Bandit Task Design
We asked LLMs to imagine being in different personalities while participating in multi-armed bandit task (65] taken from the psychology literature (66] and already applied LLMs (8).
An agent gets to interact with a two-armed bandit problem for 10 trials. The mean reward for each arm is drawn from p(0.)= (0, 10) at the beginning of task, and the reward for each trial
https: Imsys hag.chtt....ms..ooo/oeeaaaarreee org//leadeerbaad
3

---

In rhts gane, you have a rhetee berwees tho stor macnines, reprosented of by rocntne 1 and macnine 2. your gead is to creose the sics nechine that will gāve you the wast points over course
PXT elt)
trials. You have received the Hollowing soints
new reward:
1ist points recetued rachine [-3.5, -2:3]
list of points recetved from racstne 21 (5.8, 2.9, Seb, 5.8, 2.8, 1.4, 1.0)
Question:
porfirrdngg
year which
ang pft ed
ABCO
options following
Bon that is uufrictent regresarc ill toolean oxrressions said to complite. Whtch of the following is not complete?
Quettion;
a xig sind, Tioss on water,
A: (186, 101), (NOT, 0R), C: (MD, OR), (MNN))
sird, weans black
high- cosputer actence expert, which would chosse!
slue bürd. sav 1t with
old, how would deseribe "slact tooted aitatrous"?
Anseen: it
If you more 4 your old, how would you describe a "cardanak"? Intaen:
VTM (00LL
Cardinal
Black
If you more year old, how would you describe a "indigo bunting'?
footed
inferene
atbatrass
Figure I: Our three tasks are designed to analyze the effect of in-context impersonation. First, we investigate bandit tasks (pink) where the LLM must maximize the reward while impersonating different age groups. Second, evaluate the effect of domain expert impersonation natural language reasoning tasks (yellow). Third, we study the usefulness of descriptions generated with impersonation w.r.t. age, expertise, ethnicity, and gender for visual classification (green).
is drawn from p(relar. 0,.) = (( 1). Feedback of past trials is provided via prompt-chainin ie. concatenating previous choices and their outcomes to the current prompt submitted to the LLM. We analyze the set of emerging exploration strategies, assuming that an agent uses Bayes' rule to update its beliefs over unobserved parameters. If prior and rewards are normally distributed, then the posterior will be normally distributed and the corresponding updating rule is given by the Kalman filtering equations. Let (O.)/4)) = N(Ha,t Ta.)) be the posterior distribution at time-step t. Based on the parameters of this posterior distribution, one can define a probit-regression model:
p(A, = 1|w) + (3 V. 32RU,)
with & denoting the cumulative distribution function of standard normal distribution. Here, V, M1,, M2, represents the estimated difference in value and RU, = T1, F2, the relative uncertainty. One can use Eqquation[tto analyze how much an agent engages in exploitation behavior by inspecting 31 and how much the agent uses uncertainty to explore in a directed fashion by inspecting 33 [16].
For this bandit task, we consider personas of different ages. Specifically, we study ages 2, 4, 7, 13, and 20 to cover key developmental stages of carly childhood, childhood, adolescence, and adulthood where the learning progress is most pronounced in humans. The language model is prompted (see Figure the pink path) to only answer "I" or "2" depending on which arm a it would like to choose. The LLM receives rewards and the associated actions from previous trials inside the context in the form of a list.
ai., ................................ Tt-1) being the unnormalized logits from the LLM for the token of arm a, for each trial we sample action à a ((log da, ) )a,=)) where we have two arms A = 2. We do not apply temperature scaling in this case as we are only sampling single token and want it to reflect the LLM decision-making as faithfully as possible.
With log da, = log (11 = a c((),
3.3 Reasoning Task Design
In our reasoning task, the LLM has to answer a multiple-choice question regarding a given topic from the Multitask Language Understanding (MMLU) dataset [67], commonly used to benchmark LLMs [6]. The MMLU dataset consists of 57 tasks from Science, Technology, Engineering, and Mathematics (STEM), Humanities, Social Sciences, and Other, ranging from elementary, high school, college, and professional levels of complexity. We start by prompting the LLM with the context:
Please consider the following multiple-choice question and the four answer options A, B, C, and D. Question: {task} If you were {persona), which answer would you choose?

---

The task is replaced by the question and the 4 possible answers, while the persona is replaced by an expert (see Figure the yellow path). We consider three types of experts as personas. The task expert, e.g. for the high school computer science task, is "high school computer science expert". The domain expert an aggregation of all the remaining experts the same field as the task expert (but not the task expert himself), e-g. for high school computer science it would be any other STEM expert. The non-domain expert is an aggregation of the task experts from the other domains, e.g. for high school computer science it would be all Humanities, Social Sciences and Other experts.
After feeding the prompt to the LLM, the LLM prediction of the first token following the context is d (IaM(l]cc(P) and the N tokens for the possible answers of the multiple choice question o = {or) which in this case are A, B, C, and D. The predicted option is then given by
arg max()) with é d(c,], i 1...N
(3)
which are the predicted probabilities of the language model. With this approach, we are able to obtain the option with the highest probability according to the LLM and, thus, compare it with the ground truth label to measure the accuracy resulting from different in-context impersonations.
3.4 Vision and I Language Task Design
Lastly, we want evaluate the usefulness of descriptions generated by in-context impersonation for downstream vision and language tasks. We focus on challenging fine-grained classification tasks, as the generated descriptions need be domain specific for these tasks to succeed. We ask the LLMs to generate a description of a class, from the perspective of a persona. Our prompt is:
you were a {persona), how would you answer the following question in 45 words? Q: What is a/an mamee? A: It is
To avoid trivial solutions, i.c. the class name being mentioned in the description, we post-process the generated descriptions with two-step approach: first, we replace class names used in noun phrases with an appropriate pronoun whilst respecting the given numerous. Second, the class name still not removed, we re-use the same language model to process the descriptions sentence by sentence. For this, we use in-context examples, that demonstrate how to remove the class name information. The full process is documented in suppl. Section D.I.
Vision-Language Models (VLMs). We use CLIP (or variants thereof) [56 68 to perform fine- grained visual classification as a means to evaluate the usefulness of the generated descriptions. CLIP models are trained with contrastive image-text matching losses to rank matching image and text inputs highly and non-matching inputs lowly. [36] 68) show that CLIP variants generalize well match unseen texts, e.g. class names, an ability commonly referred to as zero-shot classification.
First, the image to classify is converted into a normalized feature representation I using CLIP's pre-trained vision backbone. Then, the class names are embedded into normalized feature vectors TN using the pre-trained text backbone. Next, all pairwise cosine similarities 1 TN of the respective feature representations are computed. Finally, the n* arg max TN) over these similarities reveals the most similar class n..
Inference. We generate description D!!)) with the above prompt for each class for each persona p where we use a generative approach, i.c. we auto-regressively sample a random token from the predicted logits (see Figure I] the green path). For Vicuna-133 13B 0.7 and the default top-k value of l: = 50. For ChatGPT we use the default temperature of 1.0. This continues until the model emits <end of sequence> the maximum number of tokens (96) is reached. We did not tune these values.
the default temperature of
For visual classification, we use the zero-shot classification capabilities of CLIP models, but instead of using the embedded class name itself (T,), we use the embedding of the generated descriptions (P) for each class n and for each persona p. The predicted class for each persona {(r)" is:
n²()* arg max(1 D(P))
Performance is measured by computing the classification accuracy of the test splits on both datasets. As the descriptions are sampled from the L.M output, the results of the experiments are stochastic and we repeat them five times. We report the mean performance well as 95% confidence intervals.
5

---

Experiments
Using Vicuna-13B we evaluate the two-armed bandit and MMLU language reasoning tasks. For the zero-shot image classification task using VLM we generate descriptions with both Vicuna-13B and ChatGPT. We focus on highlighting how the chosen persona changes the task performance of the LLM. As LLMs seem to be sensitive to prompts [69], we follow the prompting approach from (26 to vary our impersonation prompts. We run all Vicuma-138 experiments with each of the six prompt variations, which are shown in the suppl. Section A.I. All experiments are performed on the test splits using single A A100-40GB GPU and we mention inference times in suppi. Section A.2.
4.1 Age-based impersonation changes exploration strategies
In the bandit task, for every age group that the LLM impersonates, we perform 2k two-armed bandit games of 10 trials each for each prompt variation. We evaluate the task performance in three ways.
First, we show the average reward per trial the LLM ob- tained with personas of increasing age in Figure 2 (top). With increasing number of trials, the LLM obtains higher average reward, corroborating that Vicuna-13B is
3
Reward
2
able to learn from past trials improve its policy simi- larly to GPT-3 in [8] Moreover, as the LLM takes on a persona of different ages, we observe divergence of ob- tained rewards as the number of trials increases. Younger personas, i.e., 2- and 4-yerrolld personas, obtain smaller reward than older ones, i.e., 13- and 20-year-old personas.
w/
y/o
y/o
0
13 y/b
20 y/o
10
Trial
Secondly, analyze the resulting rewards by using regression, entering the trial number and age as indepen- dent variables. To extend the analysis,
Effection Roward
2.6
agos
2-20
0.025
20:50
evaluate age groups, from 2 to 20 and from 20 to 60, where we eval- uate ages in steps of 2 between and 30 and steps of 5 from 30 to 60. We report these results in Figure 2 (bottom left). We find that the impersonating LLMs generally im- proved over trials, i.e. they increase their rewards as they progressed over trials of a game (B 0.63, p < .001 for ages 2-20 and | 0.60, < .001 for ages 20- 60). Importantly, LLMs impersonating older participants generate higher average rewards until age 20 (3 0.17,
0.000
Age
Explration Explotee
Figure 2: Two-armed bandit task. Top: Average reward per persona (10k games of 10 trials), left: Age and # of trials have positive effect the expected reward, right: With age, exploration de- creases, and exploitation increases.
p < .001), thereby replicating a general pattern found in the developmental literature 70). We find no significant effect from ages 20-60, which also mirrors observations of stagnating mental performance of adults.
Lastly, we analyze how regression weights of the probit-regression were influenced by the age group the LLM is impersonating, again analyzing ages 2-20 and 20-60, 60. Figure 2 (bottom right) reveals that LLMs pretending to be older explored their environment less (3 -0-00.0 0.03, < .001) and exploited more (B3 0.04, p < .001) in the ages between 2-20. This pattern is in line with several results from the psychological literature which also found that children show higher levels of directed exploration (71) than adults 72). These results suggest that impersonating LLMS can recover human-like developmental stages of exploration in two-armed bandit task. If life is seen as an exploration-explotiasn
problem, then younger agents should show higher amounts of directed exploration 773 74). To the best of our knowledge are the first to show that LLMs replicate similar trends when using in-context impersonation.
4.2 Expertise-based impersonation changes reasoning abilities
Our experiments on expertise-basse based impersonation (details in Section 3.3 are conducted on the MMLU dataset (67, for which we ask Vicuna- 13B to impersonate experts from three different categories (task, domain, and non-domain). For each task we compute the task accuracy averaged over all task questions (95% confidence intervals are computed over the average task accuracy). We compare the task expert results with the average of all domain expert personas, the average of all

---

S T
Humanitiss
Socal Sciences
Other
0.3
0.4
Accuracy
0.2
0.3
0.2
0.2
0.1
0.1
0.1
0.1
OUPER
ooren
saper
Computer Securit
World Religoone
High School Macreeorrmice
HimanAgiig
0/1
0.0
Siran
Figure 3: Expertise-based impersonation on all domains of the MMLU reasoning benchmark (top) and on exemplary individual tasks (bottom). For each task, we consider four personas: the neutral, the task expert, the domain experts (all experts from the same domain except the task expert) and the non- domain experts (all experts from all remaining domains). The dashed line is the random baseline.
non-domain expert personas, the average of all neutral personas, and the random baseline (horizontal line). We consider four neutral personas, namely student, average student, person, and average person, and the six aforementioned prompt variations.
In Figure 3 (top row), as expected, when the LLM is asked to impersonate the task expert, the performance is the highest. This shows that the LLM can indeed impersonate task experts with accuracy higher than random. Similarly, the domain expert personas perform better than the non- domain expert personas. This trend holds for all four MMLU domains and thus for MMLU in its entirety. In general, we observe that the performance in the Humanities tasks is higher than the accuracy in the other domain tasks, which is in line with results reported in the literature [6] 75 [76] 67]. Overall, these results suggest that LLMs can increase their performance when asked to impersonate task experts compared to non-task experts.
To provide more details on the individual behaviors of these personas, in the plots on the bottom row of Figure B, we sample various expert personas, e.g. three positive and one negative case. The first, second and last plots indicate that the task expert persona performs better than the domain expert persona, which, in turn, outperforms the non-domain expert persona. In those cases, all experts outperform the neutral persona. For the High School Macroeconomics task, the task expert persona performs close to random and to the non-domain expert persona. This may be because, as Hendrycks et al. 67 observed, LLMs tend to perform worse on procedural problems that are calculation-heavy compared to purely verbal tasks. Furthermore, when the LLM performs close to or below the random baseline, i.e. the task is more difficult to solve for all types of experts, the impersonation trends are not as clear, since the model does not know how to solve the task well, irrespective of the persona. Thus, while in the Social Sciences field, the High School Macroeconomics task has worse performance, see that for World Religions, the exam result is higher than 60%, i.c. passing grade. Especially for World Religions and Human Aging, we observe that the task expert performs much better than the corresponding domain expert personas. We show results for all tasks in Section C. of the suppl.
Finally, since several MMLU evaluations
67 77 , can lead to small variations when comparing
different models', we include results with the MMLU official prompt in suppl. Section C.2, where we verify that our findings on impersonation are not dependent on the formulation of the task. Lastly, we also show MMLU results for social groups in C.3.
4.3 Impersonation as categorical descriptions is complementary for visual categorization
In this section, we provide experimental results on two state-of-the-an fine- ane-grrined visual categoriza- tion datasets, i.e. Caltech UCSD Birds (CUB) 778) and Stanford Cars (79], with 200 and 196 classes of birds and cars, respectively. Additional results for FGVC Aircraft 80 and Oxford Flowers [81] can be found in Section D.2 of the supplementary. We first compare how different VLMs make use of the generated descriptions, then compare different LLMs in our in-context impersonation tasks and finally provide some qualitative results.
Comparing VL. variants. We first compare the classification accuracy of different VL.Ms when the Vieuna-133 13B generated descriptions of classes are fed to the language encoder of the VLM. For
7

---

Age
0.15
Expertise
CUB
0.30
0.35
0.15
0.10
0.05
random
Car mishhnic
0.00
0.00
CLP
CLP CpenCLPP
CLIP VIT
CUP 8/16
OpenCLiP V/T
CUP WTT B/33
CLIP
OpenCLiP
CLP VTT 9132
CUP 8/16
OperCLiP VTT8/22
MIT8/32
B/G 8/32
VTT8/22
Stanford Cars
Age
Expertise
Race
Gender
Accurscy
0.4
0.4
0.
0.2
Rancom
random
0.2
runcom
Tastem
untlliiggt
Blackposoo peson
man
7yyb
13yl
20y55
mechanic
white person
woman
0.0
0.6
0.0
0.0
OpenCLi VITB.32
CLIP
CUP
CUP
CLIP VIT-8/36
OpenCLP
CLIP
CUP
OpenCLIP VIT8.22
VIT8/32
VIT8//2
VIT8/16
VIT822
VITR//22
VITB/22
V/T8/22
VIT
Figure 4: Comparing CLIP-32, C and OpenCLIP as VLMs (the language input comes from Vicuna-13B) on CUB (top) and Stanford Cars (bottom) datasets. We observe the effects of age, expertise, ethnicity and gender independent of the VLM used for fine- ane-ggaieed visual classification. The dashed line represents the random baseline.
the vision encoders we consider the Vision Transformer (ViT) 82 based B/32 and B/16 variants of
the official CLIP implementation [56] as well as the OpenCLIP B/32 ViT variant (68). The latter is replication of the original CLIP trained on a larger dataset (Laion 5B (83 ). For each CLIP variant, we use the corresponding causal transformer text encoders, which might not encode text as well as Vicuna but are able to embed the text into shared multi-modal space.
Our results in Figure 4 show that across all three CLIP variants increased age in the impersonated persona increases performance for both bird and car classification. Interestingly, there is a significant increase in performance at years of age when recognizing cars. Our expertise evaluation shows that the car mechanic persona's 's descriptions performs better than ornithologist's when recognizing cars. Interestingly, racial (column 3) and gender (column 4) personas, reveal consistent biases. While the black performs better in car classification, the white performs better in bird classification. This may indicate that there are stereotypical biases in the training data. Similarly, while the woman performs clearly better than man for bird classification, the trend is not as strong for car classification although man performs slightly better than woman. The language encoder of VLMs potentially being weaker than Vicuna,
expect these results to improve overall with stronger language encoder in the VLM but this is an orthogonal direction to explore. To confirm the significance of our results, we run Chi tests for expertise, race and gender. We consider the three CLIP models, five different seeds and the six different impersonation prompt variations. We find that for all experiments considered, (CUB, Stanford Cars} x (man/woman, black/white, ornithologist/car mechanic), J. p<0.001. Thus, we conclude that our results are significant.
We also investigate the effects of composing personas for compu- tationally feasible subset of persons. More specifically, we study all possible combinations of (Black, White] x (Female, Male] for the CUB dataset for 5 different seeds (Figure 6). With Vicuna- 13B we see weak evidence that the biases co-construct: Individually the white per- sona outperforms the black persona and the same applies to the female persona outperforming the male persona. Combined, the white female persona outperforms both the black female persona (change in race) and the white male persona (change in gender). Furthermore, we also study performance of additional genders (agender and non-binary) and races (indian, asian and hispanic) in the suppl. in Section D.5.
0.15
0.10
black male
white
0.05
black female
white
random
0.00
CLIP
CUP
OpenCUP VIT-B/32
VIT-B/32
vIT8/116
Figure 6: of personas Vicuna-133 13B.
Composition
CUB for
LLM variants We evaluate how different LLMs, namely Vicuna- 13B and ChatGPT, generate descriptions of the classes of interest. In these experiments, we keep the VLM fixed to OpenCLIP, as it is the best of the CLIP variants tested above. For computational reasons, we only evaluate on our original impersonation prompt. Figure shows the effect of L.MM impersonation on the generated descriptions evaluated on zero-shot image classification.
Comparing
8

---

Age
Expertise
Race
Gender
0:15
CUB
Accuracy
0.10
0.10
0.05
20y/o
modhanic
whice parson
woman
0.00
0.00
0.00
0.00
CharGPT
Vicuna-138
CharGPT
ChaiGPT
CharCPT
Stanford Cars
Age
Expertise
Race
Accuraty
0.4
0.4
0.4
0,4
0.2
I Pyyh
4yth
B. Tym
amitholge
btack person
man
13yro
20yl
macronic
whitt person
woman
0.0
0.0
0.0
0.0
Vicuna-138
CharGPT
Vicuna-138
ChaiGPT
Vioura-138
ChatGPT
Viouna-138 38
ChatGPT
Figure 5: Comparing Vicuna- 13B and ChatGPT as LLM variants (OpenCLIP is the VLM) on CUB and Stanford Cars. For both LLMs, the accuracy increases with increasing age, the expert persona on the respective dataset performs better and both LLMs are not free of biases, and impersonation of different genders or race affects their performance. The dashed line represents the random baseline.
For the age personas, we observe a clear trend of increased performance for both LLMs as they impersonate older characters. The progression is particularly pronouneed for ChatGPT, where Stanford Cars the 2-ycar-old persona describes different cars with similar expressions leading to 4% accuracy, but as ChatGPT's persona gets older, it becomes more accurate in describing cars, e.g. 54.9% for persona of age 20. This indicates that LLMs can replicate human language at different development stages, varying their language both in terms of vocabulary and general knowledge for accurately describing these objects as discussed exhibit higher expertise on the topic when we ask them impersonate bird expert ("ornithologist" persona) and a car expert ("car mechanic" persona). The respective domain expert persona performs approximately twice well the non-domain expert persona when using ChatGPT. Impersonating an expert, the LLM tends to describe a class in more detail and mention more discriminative features.
84. Similarly to the reasoning task, LLMs
We also observe that impersonation apparent when we ask the LLMs to impersonate a "black" or "white" person. ChatGPT tends to deseribe both birds and cars better when posing white person. Vicuna-13B. 13B, provides better descriptions of cars as a black person. Gender biases are a bit less noticeable, but we woman persona and ChatGPT identifying
reveal biases encoded in the LLMs. A race bias becomes
the other hand,
still find Vicuna- 13B giving better bird descriptions cars better as a man persona. While instruction- based fine-tune fine-tuning (64] tries to remedy social biases encoded in LLMs some extent, can still expose them through in-context impersonation.
Overall, we find that CharGPT shows larger effects, probably due its access to more diverse (fine- tuning) data. The fact that the effects described above be found with two very different language models suggests that they are a result of the overall language modeling and instruction following training on internet data instead of specific model artifacts.
Qualitative results and limitations. Figure we provide the descriptions generated by ChatGPT and Vicuna for one class, i.e. black billed cuckoo, from the CUB dataset and one class, i.e. AM General Hummer SUV 2000, from the Stanford Cars dataset. As personas, we sample all the age personas we considered in our experiments, namely 2, 4, 7, 13 and 20-year-old personas.
For both LLMs, in both datasets, observe that with increasing age, the complexity of the cabulary and attributes of the mentioned objects increases. A 2-year-old persona talks about the sound the bird or the car makes, the shapes of the wings or wheels, and the emotions attached to seeing or riding it. A 4-ycar-old persona interestingly mentions experiences sceing the bird or the car more distinctly. 7-year-old persona starts using more complicated adjective phrases, e.g. can drive on rough roads and outside places, whereas a 13-year-old persona takes it one step further, e.g. brownish gray body with distinctive rusty colored markings. Finally, a 20-year-old persona makes a more complete description of the object including where the bird is found or what the car is mainly used for. This is in line with 85] where the authors show that given the same length text, smaller children use less diverse and non-academic vocabulary, and repeat lot. Even though LLM's may faithfully represent the language of children, we qualitatively observe similar patterns. We show more examples and quantize the properties of the generated descriptions in suppl. Section D.3.

---

27 you wore opersonan, how vould you describe oclass??
ChaiGPT
Vicuna
bLack
bire that a black beak and called cackoe. It tsere
bled thec's really coolt It's slack with aill that's eulet it <lies around singo so sweet. 2t's ore of my faxordte börds mort!
334
eat Irsects and Mives North caertox, They zay thatr aggs then take care their basiiss.
other bird's eests
bird that black and
distiactive shite patch tes tati.
bire species that lives in Norch smerica, it has a black 1:33
snoue for its distinctive calls which sourds like persan Leitacing a
and a tron or gray plasogh, They are eigiatory hinds and car
curess Clock, it a inggotoory stind than can ae found in sorts and centiall
oftee te heard naking their dicrstentive "oo0-eab-roo" calll te
feurice. 2t Aa conncn band in many parts of the vorld, but it is comiidered
Uhe sumeer
spucies of concomn des to declininn populattions in sow regicts.
Strderd
Hig car that poes vrooe veeon and can ge over rocks and
a sig cer that can to offiroadd be has four whesls and a strong ingine. It ie nol and black.
banga. 22 has big wheels and is fun to ride it.
gesanal
humer sw
2009
tough
ride to
large, stlitayysttyle SUM destgond fhe off-road ane. sat
congace SWM that use seruiarturre by daeeteas Rotors (IM) froe 2000 to J006.
panstar in the earty 2000 and lcoos for Ita nagendrene
It was knean itc rugped actertor and spactaus tntentery popular
uniaue styliing Noverer, elso bnow being very large
amang both civiitans and silitary persornel. uns also leoun for ira ofr-rase for regine and
neavy,
make difficult to manoseer right
casslliiies
activities
hasting
suares city
capplng.
Figure 7: Qualitative results sampling all the age personas (2, 4, 7, 13 and 20-year-old personas) for two classes, i.e. Black Billed Cuckoo (CUB) and AM General Hummer SUV 2000 (Stanford Cars) classes. The results are obtained by querying ChatGPT and Vicuna.
One obvious difference between these two LLMs to point out is that the descriptions obtained from Vicuna appear to be longer and more detailed. Further, at earlier ages, e.g. 2 or 4, especially on CUB, the descriptions of Vicuna seem poetic. The difference between the semantic content of the descriptions of the 13-year-old persona and the 20-year-old persona seems to be less distinct in Vicuna than in ChatGPT. One final interesting observation is that Vicuna descriptions talk about the color of the car whereas the color can not be a distinguishing property of a car.
5 Broader Impact
We believe that better understanding of in-context impersonation, as well as its resulting down- stream effects, can not only help to mitigate the risk of fraud but also to understand how these newly-powerful agents behave more generally [86]. We have already seen that in-context imperson- ation boosts performance and produces biases; these results could be followed up by investigating how these characteristics emerge during training, change with increasing model size [87] or adapt with additional fine-tuning [88] Additionally, LLM providers could quantitatively test for these biases before releasing new models. We specifically discourage crafting (system) prompts for maxi- mal performance by exploiting biases, as this may have unexpected side effects, reinforce socictal biases and poison training data obtained with such prompts. Other misuses may include amplifica- tion of stereotypical biases through generated content and using impersonation to invoke fake trust. However, we believe systematically studying these biases raises awareness in the ML community and general society and serves as a first step to research mitigation strategies. Lastly, we discuss limitations of our work in suppl. Section E.
6 Conclusion
We presented evidence that in-context impersonation, that is asking LLMs to take on different roles in context, can change their performance and reveal their biases. Asking LLMS to impersonate differently aged people in a two-armed bandit task, LLMs could reproduce human-like developmen- tal stages of exploration behavior. Asking LLMs to impersonate domain experts, they performed better than LLMs that were asked to impersonate a non-domain expert. Finally, asking LLMs to impersonate various roles in vision-language task revealed not only that impersonation can boost relative performance but also recovered societal biases about a person's 's age, gender, and race.
We have demonstrated the effects of in-context impersonation on single agents performing relatively simple tasks across a limited range of personas. In future work, we want to scale up this approach to multiple L.MSs impersonating variety of personas across complex and interactive tasks (89]. Finally, we believe that in-context impersonation can also be applied to other modalities, for example to large models for video generation (90).
10

---

7 Acknowledgements
The authors thank IMPRS-IS for supporting Leonard Salewski. This work was partially funded by the Portuguese Foundation for Science and Technology (FCT) under PhD grant 5020.07034.BD the Max Planck Society, the Volkswagen Foundation, the BMBF Tübingen Al Center (FKZ: 01S$18039),, DFG (EXC number 2064/1 Project number 390727645) and ERC (853489-DEXIM).
References
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Frie Sigler, Mateusz: Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. NeurlPS, 2020.
[2] Taylor Webb, Keith Holyoak, and Hongjing Lu, Emergent analogical reasoning in large language models. arXiv:2212.0 12.0 196, 2022.
[3] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. TMLR, 2022.
[4] Enkelejda Kasneci, Kathrin Seßler. Stefan Küchemann, Maria Bannert. Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 2023, [5] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. 1XXi::2008.0255. 2021.
[6] Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. Understanding the capabilities, limitations, and societal impact of large language models. arXiv:2 02.002503, 2021.
[7] Emily M Bender, Timnit Gebru, Angelina McMillan McMillan-Majoor, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In ACM FAccT, 2021.
[8] Marcel Binz and Eric Schulz. Using cognitive psychology understand gpt-3, PNAS, 120(6), 2023.
[9] Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Christopher Pal. On extractive and abstractive neural document summarization with transformer language models. In EMNLP, 2020.
[10] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
[11] Sewon Min, Xinxi Lyu, Ari Holtzman. Mikel Artetxe. Mike Lewis, Hannaneh Hajishirzi. and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022.
[12] Sang Michael Xie, Aditi Raghunathan, Percy I Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In ICLR, 2022.
[13] Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persoma-assigned language models. arXiv:2304.05333, 2023.
[14] Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. Can chatgpt write a good boolean query for systematic review literature search? McXiv22302.03499 2023.
[15] Katherine Elkins and Jon Chun. Can gpt-3 pass a writer's turing test? Journal of Cultural Analytics, 5(2).
2020.
[16] Marcel Binz and Eric Schulz. Modeling human exploration through resourcc-rational reinforcement learning. In NeurIPS, 2022.
[17] Andrew Lampinen, Ishita Dasgupta. Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context?
In EMNLP. ACL, 2022.
[18] Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha Kush Bhatia, Ines Chami, and Christopher Re. Ask me anything: A simple strategy for prompting language models. In /CLR, 2023.
[19] Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy
Ba. Large language models human-level prompt engincers. In NeurIPS Workshops, 2022.
[20] Timo Schick and Hinrich Schütze. Exploiting cloze- quoze-qusstions for shw-shot text classification and natural language inference. In EACL, 2021.
[21] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaf- fin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian- Jian Jiang, Han Wang, Matteo Manica. Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault
11

---

Fevry, Jason Alan Fries. Ryan Teehan. Teven Le Scao, Stella Biderman. Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In ICLR, 2022. Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys. 53(3), 2020.
[22]
[23] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot lcarning compre-
hensive evaluation of the good, the bad and the ugly. TPAMI, 41(9), 2018.
[24]
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well do large language models perform in arithmetic tasks? arXiv:2304. 02015, 2023.
[25] Emre Kıcuman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language
models: Opening a new frontier for causality, arXiv:2305. 70X0022005.0000, 2023.
[26] Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot
paradigm. In CHI, 2021.
[27] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt:
Eliciting Knowledge from Language Models with Automatically Generated Prompts. In EMNLP, 2020.
[28] Nathan Hunter. The art of prompt engineering with chaiGPT. eBook, 2023,
[29] Jonas Oppenlaender, Rhema Linder, and Johanna Silvennoinen. Prompting ai art: An investigation into
the creative skill of prompt engineering. urXiv:2303.13533 2023.
[30] Seungju Han, Beomsu Kim, Jin Yong Yoo, Seokjun Seo, Sangbum Kim, Enkhbayar Erdenee, and Buru
Chang. Meet your favorite character: Open-domain chatbot mimicking fictional characters with only a few utterances. In NAACL-HLT. 2022.
[31] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: conditional transformer language model for controllable generation. arXiv:999.008855 2019.
[32] Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role-play with large language models.
ArXiv:2305. 16367, 2023.
[33] Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg- KinkKKikpatrick. Unsupervised text style transfer using language models as discriminators. NeurlPS, 2018.
[34] Katherine Crowson, Stella Rose Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castri- cato, and Edward Raff. Vagan-clip: Open domain image generation and editing with natural language guidance. In ECCV, 2022.
[35] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2021.
[36] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. NeurlPS, 2022.
[37] Nassim Dehouche and Kullathida Dehouche. What's 's a text-to-image prompt? the potential of stable diffusion in visual arts education. Heliyon, 9, 2023.
[38] Manuel Brack, Patrick Schramowski, Felix Friedrich, Dominik Hintersdorf, and Kristian Kersting. The stable artist: Steering semantics diffusion latent space. arXiv:2212.06013, 2022.
[39] Sam Witteveen and Martin Andrews.
Investigating prompt engineering in diffussion models.
mxXvv2211..15455 2022.
[40] Stephanie Lin. Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human
falsehoods. In ACI., 2022.
[41] Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv:2304.1 1082, 2023.
[42] Gati Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans. arXiv:2208. TTt::2208./0244. 2022.
[43] Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. Ai psychometrics: Using psychometric inventories obtain psychological profiles of large language models. 2023.
[44] Peter S. Park, Philipp Schoenegger, and Chongyang Zhu. "correct answers" from the psychology of artificial intelligence. arXiv:2302.07266 2023.
[45] Saketh Reddy Karra, Son Nguyen, and Theja Tulabandhula. Ai personificatio:: Estimating the personality of language models. arXiv:2204.1 224402000, 2022.
[46] Julian Foda-Forno, Kristin Witte, Akshay K Jagadish, Marcel Binz, Zeynep Akata, and Eric Schulz. Inducing anxicty in large language models increases exploration and bias. arXiv:2304.1 10004,11111,,111 2023. Moritz Hardt, and Celestine Mendler-Dunner. Questioning the survey responses of large language models. arXiv:2306. 07951, 2023.
[47] Ricardo Haminngez--llmede
[48] Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate.
Out of one, many: Using language models to simulate human samples. Political Analysis, 2023.
[49] Hang Jiang, Xiajie Zhang. Xubo Cao, Jad Kabbara, and Deh Roy. Personallm: Investigating the ability of
gpt-3.5 to express personality traits and gender differences. arXtv:2305.0255 025447 2023.
[50] Gati Aher, Rosal. Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In ICML, 2022.
12

---

[51]
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334). 2017.
[52]
Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. In AAAI/ACM AEIS, 2021.
[53] Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto, Ex- ploiting programmatic behavior of Ilms: Dual-use through standard security attacks, arXiv:2302.0573: 2023.
[54] Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc V. Le, Zhen Li, and Tom Duerig. Scaling up visual and vision-language language representation learning with noisy text supervision. In ICML, 2021.
Sung,
[55] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In CVPR, 2022.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[57] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In
ICIR, 2023.
[58] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burre and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification.
arXiv:2211.1 1 2022.
[59] Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of GPT-3 for few-shot knowledge-basse based VQA. In AAAI, 2022.
[60] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
[61] Hugo Touvron. Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302. 13971, 2023.
[62]
Dom Eccleston. ShareGPT: Share your wildest conversations with one click. https:/ /sapee///haapgt... com/
2023. [Online: accessed 15-May-2023].
[63] Lianmin Zheng, Wii-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Haotong Zhang. Joseph Gonzalez, and lon Stoica. Judging
Ilm-as-a-judge with mt-bench and chathot arena. arXiv:2306. 05685, 2023.
[64] Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. NewrlPS, 2022.
[65]
Samuel J Gershman. Deconstructing the human algorithms for exploration. Cognition. 173, 2018.
[66]
Eric Schulz and Samuel J Gershman. The algorithmic architecture of exploration in the human brain. Current opinion in neurobiology, 55, 2019.
[67]
Dan Hendrycks, Collin Burns. Steven Basart, Andy Zou, Mantas Mazeika. Dawn Song, and Jacob Stein- hardt. Mcasuring massive multitask language understanding. In ICIR, 2021.
[68]
Mehdi Cherti, Romain Beaumont, Ross Wightman. Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann. Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. arXiv:2212.. 22l::2222.7144 24.07114,222 2022.
[69] Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel J. Orr Neel Guba, Kush S Bhatia, Ines Chami, Frederic Sala, and Christopher R'e. Ask me anything: simple strategy for prompting language models. arXiv:2210.02441, 2022.
[70] Kate Nussenbaum and Catherine A Hartley. Reinforcement learning across development: What insights can draw from decade of research? Developmental cognitive neuroscience, 40, 2019.
[71] Emily G. Liquin and Alison Gopnik. Children are more exploratory and Icarn more than adults in an approach-avoid task. Cognition, 218, 2020.
[72] Eric Schulz, Charley M Wu, Azzurra Ruggeri, and Bjöm Meder. Searching for rewards like child means less generalization and more directed exploration. Psychological science, 30(11), 2019.
[73] Anna P. Giron, Simon Ciranka, Eric Schulz, Wouter van den Bos, Azzurra, Ruggeri, Björn Meder, and Charley M. Wu. Developmental changes in learning resemble stochastic optimization. 2022.
[74] Nathaniel J. Blanco and Vladimir M. Sloutsky. Systematic exploration and uncertainty dominate young children's 's choices. Developmental science, 2019.
[75] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv:2204.00 2220001111,,.. 2022.
[76] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute- ontiial- large language models. arXiv:2203. 15556, 2022.
13

---

[77] Percy Liang. Rishi Bommasani, Tony Lee. Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Zhang, Christian Cosgrove, Christopher Manning, Christopher R' e. Diana Acosta-Navas. Drew Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong. Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang. Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas F. Icard, Tianyi Zhang. Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda, Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525, 2023.
[78] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongie. The caltech-uesd
dataset. 2011.
[79] Jonathan Krause, Michael Stark. Deng, and Li Fei-Fei. object representations for fine- fineggrineed categorization. ICCV Workshops, 2013.
[80] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft.
Technical report, 2013.
[81]
Maria-Elent Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing. 2008.
[82] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[83] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, I Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv:2210.084. 2022.
[84]
John Oates and Andrew Ed Grayson. Cognitive and language development in children. Open University Press, 2004.
[85] Philip Durrant and Mark Brenchley. Development of vocabulary sophistication across genres in english children's 's writing. Springer Reading Writing, 32, 2019.
[86] Ryan Burnell, Wout Schellaert, John Burden, Tomer D Ullman, Fernando Martinez-Plumed, Joshua Tenenbaum, Danaja Rutar, Lucy Cheke, Jascha Sohl-Dickstein, Melanie Mitchell, et al. Rethink reporting of evaluation results in ai. Science, 380(6641), 2023.
[87] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXi::2001.08361, 2020.
[88] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. (eXxv:990900890) 2019.
[89] Joon Sung Park, Joseph C O' O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra human behavior. 3Xiv:2334.033440 2023.
[90] Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv:2212.0319), 2022.
14


Reviews:
[{"review_id": "BsRR6DW6Q5", "reviewer": "Reviewer_J5Gp", "summary": "This paper presents a comprehensive evaluation on the downstream impact of impersonation with the use of LLM and multimodal model (specifically CLIP). There is a couple of interesting finding that LLM, when impersonated, aligns to some behavior carried by the people in the respective norm. This happens in ages, races, genders, and degree of expertises. The finding seems to suggest that one can exploit such phenomenon to make a large model perform better on some downstream tasks. This paper also suggest that such exploitation should be concerned. The arguments are of conflict, but also interesting.", "questions": "Please see the general comments.", "limitations": "Implicitly or explicitly, this paper suggests the exploitation of stereotypes to improve model performances on downstream tasks.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduced three types of evaluation using recent LLM and multimodal CLIP to probe LLM's downstream performances: exploration-exploitation, reasoning (MMLU), and shallow multimodal understanding. There is an very interesting finding that impersonation aligns to different behavior and downstream performances on these benchmarks. An immediate takeaway is that it gives an \"interface\" to easily boost downstream performance with a discovered stereotype to impersonate. The respective concern raised in this paper is also of value to the community. In general, I am happy to see such argument and findings to appear in the conference, as long as the argument goes well.", "weaknesses": "A few downsides:\n- On one hand, this paper seems to suggest one can easy explore stereotypes to impersonate and use them to improve some downstream performances. On the other hand, this strategy is of special concern, and I believe, should be avoided. ***It's unclear what this paper's take is.*** A very simple and immediate counter argument to this paper is that, such exploitation aggregates the existing stereotypical biases in the real world. And if this concern is valid, then this paper shouldn't advertise it as a strength.\n\n- Not clear if the observation is robust against prompt variance. The only place seems to have robust eval is the two-armed bandit experiment.\n\n- The age-related evaluation is narrow. A more comprehensive range should go beyond age 20. This can be useful to probe for potential age biases and have a more comprehensive picture about the age \"mimicking\" phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their helpful comments indicating that they would be happy to see our findings appear in the conference, and we will use the rebuttal to further strengthen our argument.\n\n### Broader Impact\nBy “strength” we refer to the improved performance when impersonating experts. However, we agree with the reviewer’s take that these still fall under biases that ideally should not be present in LLMs.\nWe specifically discourage crafting (system) prompts for maximal performance by exploiting biases, as this may have unexpected side effects, reinforce societal biases and poison training data obtained with such prompts. Aside from the directions mentioned in our conclusion (lines 348-349; e.g. how do these biases emerge as a function of scale), we believe that our submission will make LLM providers aware that they should quantitatively test for these biases. We will update our discussion to clarify our stance and will adjust our narrative not to highlight any of these biases as strengths.\n\n### Prompt variance\nPlease see the general comment for a detailed analysis of prompt variance where we show that our findings are robust not only in the bandit task, but also for the reasoning (MMLU) and visual description experiments across a large set of automatically generated prompt variations. \n\n### Age evaluation\nWe address the concerns regarding age evaluation in the general comment, where we show that the effect of age replicates if all ages between 2-20 are evaluated and diminishes in the 20-60 range for the bandit task."}, {"review_id": "cYFV3BSlGX", "reviewer": "Reviewer_NysF", "summary": "Language models can play different roles using prompting techniques (e.g., adding \"you are a 3-year-old boy\" at the end of the prompt). This paper investigates whether roles in the prompt affect language model performances on three tasks: multi-armed bandit, language-based reasoning, and visual category description. The paper found that: \n1. Language models exploit more in multi-armed bandit tasks when prompted to be older. \n2. Language models perform better when prompted to be task experts. \n3. Language models describe objects better when prompted to be older. For certain object categories, language models show biases (e.g., describe cars better when prompted to be a man instead of a woman).", "questions": "- What are the dashed lines in Figure 3 (results on STEM and Humanities)?", "limitations": "See weaknesses. I'm happy to increase my rating if these are well addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper focuses on an interesting and novel topic: how role-playing (impersonation) affects language model performances on downstream tasks. This is a less-known research question in the NLP community.\n- The finding about performance differences when describing certain object categories, such as cars, when prompting language models to be different genders provides a novel perspective to understand biases in language models and our society.\n- The paper is generally well-written and can be easily understood. However, the writing should be more focused. For example, in the first paragraph of the introduction, I did not get any connection between Min et al. found prompting LLM with random labels will not degrade model performance and your research topic impersonation. Similarly, there is no need to introduce CLIP training when introducing the visual category description task.", "weaknesses": "- While the topic is to study how different roles affect language model performances, from experimental results, ages have the most significant impact on model performances, and other factors have much more minor effects on model performances. This seems obvious- it is easy to understand older people can exploit more in multi-armed bandit tasks and describe object categories more clearly. However, the most exciting finding, where gender/race can affect model performance on tasks, is not well studied. They only study two object categories from the visual object description task, and the performance difference between genders/races is much smaller and insignificant based on the plots. More systematic studies on language model biases during impersonation should significantly improve the work.\n\n- Experimental results can become more solid using more prompt variations and model families. Language models are well-known to be sensitive to prompts; the authors should mention how robust their results are when changing some words in the prompt. Moreover, more model families besides Vicuna are appreciated, especially on bias experiments. Will the finding about gender bias hold for different model families?\n\n- Besides the tasks with deterministic labels, it would also be interesting to show how impersonation affects task performances on non-deterministic tasks, such as social opinions or morality tasks. Since the major goal of impersonation is to simulate different people, it would be more interesting and convincing to show that language models indeed reflect these group opinions on non-deterministic tasks when prompted to be them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments indicating that we focus on an interesting and novel topic, that is a less-known research question in the NLP community and provides a novel perspective to understand biases in language models and society.\n### Triviality of the age effect\nWhile it seems obvious that older humans perform better in these tasks (i.e. they exploit more in multi-armed bandit tasks), to the best of our knowledge we are the first to show that LLMs replicate similar trends when using in-context impersonation. Even though this finding might seem obvious in hindsight, it is not intuitive why this would also happen in LLMs.\n### Significance of the results\nWe run $\\text{Chi}^2$ tests for the race and gender results presented in Figure 5 of the rebuttal PDF. Thus, we consider 3 CLIP models, 5 different seeds and 6 different impersonation prompt variations. We find that for all experiments considered, {CUB, Stanford Cars} x {man/woman, black/white}, p<0.001. Thus, we conclude that our results are significant.\n### Biases for more object categories\nTo clarify, CUB dataset contains 200 different bird categories (11K images) and Stanford Cars contains 196 categories of cars (16K images) with various sizes, appearances, habits etc and the visual classification task in these datasets (due to the large number of categories and fine-grained similarities between these objects) is far from solved. While a sentence on distinguishing properties of a bottle and a car can be very short (these objects have many uncommon properties), a sentence describing the differences between a black-footed albatross and laysan albatross that also remains distinguishing between a field sparrow needs to talk about many more category-specific properties. Arguably, such a fine-grained categorization task is one of the most challenging ones even if we work only in the language domain. Nevertheless, we extend our analysis to other datasets and more categories by using FGVC Aircraft [B] (100 categories of aircrafts from different manufacturers and eras) and Oxford flowers [C] (102 categories of flowers with large scale, pose and light variations).\nFor gender, we find significant performance differences when evaluating the descriptions generated by Vicuna-13B on the two additional datasets, strengthening our original argument that these LLMs exhibit biases (Figure 4). This means that descriptions generated by female personas outperform those generated by male personas across all three tested VLMs. For racial biases, we see only smaller differences across evaluation with different VLMs. The same trends hold for ChatGPT.\n### Prompt sensitivity\nPlease see the general comment for a detailed analysis of prompt sensitivity where we show that our findings are robust across all three tasks (bandit, MMLU, and vision descriptions).\n### Do findings transfer to other LLM families?\nMost open-source models such as Vicuna are fine-tuned from the same base model. Recently, a new foundational open-source model, Llama 2 (70B, Chat variant) has been released which is significantly larger than Vicuna-13B and has been trained on more data. We rerun the bandit experiments using Llama 2 and come to the same conclusions. The effect of age in the range of 2-20 on the reward is β=0.16 (p<.001) and β=0.26 (p<.001) for Vicuna-13B and Llama 2, respectively.\n\nAdditionally, we ran our experiments using the proprietary Google PaLM model via their API. When we apply in-context impersonation of racial biases, the API does not provide an answer because the LLM model output is flagged by a text classifier to be unsafe. Hence, there are already safeguards in place for some commercial services.These safeguards seem to be less sensitive for impersonation of age and gender. However, they prevent us from reliably evaluating the underlying LLM. We will add Llama 2 results and as well as a brief discussion on these existing safeguards to the paper.\n### Non-deterministic tasks\nRelated work has explored non-deterministic tasks, e.g. [D] finds that LLMs give inhumanly accurate answers specifically in open-ended questions whereas a collection of human answers is much more diverse. [E] studies group opinions via LLM-generated survey responses and found that those “currently lack the signals found in human populations”. Additionally, we discussed [40] in our related work section (lines 78-79), which simulates the survey responses of sub-populations. In this work we focus on diverse tasks that can be objectively measured. Hence, we believe expanding the set of experiments to non-deterministic tasks is out-of-scope for this work, but see them as a relevant future direction.\n### Dashed lines in Figure 3\nThe dashed lines in Figure 3 correspond to the random baseline, i.e. 25% for a 4 choice exam question. We will update the caption accordingly to clarify this.\n\n#### References\n[B]: _Fine-Grained Visual Classification of Aircraft_, S. Maji, J. Kannala, E. Rahtu, M. Blaschko, A. Vedaldi, arXiv.org, 2013\n\n[C]: _Automated Flower Classification over a Large Number of Classes_, M.-E. Nilsback, and Andrew Zisserman. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing (2008): 722-729.\n\n[D]: _Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies_, G. Aher, R. I. Arriaga, A. T. Kalai, ICML 2023\n\n[E]: _Questioning the Survey Responses of Large Language Models_, R. Dominguez-Olmedo, M. Hardt, C. Mendler-Dünner, arXiv, 2023"}, {"review_id": "dJUAtEolu0", "reviewer": "Reviewer_KqkQ", "summary": "This paper tries to understand whether LLMs can take on different personas when they do in-context generation. They have tested different settings: In a two-armed bandit setting, they ask LLMs to act as different-aged people; in a reasoning task, LLMs are asked to behave as different-level experts; and in a vision-and-language task, multiple roles have been set to the LLMs. The authors find while impersonation can improve model performance, such methods also reveal the biases in LLMs.", "questions": "- What if in each setting, you add a random persona? For example, in the domain expert case, you impersonate LLMs as \"normal student\" (or as \"specialist\"). I'm just curious how certain tokens (\"expert\") will impact the result.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Test their hypothesis across different domains and with various persona aspects. \n- They have set the persona specific to each task, e.g, expertise-based persona for reasoning ability analysis.\n- Well-presented results.", "weaknesses": "- LLMs seem to be sensitive to prompts [1]. And I'm wondering how stable the conclusion will be. \n- I'm curious to what extent the LLMs align with human behavior. For example, in the two-armed bandit task, the authors observe \"LLMs could reproduce human-like developmental stages of exploration behavior\". And I'd be curious how humans (from different ages) perform the same task. Similar to other tasks.\n\n[1] Arora, Simran, et al. \"Ask me anything: A simple strategy for prompting language models.\" arXiv preprint arXiv:2210.02441 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments indicating that we tested our hypothesis across several different domains and with various persona aspects and presented the results well.\n\n### Prompt sensitivity\nPlease see the general comment for a detailed analysis of prompt sensitivity where we show that our findings are robust to prompt variations across the three tasks we present.\nIn addition to our findings that the impersonation results are robust to prompt variations, already showing a single failure case (e.g. LLMs have biases wrt to gender and race) may indicate a problem that should be addressed. Thus our conclusion that LLMs expose biases, would still hold.\n\n### Human LLM alignment\nIt is a fairly well-established observation that children tend to explore more in reinforcement learning tasks than adults. Past work such as the ones cited below mostly focused on comparing adolescent with adult behavior using three features for the probit regression, with one additional feature quantifying the amount of random exploration. In our work, we were interested in how exploration differs between LLMs impersonating children and LLMs impersonating adults. Moreover, our own simulations assessing how recoverable the original three features are showed that one can really only reliably measure the amount of exploitation and directed exploration from the regression, which is what we ended up doing. Our findings of increased directed exploration in children align well with the literature on human exploration, for example, as reported in:\n\n_Children are more exploratory and learn more than adults in an approach-avoid task_, Liquin, E. G., & Gopnik, A. (2022). Cognition, 218, 104940.\n\n_Searching for rewards like a child means less generalization and more directed exploration_, Schulz, E., Wu, C. M., Ruggeri, A., & Meder, B. (2019). Psychological science, 30(11), 1561-1572.\n\n_Systematic exploration and uncertainty dominate young children's choices_, Blanco, N. J., & Sloutsky, V. M. (2021). Developmental Science, 24(2), e13026.\n\n### Random persona\nWe include a discussion regarding what we term the neutral baseline (to distinguish from the random baseline, e.g. 25% on MMLU) in the general comment. This baseline considers the average between four personas (student, average student, person, and average person) for MMLU and two personas for the vision tasks (person and average person). We show that, as expected, the neutral baseline performs worse than the respective expert."}, {"review_id": "aFTgg90ny0", "reviewer": "Reviewer_ZdXm", "summary": "This paper explores the ability of a LLM to engage in roleplay, and how being prompted to do so influences the models' generations. In particular, they focus on exploring how models prompted to behave like children seem to do so accurately using comparisons from the human developmental psychology literature.", "questions": "1. How and why were the particular experimental settings used chosen? For example, you ask the model to impersonate a 2, 4, 7, 13, 20-year-old. Why those numbers and not others? Why does the temperature change between single-token and multi-token generations?\n\n2. Your prompt for MMLU (\"Please consider the following multiple-choice question and the four answer options A, B, C, and D. Question: {task}\") differs from the official implementation of MMLU. Why was this chosen? **Can you report the result you using the official prompt?((\n\n3. You write \"We focus on 32 tasks (19 STEM and 13 Humanities) and for each task we compute the task accuracy averaged over all task questions (95% confidence intervals are computed over the average task accuracy). We compare the task expert results with the average of all domain expert personas, the average of all non-domain expert personas and the random baseline (horizontal line).\" Why were these 32 tasks out of the MMLU benchmark chosen? Why did you not use all of the tasks? **Can you report the results on all tasks**\n\n4. You write\n\n> We also observe that impersonation can reveal biases encoded in the LLMs. A race bias becomes apparent when we ask the LLMs to impersonate a “black” or “white” person. ChatGPT tends to describe both birds and cars better when posing as a white person. Vicuna-13B, on the other hand, provides better descriptions of cars as a black person. Gender biases are a bit less noticeable, but we still find Vicuna-13B giving better bird descriptions as a woman persona and ChatGPT identifying cars better as a man persona. While instruction-based fine-tuning [54] tries to remedy social biases encoded in LLMs to some extent, we can still expose them through in-context impersonation.\n\nThis exposes a *difference* but it's unclear if it manifests a social bias. **Can you explain how these results connected to social biases?**\n\n5. What do you believe the primary contribution(s) of the paper to be?", "limitations": "The authors did adequately.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality:** This paper is not particularly original, but that is not something I count against it. Well executed experimental evaluations of \"common knowledge\" are an essential part of science.\n\n**Clarity:** The paper is clearly and compellingly written.\n\n**Significance:** It is difficult to assess the significance of this paper due to the concerns about the experimental results I identify below. If the authors' claims are correct then the paper would be highly significant.", "weaknesses": "**Over-Claiming**\n\nThe paper makes several false or overstated claims about its results and those of others. For example, it characterizes its relationship with the existing psychology literature as follows:\n\n> Importantly, LLMs impersonating older participants generate higher average rewards (β = 0.15, p < .001), thereby replicating a general pattern found in the developmental literature [59].\n> \n>  Lastly, we analyze how regression weights of the probit-regression were influenced by the age group the LLM is impersonating. Figure 2 (bottom right) reveals that LLMs pretending to be older explored their environment less (β = −0.06, p < .001) and exploited more (β = 0.05, p < .001). This pattern is in line with several results from the psychological literature which also found that children explore their environment more [60]. These results suggest that impersonating LLMs can recover human-like developmental stages of exploration.\n\nThese results are not obviously consistent with the claims made in [59] and [60]. For example, [60] finds \"... the two groups of children did not differ significantly in their extent of generalization\" and \"... exploration did not reliably differ by age group.\" The only age group studied in this paper that would be considered an adult by [60] is the 20-year olds, and its not at all clear that the effects in this paper are qualitatively or quantitatively similar to the ones in [60]. For [59], I am unable to identify which claims specifically the authors think that they reproduce. These problems are exacerbated by the fact that the authors do not release the raw data for me to analyze.\n\nThe authors claim \"Vicuna is competitive with proprietary services such as ChatGPT,\" however according to their own source users prefer the results from ChatGPT to Vicuna over 60% of the time. Additionally, when looking at how likely a user is to find the two models substitutable, there is consistently a gap of around 10% in how often users prefer ChatGPT over Model M compared to how often they prefer Vicuna over Model M for each third Model M in the benchmark.\n\nThe authors claim\n\n> For both LLMs, in both datasets, we observe that with increasing age, the complexity of the vocabulary and the attributes of the mentioned objects increases. A 2-year-old persona talks about the sound the bird or the car makes, the shapes of the wings or wheels, and the emotions attached to seeing or riding it. A 4-year-old persona interestingly mentions experiences seeing the bird or the car more distinctly. A 7-year-old persona starts using more complicated adjective phrases, e.g. can drive on rough roads and outside places, whereas a 13-year-old persona takes it one step further, e.g. brownish-gray body with distinctive rusty colored markings. Finally, a 20-year-old persona makes a more complete description of the object including where the bird is found or what the car is mainly used for. This is in line with [68] where the authors show that given the same length of text, smaller children use less diverse vocabulary, repeat a lot, and focus on non-academic vocabulary.\n\n**Missing Citations**\n\n[Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://arxiv.org/abs/2102.07350) was one of the earliest papers exploring personas in LLMs. It introduces a \"Master Translator\" prompt and finds that performance on translation tasks improves when using it. The same authors also wrote [Role-Play with Large Language Models](https://arxiv.org/abs/2305.16367), which is likely simultaneous with this paper and not something I judge the authors negative for omitting. However I think that the paper would be enhanced by drawing on some of the content in the paper.\n\nThe authors touch on Text-to-Image and VLMs, but largely omit reference to the extensive literature on using persona-like prompts to steer image generation. Most relevant is [VQGAN-CLIP](https://arxiv.org/abs/2204.08583) which explicitly discusses this phenomenon in terms very similar to the submitted paper, including prompting the model to generate \"a child's drawing of an X.\" Other papers such as [GLIDE](https://arxiv.org/abs/2112.10741) and [ImaGen](https://arxiv.org/abs/2205.11487) also prompt the models to impersonate specific artists and/or their styles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback on the point regarding association to child development. However, we disagree with the fact that our results are inconsistent with those papers. Our results are similar to the idea described in these papers that children tend to explore their environments in a more directed fashion than adults. It is, by now, a well-established finding that children tend to show a higher tendency to engage in directed exploration than adults. \n\n[60] studied both exploration and generalization across different age groups. They found that both directed and random exploration were reliably higher for children compared to adults. Similarly, we find increased directed exploration, as measured by the effect of relative uncertainty, for LLMs impersonating younger ages. [59] has a section on “directed exploration” which corresponds to the effect of relative uncertainty in our probit regression. This whole section is about how children tend to use directed exploration more than adults.\n\nTo further link our results to the literature on developmental psychology, we have modified this section: _This pattern is in line with several results from the psychological literature which also found that children show higher levels of directed exploration than adults [60]. These results suggest that impersonating LLMs can recover human-like developmental stages of exploration in a two-armed bandit task. If life is seen as an exploration-exploitation problem, then younger agents should show higher amounts of directed exploration [F,G]._\n\nLastly, we ensure reproducibility (including data generation) by releasing our code which is already included in the supplementary.\n[F] _Developmental changes in learning resemble stochastic optimization_, Giron, A. P., Ciranka, S., Schulz, E., van den Bos, W., Ruggeri, A., Meder, B., & Wu, C. M. (accepted). Nature Human Behaviour.\n\n[G] _Systematic exploration and uncertainty dominate young children's choices_, Blanco, N. J., & Sloutsky, V. M. (2021). Developmental Science, 24(2), e13026.\n### Vicuna vs ChatGPT\nWe will adjust that Vicuna is competitive with ChatGPT in some domains. Evaluating one of the strongest open source models at the time of submission is beneficial: results are easier to reproduce, which is important overall. Our observations on the trend are very similar for both models.\n\n### LLM vocabulary & grammar\nRather than arguing that LLMs faithfully represent the language of children, we qualitatively observe interesting patterns similar to humans. In our supp Sec3.1 with several text complexity metrics we quantitatively showed that the LLM descriptions of older personas are more complex than younger personas. \n\n### Missing Citations\nThe conceptual “Role-Play with Large Language Models” paper (no experiments presented) was not publicly available at the time of our submission. It argues that some of LLMs’ behaviors could be better understood as role-taking. We on the other hand, across the cognitive, language, and vision domains extensively evaluate the behavior of different LLMs when they were explicitly asked to take on different personas. \nWe adopted the meta-prompting strategy suggested in “Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm” to create variations of our impersonation prompts to study robustness w.r.t. impersonation prompt variations (results in the prompt variations section in the general comment).\nWe will add these two papers in our related work.\n\nWe have not investigated text-to-image models (there are many) as it is out of our scope but will mention them in our related work section, as they are also prompted with text.\n\n### Questions\n\n1. *Age groups & temperature:* We chose key developmental stages of early childhood, childhood, adolescence, and adulthood. Nonetheless, we reran our experiments (with all prompt variations) for more ages. We discuss these results in the general comment, where we find a significant effect of age on the reward for ages until 20 (β=0.15,p<.001), but no significant effect from ages 20-60 (β=-0.01,p<.1).\nDifferent temperatures were chosen based on the recommended default values of each LLM as this will reflect the most common use case but the same value may not generalize across LLMs, e.g. text properties depend on a variety of factors including training data, model scale, model architecture.\n2. *Official MMLU prompt:* Several MMLU evaluations [57, H], lead to small variations when comparing different models’ ranks. We include results with the MMLU official prompt (see Figure 1 under Orig. Ctx 1 IP), i.e. by using the MMLU prompt at the start and keeping our impersonation strategy. Our expertise-based impersonation trends still hold, and the absolute accuracy values improve. Indeed, our findings on impersonation are not dependent on the formulation of the task.\n3. *All MMLU tasks:* Due to space restrictions, we focus on the STEM and Humanities tasks but the expertise trends hold for the other domains. We include results for all domains in Figure 3, e.g for STEM the non-domain experts are all the non-STEM experts.\n4. *Social Biases:* We agree that the connection to stereotypical social biases is ambiguous. Our goal is not to test for alignment between LLM biases and human social biases, but to show the existence of LLM biases in sensitive categories such as gender and race. We will adjust this statement.\n5. *Primary contributions:* We systematically investigate impersonation over multiple domains (cognitive, visual, language) and architectures (VLMs + LLMs) which is further strengthened by the findings in this rebuttal. We show that dataset specific expertise can improve performance, but biases also exist.\n\n[H] _Holistic Evaluation of Language Models_, Bommasani, Rishi, Percy Liang, and Tony Lee. Annals of the New York Academy of Sciences (2023)."}]

Author rebuttal:
# General Response

We thank the reviewers for their helpful and insightful comments. As summarized by reviewer J5Gp, NysF and KqkQ, we present a comprehensive evaluation on the downstream impact of impersonation/roleplay with the use of LLM (two variants) and multimodal model (three variants of clip) in three tasks, namely multi-armed bandit (and show exploration-exploitation capabilities), language-based reasoning (and show expertise capabilities), and visual category description (and show multimodal understanding capabilities). To the best of our knowledge, ours is the first systematic evaluation that shows LLM, when impersonated, aligns to some behavior carried by the people (as pointed out by reviewer J5Gp) in the respective norm.

All reviewers (ZdXm,NysF,KqkQ,J5Gp) agree that our submission is clearly and compellingly written. Focusing on an interesting and novel topic that is less-known to the NLP community (NysF), our well executed experimental evaluations (ZdXm) provide a novel perspective to understand biases in language models and our society (NysF). Our approach to test our hypothesis across different domains and with various persona aspects by setting the persona specific to each task was well presented (KqkQ). A very interesting finding is that impersonation aligns to different behavior and downstream performances on three benchmarks (J5Gp). Reviewer ZdXm said “if the authors' claims are correct then the paper would be highly significant”. With this rebuttal, we would like to show with various additional experiments and extended discussions that our claims are indeed correct.

Our additional experiments requested by the reviewers helped us to clearly show that,
1. Our framework is robust to automatic prompt variations for in-context impersonation. Our results are independent of how the MMLU task is phrased and presented to the LLMs.
2. Our observations hold for the age effect when evaluating different specific ages and for the expertise effect when evaluating on additional vision datasets and MMLU tasks.

### Robustness to prompt variations (common request):
We show that our findings are robust to prompt variation of in-context impersonation and will update our manuscript accordingly. We follow the meta-prompting approach from [A] as suggested by reviewer ZdXm and automatically created 5 prompt variations:
- Should you be transformed into a {persona}
- Imagine you are a {persona}
- Should you assume the role of a {persona}
- Were you to take on the persona of a {persona}
- In the case of you being a {persona}

We ran all our experiments (multi-armed bandit, reasoning and vision tasks) with these additional prompts and averaged over the 6 prompt variations.

In the bandit task, we see a significant effect of age on the received reward which was β=0.15 (p<.001) for “If you were a {persona}” and β=0.16 (p<.001) with all prompt variations. Similarly, the same effect of age on exploration (β=−0.06, p<.001) and exploitation (β=0.05, p<.001) can still be observed on average for all prompts: β=−0.03, p<.001 and β=0.04, p<.001, respectively.

On the MMLU reasoning task (see Fig. 1 _Our Ctx 6 IP_), the expertise trend remains the same with these additional prompt variants, i.e. the task expert outperforms the domain expert which, in turn, outperforms the non-domain expert.

In the vision tasks, for CUB we see the same significant effect of age (see Fig. 5 top) that we had originally reported: Descriptions generated by older personas outperform those by younger personas. Albeit decreased in effect size, we find that expertise and race has an effect on downstream performance. Conversely, the effect of gender is even larger than what we had previously reported.
On Stanford Cars (Fig. 5 bottom) we see slightly reduced effects of race and gender, but the effect of expertise is very clear. The age effect, that older personas produce better descriptions, becomes even more clear.

### Neutral baseline (requested by reviewers KqkQ and DtKQ (ethics))
We include results on the reasoning task with the neutral baseline in Fig. 1. We consider the average of 4 neutral personas: student, average student, person, and average person. As expected, the neutral persona performs worse than or on par with the non-domain expert. We will update the supplementary with results for all domains, all of which show the same behavior.
We also evaluate appropriate neutral personas for the vision task, namely average person and person. On CUB we find for Vicuna-13B that the dataset specific expert outperforms the average neutral persona. 

### Age evaluation (requested by reviewers ZdXm and J5Gp)
We ran additional experiments with Vicuna-13B (all prompt variations) extending the age-range beyond age 20 up to age 60 on the bandit task. We consider steps of 2 from 2 until 30 and steps of 5 from 30 until 60. The effect of age on the task performance diminishes after the age of 20 and becomes more stable. While between ages 2-20, there is a clear effect of age on the reward (β=0.16,p<.001), between ages 20-60, the effect is not significant (β=-0.01,p<.1).
From age 2 to 20 exploration decreases (β=0.04,p<.001) while exploitation increases (β=0.03,p<.001). After the age of 20, both exploration and exploitation decrease (β=-0.02,p<.001 and β=-0.02,p<.001). 
We also evaluate more ages on the vision task and observe a similar trend. On CUB, we observe a positive effect on the classification accuracy for ages 2 to 20 (β=0.69,p<.001), but the effect is insignificant for ages 20 to 60 (p-value of 0.101).
In conclusion, these observations further strengthen our initial claims that LLMs are able to impersonate the behavior of early developmental stages of humans. We will add these results and a discussion to the paper.

[A]: _Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm_, Reynolds, L. and McDonell, K. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (2021).

Meta-review: The authors explore the effects of roles (social identity, domain expertise) in LLM instructions on downstream tasks. The findings are interesting, showing that models respond differently when instructed to follow different roles for the same task.

All reviewers agree that this is quality work, pointing out the timeliness of this study, the impact it can have in our society, and the clarity of presentation. The reviewers, however, also raised some concerns regarding the robustness of the method, the role of age, as well as some ethical concerns. The authors wrote comprehensive responses to reviewers and ethical committee and most concerns were addressed. For these reasons, I am happy to accept this paper.
Decision: Accept (spotlight)