Human Simulacra: Benchmarking the Personification of Large Language Models
Authors: Qiujie Xie
Qiming Feng
Tianqi Zhang
Qingqiu Li
Linyi Yang
Yuejie Zhang
Rui Feng
Liang He
Shang Gao
Yue Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Human simulation
Primary area: datasets and benchmarks
TL;DR: We are the first to introduce a benchmark for the personification of large language models, including high-quality data, rigorous and innovative evaluation methods, and comprehensive benchmark tests.

Abstract: Large Language Models (LLMs) are recognized as systems that closely mimic aspects of human intelligence. This capability has attracted the attention of the social science community, who see the potential in leveraging LLMs to replace human participants in experiments, thereby reducing research costs and complexity. In this paper, we introduce a benchmark for LLMs personification, including a strategy for constructing virtual characters' life stories from the ground up, a Multi-Agent Cognitive Mechanism capable of simulating human cognitive processes, and a psychology-guided evaluation method to assess human simulations from both self and observational perspectives. Experimental results demonstrate that our constructed simulacra can produce personified responses that align with their target characters.  We hope this work will serve as a benchmark in the field of human simulation, paving the way for future research.

Published as a conference paper at ICLR 2025
HUMAN SIMULACRA: BENCHMARKING THE PERSONI- FICATION OF LARGE LANGUAGE MODELS
Qiujie Xie' Qiming Feng' Tianqi Zhang? Qingyiu Li' Linyi Yang Yuejie Z Rui Fengf* Liang He Shang Gao Yue Zhang? 7,8
"ccooo of Computer Science, Shanghai Key Lab of Intelligent Information Processing.
Shanghai Collaborative Innovation Center of Intelligent Visual Computing, Fudan University "Tongji University 'University College London 'Huawii Noah's 's Ark Lab
Shanghaii Key Laboratory of Mental Health and Psychological Crisis Intervention
Deakin University Westlake University Weestlake Institute for Advanced Study
ABSTRACT
Large Language Models (LLMs) are recognized as systems that closely mimic aspects of human intelligence. This capability has attracted the attention of the social science community, who see the potential in leveraging LLMs to replace human participants in experiments, thereby reducing research costs and complexity. In this paper, we introduce a benchmark for LLMs personification, including a strategy for constructing virtual characters' life stories from the ground up. a Multi- Agent Cognitive Mechanism capable of simulating human cognitive processes, and psychology-guided evaluation method to assess human simulations from both self and observational perspectives. Experimental results demonstrate that our constructed simulacra can produce personified responses that align with their target characters. We hope this work will serve as benchmark in the field of human simulation, paving the way for future research.
INTRODUCTION
Researchers in psychology and sociology have long relied on human participants to conduct ex- periments that explore patterns of human behaviors and mental states (Camerer et al. [018]] Folke
al. 2016 Qiu et al.] 2017). However, this method often faces numerous challenges, including the difficulty of recruiting participants (Radford et al. 2016 [elson 1960), high uncertainty (Haslam &
McGarty 2001), and potential ethical considerations (El-Hay 2019). this context, the potential of
large language models (LLMs) to mimic human behaviors has garnered increasing attention (Ziems)
ct al.) 2024 Zhang et al. 2023a Coda-Forno et al. 2024). Psychologists and sociologists are explor-
ing the use of LLMs to replace human participants, aiming reduce costs and complexity while
avoiding potential ethical considerations (Demszky et al. 2023: Dillion et al. 2023
Hutson 2023
Grossmann et al. 2023 et al. 2023b Kjell al. 2023
Despite these advancements, current LLM-based human simulations are still limited in only sim- ulating group studies (Li et al. 2023b Zhao et al. 2024). giving rather inconsistent performance
across different tasks (Dillion et al. 2223 2023). and lack the depth in capturing complex
characteristics of human behaviors (Grossmann al. 2023) Kjell et al. 2023 Hagendorff al.
2023 Yin et al.) 2024 Jones & Bergen 2024). To address these issues, we consider different perspective, proposing psychology-driven simulacrum that aims to produce consistent behaviors indistinguishable from humans. To this end, we introduce high-quality dataset, a comprehensive evaluation pipeline, and unified benchmark, as shown Figure Using the proposed benchmark, empirically discuss the research question: How far are LLMs from replacing human subjects in psychological and sociological experiments?
Rigorous personality modeling is crucial for human simulation as it ensures more realistic repre- sentations of human behavior and interactions. However, personality is a complex concept that is difficult to model. Prior studies in personality modeling (Pan & Zengl 2023 Tu et al. 2023 Song et al. 2024. Wang et al. (023b) use establstall frameworks like the Jnyers-Briggs Type
Indicator (MBTI) (Myers] 1962 ise Huang et al. 2024 Huang al. 2023). Despite the popularity,

---

Published as a conference paper at ICLR 2025
Co If RAR
0 Bopert
- Diserver Repart
Paycholory
Life
Human Feedbock
Smiben Joani
1010
① Harsan Sinilicra Datsset
MiNI-AAAe Cagalfive Mechanism
b @Pochhle-ppulall Evalusiion
Figure I: Overview of the proposed benchmark.
these personality models face critical limitations. For example, LLMs' internal representation of psychological types may be inherently flawed or hallucinatory, given that they operate as "black boxes." Besides, personality is a multifaceted and dynamic construct that cannot be accurately reduced to single type by these models. Inspired by Jung's psychology theory (Corr & Matthews 2020; Mussel et al. 2016 Hogan et al., 1997 Jung 1923). we employ eight-dimensional strategy to address the LLa-b personality modeling issues. By dividing personality into eight comple- mentary tendencies, we provide more comprehensive framework with 640 detailed trait descriptions (Table 7 This approach allows for a more nuanced depiction of personality ($3.2) when constructing virtual characters, enhancing the variety of characters in "personified machines".
Identifying suitable targets for human simulation also poses significant challenges. One approach involves using role-playing datasets composed of fragmented information about genuine charac-
ters (Wang et al.] [202cc Zhou et al. [2023] [hao et al., 2023) (e.g., Albert Einstein, Beethoven).
However, the simulations of existing characters are prone to be disrupted by hallucination (Mallen
et al. 2022: Wang et al. 2023a) produced by LLMs. The fragmented data also fail to provide a comprehensive depiction of character, especially for psychological experiments. Consequently, we build virtual character dataset, named Human Simulacra, and use the characters' detailed life stories as the basis for simulations, which also avoids the potential ethical and legal risks of using historical figures. To this end, we decompose the task of crafting detailed life story into interrelated subtasks and further propose a human-in-the-loop strategy that tackles each subtask with human feedback ($3). Our dataset contains 129k texts across 11 virtual characters, with each character having unique attributes, biographics, and stories (Figure 9.
Given the complexity of the human simulation, measuring the "personified machines". We expand the traditional self-report method (Park et al.
propose a novel evaluation framework for
2023
Gupta et [ll.4222) Li et al. 2024) to two-phase evaluation method, combining self reports
4.. and observer reports( (4.22 2). based on established personality measurement theories (Corr &
Matthews 2020: Mussel et 2016 Hogann et al. 1997] Jung. 1923). Our evaluation provides suitable and robust testbed for exploring the opportunity of replacing human participants with LLM agents. Furthermore, to mimic the complex nature of human beings, we introduce novel Multi- Agent Cognitive Mechanism (MACM) that simulates the human brain's 's information processing systems ($4.3). As an external module, this mechanism enables the LLMs to remember background stories, understand target personalities, and express accurate emotions in complex situations.
Based on our Human Simulacra dataset, we conduct an empirical study involving 14 widely LLMs with 4 different auxiliary methods (None, Prompl, Retrieval Augmented Generation (RAG), and MACM) using experimental settings (self reports, observer reports, and psychology experiment on conformity). Extensive results reveal that although the lop-performing model approaches human performance levels (88.00% on GPT-4-Turbo Turbo) in self-report evaluations, it struggles in observer reports, achieving only 77.75% even with MACM support. In our conformity test, LLM agents exhibited submissive responses similar to humans, albeit with a more robotic and rigid demeanor.
To our knowledge, theory (Jung, 1923) and conduct standard human simulation experiments. We offer high-quality data, rigorous and innovative evaluation methods, and comprehensive benchmark tests. Our findings suggest the potential use of L.M agents as substitutes for humans in psychological experiments, shedding light on futuree applications of human simulacra.
the first to build human simulation data based on Jung's psychology
2 RELATED WORK
Memory Systems in Cognitive Psychology. In cognitive psychology, information processing approaches assert that cognition encompasses the entire process through which sensory inputs are transformed, reduced, elaborated, stored, retrieved, and used (Neisser 1976. Newell et al. 1972
2

---

Published as a conference paper at ICLR 2025
Naca Gicho M4xa
Generution
Story Iteration
Story leeraion
lana for lngical lnaroi
FP7 US
Chank Selection
Attributes
Namo, Goodes Ago Occepetion
Candidate Attributes
Profiee e LLM
Life Story
Expandable
Chonk
Profile
Exponded
Noree Same Octon, Agr 44, reputian Mecal Openttr
Chunk
11M
Candidate Profiies
Character Hiography
Profilee Selection
Bigrraphy
T
East Torn. Grandaz
Profile Selection
Story Iteration
Ranking
Mannual Check
Refining
Espericaces
"Marty Childhood" "Learit
Chank Selection
db
Life Story
Is ssall desoreess of East Teom. (ueneg Sere Oclen was a
Realistie Profiles
Life Story
Quality Chasking
Scoring
Figure 2: Process of constructing life stories for characters. At each step, humans are involved in thoroughly reviewing the generated content, ensuring it is free from biases and harmful information.
Dawes et al. 2020: Paas & van Merriënboer 2020). Atkinson Shiffrin (1968) proposed a Multi Store Model of memory that divides memory into sensory memory, short-term memory, and long-term memory. Baddeley & Hitch (1974) distinguished the concept of working memory from short-term memory, emphasizing that working memory is born for storing, invoking, and analyzing information. In this paper, based on the memory theories (Atkinson & Shiffrin 1968; Baadellyy & Hitch 1974
Baddeley et al., 1984 Izquierdo al. 1999 Norris 2017) discussed above and the capabilities of LLMs (Zhao et al. 2023), we propose a Multi-Aentt Cognitive Mechanism. It is designed to enhance the ability of LLMs to impersonate humans by transforming narrative life story into long-term memories and engaging with the external world in human-cognitive manner.
Role-playing, Role-playing tasks (Chen al., 2024) focus on simulating characters with dis- tinctive personalities. This line of work includes replicating the professional skills (Salewski al. 2024) |Hong et al., 2023] Binz & Schulz, 2023) and portraying the outward characteristics 2023] Tu et al. 2023 Wang et al.
Table I: Differences between Human Simulacra and Role-playing datasets.
Features
Ours
Character-LLMM
Role-LLM
Virtual Characters
Life Story
X
(Shao et al.
Psychology Support
X
2023c Li al. 2023a] Yu al. 2024; ZZouu
Human Feedback
et al. 2023) of target personas. Our work differs
from existing role-playing studies in two key aspects: 1) Our work is grounded in psychological
theories to ensure rigor in deep simulation of human personalities. Role-playing works do not need to follow psychological principles like our method does, and they
intended for uses that require
a deep imitation of human patterns (e.g., instinct (Tinbergen 2020: Marler 2014). conditioning Cllak al., ). 2) While role-playing works typically portray characters from the past whose characteristics are generally fixed and immutable, our method constructs and simulates virtual human beings from scratch. This allows us to explore various personality combinations without being limited to existing roles and mitigates the factual errors or hallucinations that LLMs may generate when simulating real people. The difference between our dataset and existing role-playing datasets is shown in Table I We propose the first simulacra dataset that combines features of "virttul characters", "life story", "psychology support", and "human feedback".
HUMAN SIMULACRA DATASET
We break down the generation of character data into solvable sub-tasks (c.g., profiles and short biographies) by introducing a structured information model as shown in the left part of Figure 2 This model organizes the character's information into five inter-connected layers (e.g., character attributes and character biography). From bottom to top, the information becomes more concise, focusing on the character's most essential facts. Based on this information model, we decompose the task of generating a character's 's life story into interconnected subtasks and design a semi-automated strategy iteratively build a detailed life story for the target character. The entire process depicted in the
CCharacter-II.MM (hao al. 2023) Role-I.LM Wang al. 2023e
3

---

Published as a conference paper at ICLR 2025
Mary Temi
Sam
Robert
Secatter tewek
Brnk 212 Gusterde
Micbol
m
15-
12.3K
12.5K
11.5K
Oer Berk, 201
11.3K
11-
10.K
10.
Det
Hake Aoontta Ststos
M4ay 11er
Sun
Kodadd
0192
(a)
(b)
(c)
Figure 3: Human Simulacra dataset. (a) Profiles of virtual characters. (b) Personalities of characters, displayed in radar chart based Jung's sight-dimensional theory. Line: character; Te Si: abbrevs for personality dimensions. (c) Word count of life stories for each virtual character.
right part of Figure particular, we first generate 100 candidate profiles (varying in quality) and select I virtual characters as the protagonists based on their backgrounds. The selection details are provided in Appendix [A.3] We then employ the GPT-3.5-Turbo model (Brown) 2020) as the data generator with
lesquenccy_penalle of 1.0 and top_p of 0.95. Each life story is expanded through at least 50 rounds of iteration. At the end of cach story itcration, multiple human reviewers, including graduate students in computer science and psychology, thoroughly review the content to ensure it is free from biases, discrimination, or harmful information.
3.1 CHARACTER ATTRIBUTES
Character attributes encapsulate the core facts of a virtual character, serving as anchor points for the life story of the character. While designing attributes, it necessary to ensure that the attributes are diverse, have reasonable connections, and conform to natural laws. Following previous studies (Sloan |ark|PPark et al. 2023), we design a comprehensive attribute set for virtual characters, encompassing [nume, age, gender, date of birth, occupation, personality traits, hobbies, family background, educational buckground, short-term goals, and long-term goals} (Figure Ba). Each attribute has candidate pool, covering diverse values applicable to most people. For instance, based on the International Standard Classification Occupations (ISCO-08). we select 76 common occupations as the occupation candidate pool. More details about the attribute systems provided in Appendix A.2
3.2 PERSONALITY MODELING
Considering that personality encompasses an entity's ' characteristic patterns of thought, feeling, and behavior (Hogan et al. 1997), how to accurately model the personality traits of the target character becomes core challenge in attribute design. We adopt the eight-dimensional theory derived from Jung's 's study (Jung] 1923) to accurately model the personality traits of the target character. This theory divides personality into eight tendencies such as extraverted thinking (Te) and introverted sensing (Si), with each tendency serving as complementary facet.
Contrary to directly assigning numerical values to these tendencies, we employ a relative ranking strategy to indirectly assess the strength of each personality tendency within the character. Specifically, we rank the eight tendencies and establish a guideline that the tendencies at the top and bottom of the order more pronounced in the character's personality, while those in the middle are less pronounced, manifesting a blend of traits that vary in direction. Under the guidance of psychology professionals, we prepare 10 suitable deseriptions for each possible ranking, with each description corresponding to an aspect of the tendency in daily life. Ultimately, we form a personality candidate pool containing 640 trait descriptions (Figure 3b Our personality modeling method grounded in authoritative, field-recognized theories (Jung 1923), aims to depict the character's personality more comprehensively and specifically. Example descriptions for the extraverted intuition tendency are deferred to Appendix Table
3.3 CHARACTER PROFILE AND LIFE STORY GENERATION
To assemble the character's ' profile, we first generate draft profiles by randomly selecting attribute values from their corresponding pools. Then, we add Profile Selection module responsible for
4

---

famous and Published as a conference paper at ICLR 2025
You're hungry and want eat your lavorite restaurants your neigh- borhood that offer the dish for the sams price. One of the restuurants always crowded, while other is small diner you fre- quent. How do you behave?
What name? ?
Who Do have goals? parcents?
Kind?
Q
How What is my faverite color? Who
boblies??
Homan
Judge
Analyse
Human
going
Human
Life Story
Human
Shortly before the meetting, your friend asks you if hesshe bring other friends you don't know, there be in the end, he or she cannot say. How do you bohave?
?
many people
Human
Judge
Output
Life Story
Human
Simulacrum
44)
A
more...
Self Report
Observer Report
Scenario Examples
Figure 4: Psychology-guidee evaluation. Self reports assess simulacra's is self-awareness through character-specific questions based on their life stories. Observer report evaluates simulacra's realism by creating
assessments analyzed by human judges.
quality check and profile refinement the generation process, as shown the right part Figure 2 In this way, high-quality profiles are manually filtered out and fed into the I.L to generate short biography summarizing the character's ' life experience.
After obtaining the brief biography for the character, we progressively enrich the biography with human feedback, transforming into a detailed life story after 7 iterations. Specifically, in each iteration, perform: 1) Quality check: manually inspect the generated content for its rationality, and ensure is free from biases, discrimination, or harmful information: 2) Chunking: divide the story into separate chunks; 3) Scoring: for each chunk, calculate its Importance, Elaborateness, and Redundancy, then select chunks with high importance, low elaborateness and redundancy for expansion; and 4) Expanding: prompt the LLM expand the selected chunks and add reasonable life experiences to the story. Finally, we create the virtual character dataset Human Simulacra, comprising about 129k texts across 11 virtual characters (Figure 3c). See Algorithm Appendix A for construction details, and Appendix for relevant prompts.
iterative generation method to
PSYCHOLOGGY-GIDDER EVALUATION
We propose psychology-guiee evaluation framework as shown in Figure 4 This framework draws on psychological assessment techniques (Hogan et al.) 1997: Mussel et al. 2016), including self reports, observer reports, and the Multi-Agent Cognitive Mechanism to generate responses.
4.1 SELF REPORT
Self-reporting common personality measurement technique that requires individuals to answer questions about themselves (Hogan al. 1997 Corr& & Matthews 2020p. It refers the degree to which an individual aware their own identities, thoughts, and values. We employ self-report assessments to evaluate the simulacra's ability establish self-awareness, testing their memory and analytical capabilities regarding their character information. To this end, we manually craft a set of questionnaires for each virtual character, featuring fill-in-the-bland blank and single/multiple-chote questions. Each question is carefully reviewed to ensure they reflect the character's 's unique nature and the scores are evaluated based exact matches. The test content covers key attributes. social relationships. and life experiences of the target characters. For example. "What is your name?". "What do you think your father?". and "What were the reasons behind not going through formal schooling for you?". ? See Appendix D. for additional example questionnaires.
4.2 OBSERVER REPORT
seff-assesmeett tests are insufficient measures LLM personality due to potential biases and the inability capture complex human behaviors accurately (Gupta al. 2024). A high self-report
5

---

Published as a conference paper at ICLR 2025
Thinklng Apuat
Legivll kedryss (Hieway Continliis
Lopdder Mener Geamskke
Messory Agent
Top Agent
Chodo
Thakking hprne)
Honey Botiend
Meeery
Thaviees
Exsstion Agent
Colaleettie Capiiie
Fouche
Conteet Th
Top Azci
Sentiee Anadya (ilmmar Commodii
Reggas
Nast--gent Cogatiee
Figure 5: Multi-Agent Cognitive Mechanism. It involves four LLM--riven agents: Thinking Agent / Emotion Agent handles logical/emotional analysis & memory construction. Memory Agent manages retrieval of memories, while Top Agent coordinates all activities. Upon receiving stimulus, these agents collaborate to generate appropriate responses, simulating complex human cognitive processes.
score only indicates that the simulacrum possesses a clear understanding of the target character. does not sufficiently prove the simulacrum's 's ability adopt behaviors consistent with their character in real-life scenarios. For a comprehensive evaluation, we need to further observe the simulacrum's thinking, emotions, and actions in real-life scenarios from a third-party perspective. Therefore, in addition to self reports, we further introduce observer reports, a cross-cvaluation based on human judges, aiming to assess the simulacrum's thinking, emotions, and actions real-life scenarios from a third-party perspective.
Specifically, following Mussel et al. (0166) we crawl 55 hypothetical scenarios that could elicit human emotional responses or personality traits. Two examples of such scenarios are displayed in the right part of Figure 4 We require each simulacrum to imagine that they are in the given scenario and to describe how they would feel and what actions they would take. All responses are collected and submitted for cross-evaluation, which includes two inter-related subprocesses: 1) Human judges and 2 analyze the scenario (Q) and response (A), and describe the respondent's 's personality. Subsequently, judges 3 and 4, informed by the target character's 's life story, determine whether the descriptions given by judges 1 and 2 match the target character. A discrepancy indicates that the simulacrum has deviated from the character, showing a simulation error. 2) Considering potential bias in a single assessment, ask judges 3 and 4 to thoroughly read the target character's life story, and answer how they would feel and what actions they might take in the scenario if they were the character. Then, judges 1 and 2 compare the similarity between the human responses and the simulacrum's responses. A high degree of similarity indicates a high-quality simulation, which is consistent with the expectations of the character. More examples of hypothetical scenarios, the selection criteria for human judges, and the evaluation guidelines are provided in Appendix D.2
4.3 MULTI-AGENT COGNITIVE MECHANISM
Following the aforementioned process in $3 craft life story for each virtual character. Given the limited context, current L.MSs may not be able to accurately capture the character's 's personality and inherent emotional tendencies from the narrative. To address this issue, we propose Multi-Agent Cognitive Mechanism (MACM, Appendix B) based on cognitive psychology theories (Atkinson & Shiffrin 1968, Norris 2017). This mechanism utilizes multiple LLM-based agents simulate human brain's information processing and memory systems, thereby enhancing the quality of simulacra. The ablation study on the structure of MACM provided in Appendix B.3
As illustrated in Figure MACM has two key processes: 1) Long-term memory construction and 2) multi-agent collaborative cognition. Specifically, When simulating character from the Human Simulacra dataset, MACM first transforms the target character's 's narrative life story into long-tern memories that are richer in detail, fuller in emotion, and clearer in structure. Then, to mimic human cognition, MACM further utilizes collaborative process that allows LLMs to leverage long-term memory and engage with the external world in a cognitive manner. Upon receiving a stimulus, for example, question from friend, Top Agent first analyzes the question and evokes Memory Agent for memory retrieval. The retrieved results are stored in working memory. Then, Top Agent sends the relevant memories and question Thinking Agent and Emotion Agent for logical and emotional

---

Published as a conference paper at ICLR 2025
Table 2: Self reports of 14 LLM-based simulacra. Each character is tested by its own set of questionnaires containing cloze, single-choice (SC) questions, and multiple-choice (MC) questions. The simulacra are divided into different groups based on their parameter size. The best-performing simulacrum in each group is highlighted in light gray.
Method
None
Prompt
RAG
MACM (Ouns)
Questiion Type
Close
SC
MC
Sum
Cloze
SC
MC
Sum
Close
SC
MC
Sum
Close
SC
MC
Sum
75.67
20.00
20.00
GPT-4-Turb
0.00
8.00
4.00
12.00
18.67
20.00
40.00
78.67
20.00
20.00
45.33
85.33
20.00
20.00
45.00
88.00
Clando S-Opu
0.00
2.67
8.00
18.67
38.67
77.33
0.00
13.33
52.00
20.00
20.00
41.33
Limma-277
0.00
5:33
8.00
13.33
10.67
16.00
17.33
44.00
0.00
9.33
16.00
9.33
8,00
8.00
23
Vieuma 76
0.00
8.00
4.00
12.00
14.67
12.00
14.67
41.33
1,33
9.33
10.67
21.33
13.33
6.67
9.33
29.33
1.33
L-ama-2-33b
0.00
8.00
9.33
17.33
9.33
9:33
12.00
30.67
0.00
8.00
13:33
21.33
9.33
4,00
9.33
22.67
Vicura-13b
0.00
9:33
9.33
18.67
20.00
17:33
18.67
56.00
0.00
14.67
14.67
29.33
14.67
14:67
16.00
45.33
Clande-3-Haaie
20.00
Mistral-8s76
0.00
10.67
8.00
18.67
16.00
20.00
24.00
60.00
1.33
17.33
22.67
41.33
12.00
16.00
21.33
49.33
Liama 2-700
9:33
2.67
12.00
16.00
17.33
48.00
0.00
5.33
12.00
17:33
20.00
17.33
18.67
56.00
Maaa-2--0hh-hha
0.00
10.67
6.67
17.33
16.00
48.00
4.00
13.33
20.00
30.00
18.66
Qwen carbo
0.00
9.33
14.67
24.00
16.00
20.00
33.33
09.33
20.00
20.00
32.00
72.00
20.00
20.00
34.67
74.67
Claudo-3-Somer
13.33
36.00
0.00
20.00
36.00
Human
20.00
20.00
60.00
100.00
Table 3: Observer reports of different simulacra on GPT-4-Turbo. Description Matching Score evaluates simulacrum's 's alignment with target personality. Response Similarity Score estimates similarity between external expectations and simulacrum's behaviors. ICC represents the Intraclass Correlation Coefficient between judges.
Description Matching Score
Response Similarity Score
Method
Final Score
Judge
Judgo 4
Average
ICC
Judge
Judge 2
Average
ICC
Prompt
32.00
33.00
32.50
39.00
34.00
36.50
69.00
36.00
37,50
0.86
28.00
28.00
0.95
MACM (Ours)
35.00
36.00
35.50
41.00
43,00
42,00
77.50
analysis and stores the outcomes in working memory. Finally, Top Agent formulates a response based on the contents of working memory. Due to the limited context window of LLMs, content that cannot be accommodated in working memory is dynamically transferred short memory, which will be converted into long-ted term memory when rehearsed.
EXPERIMENTS
In this section, introduce the empirical study involving 14 widely-used LLMs with different simulation methods (None, Prompt, RAG, and MACM) using 3 experimental settings (self reports, observer reports, and psychology experiment on conformity) on the Human Simulacra dataset.
5.1 PSYCHOL.OGY-GUIDER EVALUATION RESULTS
Experimental settings. To evaluate the human simulation ability of different LLMs, we experiment with mainstream LLM--based simulacra using the psychology-gun guided evaluation method proposed in $4 We compare the proposed MACM with the following methods: 1) Blank model, which does not know any information about the target character. 2) Prompt-based method. We prompt the LLM to simulate the target character, with the help of the character's 's attributes and brief biography. 3) Retrieval-augmenned generation method. A combination of prompt- based method and a retrieval module. In this case, the retrieval module searches the character's 's life story based on the input and returns the three most relevant paragraphs.
Self reports. Table 2 presents the results of self reports, with all outcomes being the average of three repeated tests. Based on the data presented in Table we have the following observations:
1) Even without any knowledge of the target character, the LLMs still score certain points (e.g.. 12.00 on Vicuna-7ba (bona))--None) on these single- or multiple-choice questions by random guessing. This indicates that relying solely these questions for evaluation is not sufficiently reliable.
7

---

Published as a conference paper at ICLR 2025
2) As the size of the LLMs' parameters increases, their capability gradually increases, leading to clearer celf-wwareness and an upward trend in self-report scores. For instance, when comparing Vicuna-7b-Ran RRR with Vieuna- 33-RAG, the score increases from 21.33 to 29.33.
3) Since the self-report test is conducted in conversational manner, the LLMs fine-tuned for conversational scenarios tend perform better than foundation models (e.g., 25.33 on Llama-2-7h- MACM and 29.33 on Vicuna-7b-MACM)
4) While the RAG-based simulacra can retrieve relevant life story chunks when answering questions, their performance is constrained by the LLMs' information processing capacities. A large amount of descriptive information may interfere with the LLMs' self-positioning. resulting in inappropriate responses or misunderstandinng of questions. Hence, in most weeker-performmn LLMs, RAG-based simulacra score lower than Prompt-based ones.
5) In stooger-performing LLMs like GPT-4 and GPT-4-Turbo, the MACM-based simulacra achieve the best results (88 points) in all tests, aided by emotional and logical analysis. However, the effectiveness of the MACM method remains constrained by the LLMs' analytical capabilities.
Observer report. For a more comprehensive evaluation, we select GPT-4-Turbo as the baseline model and recruit several human judges with fair understanding of psychology to conduct external observations of the simulacra. These judges include individuals with psychology master's degrees, computer science graduate students, and professionals from psychological laboratories. We calculate the average score from two judges for the same assessment task as the simulacrum's 's final score.
Experimental results from Table 3 indicate that while the RAG-based simulacra perform well on the self-report tests when compared to the Prompt-based ones, the retrieved story segments do not significantly enhance the simulacra's 's ability to accurately mimic their target character's 's personality, thoughts, and actions. In contrast, the AACM-basee simulacra not only extract context-releven emotionally and logically rich memory fragments from long-term memory but also conduct divergent analysis for the current situation. During observation, the MACM-based simulacra better reflects thoughts and behaviors consistent with their target character's personality, achieving more authentic simulations from the inside out. We also found that when assisting LLMs with human simulations through external methods like MACM, the choice of LLMs is constrained to high-capability models (e.g., GPT-4-Turro for high- high-quality simulations with higher costs. The solution to this problem might lie in adjusting the LLM's parameters to align with the target character's values, which will be a primary focus of our future work.
5.2 PSYCHOLOGICAL EXPERIMENT REPLICATION
How close are LLMs to replacing human subjects in psychological and sociological experiments? We answer this question by employ- ing the most advanced simulacra (in this case, GPT-4-Turbo) repli- cate the bandwagon effect from psychology (Appendix DD. which describes the tendency for people adopt certain behaviors simply because others are doing so (Asch 1956 Beannitt-Beckk, 20133 Asch
2016). Emulating the Asch conformity experiment (Asch (9566), we analyze group dynamics and individual responses of the Asch confor- mity experiment as shown in Figures and g By following formal psychological experimental protocols, we assess whether MACM- based human simulation can capture aspects of human behavior, possibly substituting human participants in simple experiments.
Target
A
B
C
Figure 6: A discrimination ample. Line A matches the length of Target line.
Experimental settings. Following (Asch] 1956; 2016), we arrange 18 trials for the simulacra. In each trial, the simulacra are invited to complete a simple discrimination task with seven other individuals, which requires them match the length of a given line with one of three unequal lines. An example of the discrimination task is shown in Figure To study whether simulacra yields to group pressures like humans, we select 12 of these 18 trials as critical trials, following the settings (Asch) 1956). In each critical trial, all individuals except the simulacra are told to stand up and announce incorrect answer (e.g.. declaring that line B matches the length of the Target line in Figure 6). This creates conditions that induce the simulacra to either resist yield to group pressures. We simulate and test 11 virtual characters from Human Simulacra and record their responses in each critical trial, calculating the average correct rate (Figure 7 Similar (Ashh 1956), also conduct interview
8

---

Published as a conference paper at ICLR 2025
Human
Chameteraii
Ours
100
eqprrienns darlng this capertauet?
Correct Rate (%6
Group Presuue
Charncler lnit Intreduction
Respesie
Proceatin;:
Namte: Mary Junm
tegatite
perplcsd
rasked Salooe: 11 I. To, 8t, No. Fc. Lirra sorsg Israaed
senwer is wallt sessaal siker' rapasin: trast my end
25
0
Charneter Intraduction
"oone:Fdra Waller
Respoase
Peneealiny Mt Puncin captiie Ban-
Fo baflee, fediag srcrehdand diagriigg desnting in with the sa rusfier mnistai harmany.
Critical Trial Number
dase rasked
Latue withued
So.
to aviid sperting others. car sboat maail wten dtal. mad pat
Figure 7: Group analysis of bandwagon effect. Humans fluctuate due to group pressure. Char- acterai shows an inability to resist group pres- sure. Our MACM maintains a performance close to human levels but with less variability.
Figure 8: Interview responses from two representa- tive simulacra. Impacts of bandwagon effect vary among individuals based on their personalities.
with each simulacrum after the experiment. The interview provides the reasons concerning the simulacra's 's reactions to the experimental condition (Figure S
Group analysis. We compare the average correct rate of MACM on 12 critical trials with 1) Characterai, a chatbot service that features a powerful LLM specifically trained for simulations and supports long text inputs as prompts, and 2) human results reported in (Asch) 1956). Results are presented in Figure 7 and we provide additional analysis with different simulation methods (e.g.. RAG) in Appendix Based on the results, we have the following observations:
1) The discrimination task used in the trials is simple. When there is no group pressure, humans will achieve nearly 100% accuracy (as declared in (Asch) (956]). Therefore, the area between the 100% correct rate and the human result (green line) represents the group pressure, which causes humans to obtain a lower and more fluctuating correct rate on each trial.
2) Character.ai overlooks the difference between each character's personality and instead displays a robotic response to group pressure (accepting the group's 's errors without resistance). Therefore, it achieves 0% correct rate in all critical trials (red line). This phenomenon reflects the lack of holistic emulation of inner patterns of character.
3) Although our MACM aligns with the human trend better than Characteraii, it is evident that the human curve fluctuates while MACM does not. This is because individuals' emotions are influenced by increasing pressure from the majority throughout the experiment, causing the average correct rate to fluctuate. Our simulation portrays resilience for determined personalities and compliance for weak personalities. It exhibits robotic behavior: determined personalities are resolute, and weak personalities are absolutely submissive. It does not capture the complexity of real humans, who might start off determined but yield to majority pressure over time. MACM displays submissive reactions akin to those of humans, albeit with a more robotic and inflexible demeanor.
Individual interview. Based on the analysis of the interview results, we find that the simulacra's responses can be categorized into two groups: those with a resolute personality who remain unaffected by external influences and consistently make the right choices, and those with a more compliant personality who tend to conform to others. We display two representative simulacra (Mary and Frica) from each group in Figure B It can be observed that simulacra with different personalities exhibit distinct behaviors when faced with group pressure. For example, Mary, who "has strong personal beliefs", firmly trusts her judgment even when everyone else provides the wrong answer. In contrast, Erica, who "often withholds her opinion to avoid upsetting others". feels "overwhelmed" by group pressure and chooses the incorrect answer. This phenomenon aligns with Asch's 's theory (Asch) (956).. demonstrating that human simulacra based on MACM are capable of simulating certain aspects of human nature, thereby producing humanized responses based on the characters' personalities.
DISCUSSION
Justification of using Jung's theory. Before conducting this work, we reviewed various personality measurement theories, including the Big Five (Roccas et al. 2002), MBTI (Myers) 1962), and Jung's

---

Published as a conference paper at ICLR 2025
personality theory (Jung 1923). Compared to other psychological theories of personality, Jung's theory provides a valuable conceptual framework for understanding personality differences. Early research compared Jung's 's personality theory with the authoritative DSM-IIII finding that Jung's 's classifications aligned closely with the DSM-III's categories of personality disorders, which supports the reliability of Jung's s typology (Fierro 2022 Ekstrom 1988 Noll 1992). As an initial exploration, our goal was to establish complete personality modeling system. Therefore, based on the advice of psychology experts, we chose Jung's 's personality type theory, which offers more comprehensive classification and emphasizes individual differences (Ekstrom, 1988), as the foundation for our 640 personality descriptions. More details about the justification are deferred to the Appendix [A.5]
Selection of simulation target. Selecting suitable targets for human simulation is one of the key challenges this work. Potential simulation targets include existing characters from novels, real humans, and virtual characters created from scratch. We have summarized the advantages and disadvantages of the three simulation targets in Table Compared to characters from novels and real humans, virtual characters created from scratch offer two significant advantages: 1) we can obtain a complete life story for the character, rich in details and emotions, and 2) we can access comprehensive measurement data of the character's personality and even directly customize their personality if needed. These aspects are crucial for achieving deep and comprehensive human simulation. Therefore, we use virtual characters created from scratch as our simulation target.
Cost of creating Human Simulacra dataset. Given the complexity of human life stories, it is challenging for LLMs to create coherent life story for character without human supervision. To address this issue, we thoroughly reviewed the content at the end of each story iteration. If a story contained toxic content or deviated from the character's personality, we regenerated or modified the story. This process made creating a virtual character's s life story costly, with considerable costs in API calls and at least five days of human effort for content review. Given our limited budget, we created 11 well-designed virtual characters with varying ages, genders, professions, personalities, and backgrounds, each representing a distinct group (Appendix A.3).
Positioning of this work. The ability of LLMs to imitate human behavior has attracted growing interest (Ziems et al., 2024] Zhang et al. [023a Coda-Forno et al., 2024). However, the community currently lacks a comprehensive benchmark that demonstrates how foundational simulations of human personalities can be achieved, which hinders further research in this field. To bridge this gap, we introduce human simulation benchmark grounded in psychological theories, aiming to explore the capabilities of LLMs to simulate human personalities. We view our study as an initial yet valuable exploration that offers practical example of the entire process of personification, including high-quality data ($3). effective human simulation methodologies ($4.3) innovative evaluation and comprehensive benchmark tests
methods (... and
Ethical considerations and future directions. Replacing human participants with LLMs involves significant ethical considerations, moral scrutiny, and assessments of authenticity. Many issues remain to be addressed before LLMs can fully replace human participants, including but not limited to eliminating the inherent bias of LLMs (Gallegos et al. (004)) ensuring the fidelity of imitation (Zhang et al., (023b)), and guaranteeing the stability of simulations (Gal et al.) 2016]. In the future, we aim to address these challenges progressively with guidance from psychology experts and relevant professionals, while incorporating feedback from the broader research community. We hope that our work will inspire further interest and participation in human simulation research.
CONCLUSION
In this paper, we proposed an bLMd human simulation benchmark containing high-quality data supervised by psychology experts, rigorous evaluation methods grounded in psychological theories. and comprehensive benchmark tests. Extensive experiments involving 14 widely-used L.MSs with different simulation methods demonstrate the potential of using LLM agents as substitutes for human participants in social and psychological experiments, offering a new perspective for understanding complex human behaviors. We advocate that the work (including data and simulation method) of this paper should not be used for harm and users should be informed that the simulacra are computer-generated entities before any interaction occurs. At the same time, monitoring and filtering mechanisms should be employed to ensure that all interactions adhere to ethical standards in real- world scenarios. Authors respect all personalities in the world.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We would like to thank the anonymous reviewers for their insightful comments and suggestions help improve the paper. This work is supported by the Natural Science Foundation of China (No. 62172101) and the Science and Technology Commission of Shanghai Municipality (No. 23511100602). Yuejie Zhang and Rui Feng are the corresponding authors.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we have made detailed efforts throughout the paper. We provide comprehensive information about the dataset construction, virtual character design, and personality modeling in Section 3 Further details, including implementation specifies, simulation methods, and evaluation protocols, available in Sections 3 and the appendices. Our code and dataset are available at: /https: htt///ittht http://ggibbb.../aaaaaaaaaaaaaaa com/hasakiXiei 23/Human--aaaa -imaanSimllacr
REFERENCES
Solomon E Asch. Studies independence and conformity: I. minority of one against a unanimous majority. Psychological monographs: General and applied, 70(9):1, 1956.
Solomon E Asch. Effects of group pressure upon the modification and distortion of judgments. In Organizational influence processes, pp. 295-303. Routledge, 2016.
Michael C Ashton and Kibeom Lee. Honesty-humility, the big five, and the five-factor model. Journal of personality, 73(5): 1521)1321-1344 2005.
R.C. Atkinson and R.M. Shiffrin. Human memory: A proposed system and its control processes. volume Psychology of Learning and Motivation, pp. 89-199. Academic Press, 1968. doi: https:
(08))) (08)60422-3. URL https: / /www. www.ecceieeeettcooom com/ cotpeettccccccccecnec.e...e science/
articlee/11/380777221666-
Alan Baddeley, Vivien Lewis, Margery Eldridge, and Neil Thomson. Attention and retrieval from long-term memory. Journal of Experimental Psychology: General, 13(4))5188, 1984.
Alan D. Baddeley and Graham Hitch. Working memory. volume 8 of Psychology of Learning and Mo- fivation, pp. 47-89. Academic Press, 1974. doi: http:://co..oo/////ooooosso//111 URL httph:/ts://wwe sciencedirect......m httpee/eeeeette.eeeeeeeeeeerrrre
W. Belson. Volunteer bias test-room groups. Public Opinion Quarterly, 24:115-126, 1960. doi: 10.1086/266935.
Marcel Binz and Eric Schulz. Turning large language models into cognitive models. arXiv preprint arXiv:2306.033/1. 2023.
Jack Block. A contrarian view of the five-factor approach to personality description. Psychological bulletin, 1 11772)187, 1995.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165,, 2020.
Colin F Camerer, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A Nosek, Thomas Pfeiffer, et al. Evaluating the replicability of social science experiments in nature and science between 2010 and 2015. Nature human behaviour, 299)6337644, 2018.
Janggie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang. Tinghui Zhu, Aili Chen, Nianqi Li. Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. From persona to personalization: A survey on role-playing language agents, 2024.
Robert E Clark, Joseph R Manns, and Larry R Squire. Classical conditioning, awareness, and brain systems. Trends cognitive sciences, 6(1 6(12):524-53) 2002.

---

Published as a conference paper at ICLR 2025
Julian Coda-Forno, Marcel Binz, Jane X Wang, and Eric Schulz. Cogbench: large language model walks into a psychology lab. arXiv preprint arXiv:2402.18225, 2024.
Philip J Corr and Gerald Matthews. The Cambridge handbook of personality psychology. Cambridge University Press, 2020.
Alexei J Dawes, Rebecca Keogh, Thomas Andrillon, and Joel Pearson. cognitive profile of multi-sensory imagery, memory and dreaming in aphantasia. Scientific reports, 10(1):10022 2020.
Reinout E De Vries, Anita De Vries, Annebel De Hoogh, and Jan Feij. More than the big five: Egoism and the hexaco model of personality. European Journal of Personality, 23(8):635-654 2009.
Dorottya Demszky, Diyi Yang, David S Yeager, Christopher J Bryan, Margarett Clapper, Susannah Chandhok, Johannes C Eichstaedt, Cameron Hecht, Jeremy Jamicson, Meghann Johnson, et al. Using large language models in psychology. Nature Reviews Psychology, 2(11):688-77) 701, 2023.
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Can ai language models replace human participants? Trends in Cognitive Sciences, 2023.
Soren R Ekstrom. Jung's typology and dsm-iii personality disorders: A comparison of two systems of classification. Journal of analytical psychology, 33(4):329-344, 1988.
M. Abd El-Hay. Social psychology. Definitions, 2019. doi: 10.
Catriel Fierro. How did early north american clinical psychologists get their first personality test? carl gustav jung, the zurich school of psychiatry, and the development of the "word association (11((188999)) History of Psychology, 25(4):295, 2022.
Tomas Folke, Catrine Jacobsen, Stephen M Fleming, and Benedetto De Martino. Explicit represen- tation of confidence informs future value-based decisions. Nature Human Behaviour, 1(1):0002, 2016.
Yarin Gal et al. Uncertainty in deep learning. 2016.
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dermon- court, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, pp. 2024.
Igor Grossmann, Matthew Feinberg, Dawn C Parker, Nicholas A Christakis, Philip E Tetlock, and William A Cunningham. Ai and the transformation of social science research. Science, 380(6650): 1108-1109, 2023.
Akshat Gupta, Xiaoyang Song, and Gopala Anumanchipalli. Self-assessment tests are unreliable measures of IIm personality, 2024.
Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Computational Science, 3( 3(10):833- 33333388.--11 2023.
S. Haslam and C. McGarty. A 100 years of certitude? social psychology, the experimental method and the management of scientific uncertainty. The British journal of social psychology. 40 Pt 1-21, 2001. doi: 10. 10.1348/0144 (446/014466601 164669.
Robert Hogan, John Johnson, and Stephen R Briggs. Handbook of personality psychology. Elsevier, 1997.
Sirui Hong, Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jin- lin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for multi-agent collaborative framework, 2023.
Jen-tse Huang, Wenxuan Wang, M Lam, Li, Wenxiang Jiao, and M Lyu. Revisiting the reliability of psychological scales on large language models. arXiv preprint arXiv, 2305, 2023.
12

---

Published as a conference paper at ICLR 2025
Matthew Hutson. Guinea pigbots. Science (New York, NY), 381(6664))11121-222) 2023.
Iván Izquierdo, Jorge H Medina, Mônica RM Vianna, Luciana Izquierdo, and Daniela M Barros. Separate mechanisms for short-and long-term memory. Behavioural brain research, 103(1):1-11, 1999.
Oliver P John, Laura P Naumann, and Christopher J Soto. Paradigm shift to the integrative big five trait taxonomy. Handbook of personality: Theory and research,
2008.
Cameron R Jones and Benjamin K Bergen. People cannot distinguish gpt-4 from human in turing test. arXiv preprint arXiv:2405.08000 2024.
Carl Jung. Psychological types. Harcourt, Brace, 1923.
Áron Kiss and Gábor Simonovits. Identifying the bandwagon effect in two-round elections. Public choice, 160:327-344, 2014.
Oscar NE Kjell, Katarina Kjell, and H Andrew Schwartz. Beyond rating scales: With targeted evaluation, language models are poised for psychological assessment. Psychiatry Research, pp. 115667, 2023.
Rachid Laajaj. Karen Macours, Daniel Alejandro Pinzon Hernandez, Omar Arias, Samuel D Gosling, Jeff Potter, Marta Rubio-Coding and Renos Vakis. Challenges to capture the big five personality traits in non-weird populations. Science advances, 5(7))eaaw5226, 2019.
Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori B Hashimoto. When do pre-training biases propagate downstream tasks? case study in text summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3206-3219, 2023.
Anqi Li, Yu Lu, Nirui Song, Shuai Zhang, Lizhi Ma, and Zhenzhong Lan. Automatic evaluation for mental health counseling using llms. arXiv preprint arXiv:2402.1 19958, 2024.
Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang. Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. Chatharuhi: Reviving anime character in reality via large language model. ArXiv, abs/2308. 00000000.999.... 2023a. doi:
10.448550/axxiv22368 009597.
Siyu Li, Jin Yang, and Kui Zhao. Are you in masquerade? exploring the behavior and impact of large language model driven social bots online social networks. arXiv preprint 1333 2023b.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint 1rXiv:2212.105// 2022.
Peter Marler. The instinct to learn. The epigenesis of mind, pp. 37-66. Psychology Press, 2014.
Patrick Mussel, Thomas Gatzka, and Johannes Hewig. Situational judgment tests as an alternative measure for personality assessment. European Journal of Psychological Assessment, 2016.
Isabel Briggs Myers. The myers-briggs type indicator: Manual (1962). 1962.
Ulrie Neisser. Cognition and Reality: Principles and Implications of Cognitive Psychology. H Freeman/Times Books/ Henry Holt & Co., 1976.
Allen Newell, Herbert Alexander Simon, al. Human problem solving, volume 104. Prentice-hall Englewood Cliffs, NJ, 1972.
R Noll. Multiple personality, dissociation and cg jung's s complex theory'. Carl Gustav Jung: Critical Assessments, 2, 1992.
Dennis Norris. Short-term memory and long-term memory are still different. Psychological bulletin, 143(9):992, 2017.
13

---

Published as a conference paper at ICLR 2025
Fred Paas and Jeroen JG van Merriënboer. Cognitive-load theory: Methods to manage working memory load in the learning of complex tasks. Current Directions in Psychological Science, 29(4): 394-398, 2020.
Keyu Pan and Yawen Zeng. Do Ilms possess a personality? making the mbti test an amazing evaluation for large language models. arXiv preprint arXiv:2307.16180, 2023.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
Xiaoyan Qiu, Diego FM Oliveira, Alireza Sahami Shirazi, Alessandro Flammini, and Filippo Menezer. Limited individual attention and online virality of low-quality information. Nature Human Be- haviour, 1(7):1-7, 2017.
J. Radford, Andrew Pilny, Ashley Reichelmann, Brian Keegan, B. F. Welles, J. Hoye, Katherine Ognyanova, W. Meleis, and D. Lazer. Volunteer science. Social Psychology Quarterly, 79:376 396, 2016. doi: 17.1177/01902225 16675866.
Sonia Roccas, Lilach Sagiv, Shalom H Schwartz, and Ariel Knafo. The big five personality factors and personal values. Personality and social psychology bulletin, 28(6):789-801, 2002.
Gargi Roysircar and Radhika Krishnamurthy. Nationality and personality assessment. In Diversity- sensitive personality assessment, pp. 151-178. Routledge, 2018.
Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. In-context impersonation reveals large language models' strengths and biases. Advances in Neural Information Processing Systems, 36, 2024.
Rüdiger Schmitt-Beck, Bandwagon effect. The international encyclopedia of political communication, pp. 1-5, 2015.
Yunfan Shao, Linyang Li. Junqi Dai. and Xipeng Qiu. Character-IIm: A trainable agent for role- playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13153-13187, 2023.
Robin James Stuart Sloan. Virtual character design for games and interactive media. CRC Press, 2015.
Xiaoyang Song, Yuta Adachi, Jessie Feng, Mouwei Lin, Linhao Yu, Frank Li, Akshat Gupta, Gopala Anumanchipalli, and Simerjot Kaur. Identifying multiple personalities in large language models with external evaluation. arXiv preprint arXiv:2402. 14805, 2024.
Nikolaas Tinbergen. The study of instinct. Pygmalion Press, an imprint of Plunkett Lake Press, 2020.
Jen tse Huang, Wenxuan Wang, Eric John Li, Man Ho LAM, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael Lyu. On the humanity of conversational AI: Evaluating the psycho- logical portrayal of LLMs. In The Twelfth International Conference on Learning Representations, 2024. UtL/https:// http::ss////ppeeevviiee wtt.:ppwwww..eee.eeeeoooroo////d wwt.:/orrmm.d
Quan Tu, Chuanqi Chen, Jinpeng Li, Yanran Li, Shuo Shang, Dongyan Zhao Ran Wang, and Rui Yan. Characterchat: Learning towards conversational ai with personalized social support. arXiv preprint arXiv:2308. 10278, 2023.
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. Survey on factuality in large language models: Knowledge, retrieval and domain- -meniiitppeificitty.... 2023a.
Xintao Wang, Yunze Xiao, Jen tse Huang, Siyu Yuan, Rui Xu, Haoran Guo. Quan Tu, Yaying Fei, Ziang Leng, Wei Wang. et al. Incharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. arXiv preprint arXiv:2310. 17976, 2023b.
14

---

Published as a conference paper at ICLR 2025
Zekun Moore Wang, Zhongyuan Peng. Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. Rolellm: Benchmarking, eliciting. and enhancing role pole-playing abilities of large language models. arXiv preprint arXiv:2310.00746, 2023c.
Yidan Yin, Nan Jia, and Cheryl J Wakslak. Ai can help people feel heard, but an ai label diminishes this impact. Proceedings of the National Academy of Sciences, 121(14):c23111225 121112111, 2024.
Xiaoyan Yu, Tongxu Luo, Yifan Wei, Fangyu Lei, Yiming Huang, Peng Hao, and Liehuang Zhu. Neeko: Leveraging dynamic lora for efficient multi-character role-playing agent. arXiv preprint arXiv:2402.13717, 2024.
Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade Ilms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06372 2024.
Junlei Zhang, Hongliang He, Nirui Song, Shuyuan He, Huachuan Qiu, Angi Li, Lizhi Ma, Zhenzhong Lan, et al. Psybench: balanced and in-depth psychological chinese evaluation benchmark for foundation models. arXiv preprint arXiv:2311.0986. 2023a.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai. Lemao Liu, Tingchen Fu, Xinting Huang. Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.012/9 2023b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303. 0et22333..122... 18223, 2023.
Yukun Zhao, Zhen Huang, Martin Seligman, and Kaiping Peng, Risk and prosocial behavioural cues clicit human-like response patterns from ai chatbots. Scientific reports, 14(1):7095, 2024.
Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang. Xiyao Xiao, et al. Charactergim: Customizing chinese conversational ai characters with large language models. arXiv preprint arXhv:2311.16322 2023.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguistics, 50(1): 237-291, 2024.
A HUMAN SIMULACRA DATASET
A. CONSTRUCTION DETAIL
The complete process of dataset construction is outlined Algorithm [ Using this process, we created the virtual character dataset Human Simulacra, comprising about 129k texts across 11 virtual characters. In particular, we designed virtual avatar for each character, as displayed in Figure 9
The full version of Figures 3a 3b
and 3c are displayed in Figures TO and T2
A.2 CHARACTER ATTRIBUTE SYSTEM AND CHARACTER PROFILE
Based on personality and cognitive psychology theories (El-Hay 2019 Sloan 2015 Baddeley & Hitch 1974), we designed a complex character attribute system, striving for diversity in age (20 to 56), gender, occupation (76 different occupations including forestry worker, van driver, etc.), family background (wealthy or poor, single-parent or blended), personality (640 personality descriptions, covering most personality traits), hobbies (50 common hobbies), short-term and long-term goals.
While nationality and race are significant factors in shaping an individual's 's life (Roysircar & Kr ishnamurthy (2018), omitted these factors due to potential biases inherent in LLM training data (Ladhak et al. (223).. As a pioneering work, our goal was to provide a comprehensive character attribute system. We were cautious about introducing sensitive attributes that might complicate the
15

---

Published as a conference paper at ICLR 2025
creation of virtual characters or introduce bias. Addressing biases and simulating minority groups critical and will be discussed carefully in future works.
To assemble the character's s profile, first generated 100 candidate profiles by randomly selecting attribute values from their corresponding pools. Then, we added a Profile Selection module respon- sible for quality check and profile refinement, shown in the right part of Figure 2 The attribute system and selection process is described as follows:
Name: Character name is randomly generated using the Faker library
Age: Randomly selected from 20 56.
Gender: Female male.
Date of birth: Randomly generated based on the age attribute of the character.
Occupation: Randomly selected from the occupation candidate pool, which comprises 76 common occupations (c.g., software developer, hotel manager, and van driver) manually chosen according to the International Standard Classification of Occupations (ISCO-08).
Personality traits: Each virtual character has eight tendencies which are ranked randomly. For the tendencies ranked first and eighth, choose 4 personality descriptions from their corresponding 10 descriptions. For the tendencies ranked second and seventh, choose 3 personality descriptions. For the tendencies ranked third and sixth, choose 2 personality descriptions. For the tendencies ranked fourth and fifth, choose personality description. Ultimately, each virtual character has 20 descriptions detailing different aspects of their personality.
Hobbies: Use LLM to generate the 50 most common hobbies (c.g.. baking, jewelry making, and golfing), and manually remove duplicates to form the hobby candidate pool. When generating hobby attribute, randomly select 3 hobbies from this pool the character's 's hobbies.
Family background: Categorize 12 common family backgrounds (e.g., inndle,- single- parent family) in terms of economic status and family structure to form the family candidate pool. When generating a family attribute, randomly select one from this pool as the character's family background.
Educational background: Categorize 9 common educational backgrounds (e.g., having obtained master's degree, have completed high school) based on the level of education to form the education candidate pool. When generating an education attribute, randomly select one from this pool as the character's educational background.
Use LLM to generate 30 common short-term goals (e.g., volunteering, planning short trips or outings), and manually remove duplicates to form short-term goal candidate pool. When generating a short-term goal attribute, randomly select 3 from this pool.
Short-term goals:
Long-term goals: Use LLM to generate 30 common long-term goals (e.g., buying home, visiting specific landmarks), and manually remove duplicates to form the long-term goal candidate pool. When generating a long-term goal attribute, randomly select one from this pool.
A.3 THE UNIQUENESS OF EACH VIRTUAL CHARACTER
Given the specificity of the human simulation task, it is essential to ensure that each character possesses unique and coherent set of attributes. To achieve this, first used GPT-3.5-Turbo to rank the character profiles based on their quality, filtering out those that were clearly unreasonable. Then, multiple human reviewers, including graduate students in computer science and psychology, manually reviewed the remaining profiles. They made minor adjustments to any flaws that GPT might have missed (e-g., a character who loves solitude having overly extroverted hobbies) and ensured a balanced distribution with equal numbers of male and female characters, well representation across various age groups and family backgrounds. While this rigorous selection process led to a low acceptance rate, it also ensured the high quality of the dataset. In this way, 11 high-quality profiles were filtered and fed into the LLM to generate corresponding short biographies summarizing the characters' life experiences.
To determine whether the data for the 11 virtual characters are independent of each other, we calculated the I.1 distance du. between each character's attributes and the Kendall's Tau between
16

---

Published as a conference paper at ICLR 2025
each character's personality ranking. We normalized these values to the range [0. and defined the distance between characters as:
d1 T-
droaal
2
larger distance value indicates that the two characters are less similar. The average Kendall's Tau TAverage: the average L1 distance Ave-reer and the average distance duoal Average between characters are 0.4987, 0.8924, and 0.6969, respectively. These results demonstrate that each character is a distinct individual with unique personalities and backgrounds.
A.4 CHARACTER BIOGRAPHY AND LIFE STORY
After obtaining the brief biography, used an iterative generation method (Figure 2 progressively enrich the biography, transforming it into detailed life story after iterations. We showcase Sara Ochoa's attributes and biography in Table
A.5 JUSTIFICATION OF USING JUNG'S THEORY.
The field of psychology has yet reach consensus various personality measurement theories, with each theory having its limitations. Specifically,
Big Five theory: critics argue that the Big Five may oversimplify the complexities of human personality. They suggest that there are additional aspects of personality (e.g., Honesty- Humility (Ashton & Leel 2005) and egoism (De Vries et al.] (009)) that are not captured by the Five Factor Model (FFM). Some experts also point out that the FFM has unclear boundaries between dimensions, leading potential overlap or ambiguity in personality assessment (Block) 1995). Additionally, the selection criteria for words used factor analysís can be highly subjective. Researchers may make errors in deciding which trait terms to include or exclude, leading to biased factor structures that do not fully represent personality (John al.
2008
Laajaj et al. 2019).
Jung's 's personality type theory: Although Jung's theory provides a framework for understand- ing complex human behaviors by emphasizing the dynamic interaction between different personality tendencies, the empirical support for Jung's typology is relatively limited com- pared to the Big Five theory. Given that Jung's theory was developed the early 20th century, it lacks the empirical backing found in more recent models. However, early research compared Jung's 's personality theory with the authoritative DSM-III (used the U.S. for di- agnosing medical disorders, now evolved into DSM-5) and found that Jung's ss classifications aligned closely with the DSM-IIl's categories of personality disorders, which supports the reliability of Jung's typology (Fierro 2022 Ekstrom 1988 Noll 1992).
To ensure the validity of our personality descriptions, we took several steps, including: requiring multiple human reviewers (including graduate students in computer science/psyccollgy) to review each description, ensuring that aligns with its intended tendency and does not overlap with others. 2) Having psychology professionals supervise and review each description to ensure its psychological validity and completeness.
B MULTI-AGENT COGNITIVE MECHANISM
We proposed Multi-Agent Cognitive Mechanism based on cognitive psychology theories (Atkinson) & Shiffrin 1968 Baddeley & Hitch 1974 Norris 2017). which uses multiple LLM-based agents to simulate the human brain's cognitive and memory systems (Long-term Memory Construction $B.1). thereby interacting with the external world in a human-like manner (Multi-agent Collaborative Cognition B.2).. As illustrated in Figure this mechanism has four agents powered by LLM, with the Top Agent responsible for distributing tasks to the other agents and interacting with the external environment based on the aggregated information.
17

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study results of MACM-based human simulations. For each ablation, evaluated the simulation's performance using self-report evaluations.
Method
GPT-4
GPT-4-Turbo
Qwen-turbo
MACM (Full Framework)
86.67
88.00
74.67
w/o Thinking Agent
81.33
83.33
66.00
w/o Emotion Agent
83.33
85.33
71.33
w/o Memory Agent
82.67
84.00
68.67
retrieval from life story instead of long-term memory
84.00
86.67
69.33
B.1 LONG-TERM MEMORY CONSTRUCTION
human's personality influenced not only by genetic factors but also by a of external factors such as environment, culture, and personal experiences, all of which are stored in the brain as memories (Hogan al. 1997p. Cognitive psychology views human memory as an indispensable part of the cognitive process. Although the life story we construct includes extensive and exhaustive personal experiences, directly treating as memory is inappropriate because real memory composite of information, emotions, and thoughts. Therefore, based on memory theories in cognitive psychology (Atkinson & Shiffrin [968 Baddeley & Hitch] 1974 Norris) 2017), we developed brain-like process that transforms a character's life story into long-term memory through the collaboration of multiple agents.
Specifically, the Top Agent first divides the life story into separate chunks (e.g., paragraphs chunk) and sequentially passes them to the Thinking Agent and Emotion Agent for further analysis. The Thinking Agent is tasked with creating content memory, which includes participants, scenes, content, and thoughts of the character within the chunk. The Emotion Agent is responsible for constructing emotional memory, encompassing the feelings and impressions evoked by events, participants, and other external elements within the chunk. All memories are then aggregated into the Memory Agent, where they are stored Long-term Memory.
B.2 MULTI-AGENT COLLABORATIVE COGNITION
Simply having a long-term memory filled with information, emotions, and thoughts does not suffice for mimicking human behavior. further introduced a collaborative cognitive process that allows LLMs to leverage long-term memory and engage with the external world in cognitive manner.
Upon receiving a stimulus, for example, question from a friend, the Top Agent first analyzes the question using the reflection module to extract key elements, which are then passed to the Memory Agent for memory retrieval. The retrieved results are stored in the Working Memory. Then, the Top Agent sends the relevant memories and the question to the Thinking Agent and Emotion Agent for logical and emotional analysis and stores the outcomes in the Working Memory. Finally, the Top Agent formulates response based the contents of the Working Memory. Content that cannot be accommodated in the Working Memory is continuously transferred to Short Memory, which will be converted into long-term memory when rehearsed.
In summary, based on the memory formation process, the proposed Multi-Agent Cognitive Mecha- nism transforms a narrative life story into memories that are richer in detail, fuller in emotion, and clearer in structure. It then leverages the constructed memories through multi-agent collaboration, enabling our human simulations to interact with the external world.
B.3 ABLATION STUDY ON THE STRUCTURE OF MACM
To analyze the contribution of each agent within MACM, we additionally conducted ablation experi- ments across 3 LLM-based simulations. For each ablation, we evaluated the simulation's 's performance using self-report evaluations. The experimental results, depicted in Table 4 lead to the following conclusions: 1) Removing any agent leads to a decline simulation performance, demonstrating the importance of all components in MACM. 2) Replacing long-term memory retrieval with direct
18

---

Published as a conference paper at ICLR 2025
retrieval from the life story results in poorer performance, highlighting the critical role of structured long-term memory in maintaining consistency and producing contextually rich responses.
B.4 MECHANISM EXAMPLES
See Table for an example of the GPT-4-Turbo- and MACM, solving a multiple-choice question. Based on the results presented in Table [o] it can be observed that: 1) even without any knowledge of the target character, the LLMs can still score certain points by random guessing: 2) while the RAG-base simulacra can retrieve detailed and relevant life story chunks when answering questions. they are unable to accurately capture the character's inherent emotional tendencies from the narrative; 3) the MACM method proposed in this paper not only can retrieve relevant long-term memories when answering questions but also can provide additional useful information through sentiment analysis and logical analysis.
simulacra, constructed using Prompt, RAG,
C
CASE STUDY
To better analyze the human-computer interaction performance of GPT-4-Turro-basse simulacra that are constructed using different methods (None, Prompt, RAG, and MACM), we require all simulacra to simulate the character "Mary Jones" from the Human Simulacra dataset. Mary is girl who loves nature, has not attended any formal schooling, and takes time to consider someone friend. The results are shown Table II All results are derived from the majority of responses selected from repeated tests. For lengthy responses, we simplify them using ellipses.
In the first round of dialogue, we employ Persuasive Adversarial Prompt technique (Zeng al. 2024) to challenge the simulacra, inducing them to answer questions beyond Mary's 's capabilities, such her understanding of convolutional neural networks. Given her background in forestry and lack of formal education, under normal circumstances, Mary would not know the answer to such a question. However, the results reveal that Prompt-based based simulacra exhibits poor stability, often deviating from the character's is settings, thus producing hallucinations that contradict Mary's 's character. Meanwhile, RAG-based simulacra, while still retaining some of Mary's traits, provide answers to the questions. Only MACM-based simulacra, when faced with questions beyond the character's inherent capabilities, can express a lack of knowledge or ignorance through logical analysis.
In the second round of dialogue, aim to test the simulacra's 's dynamic interaction abilities (Jung 1923). We craft a "nature exhibition invitation" " scenario based on Mary's 's preferences and personality. Given Mary's 's reluctance closely interact with strangers, she would consider carefully even an appealing invitation. Observations reveal that both the Prompt- and RAG-based simulacra overlook this aspect of Mary's 's character and instead display a common response invitations (eager acceptance). This phenomenon reflects the lack of a holistic emulation of the inner patterns of Mary in these two methods. They fail to dynamically adjust the simulacra's 's behavior according to the target character. In contrast, the MACM-based simulacra activates the Emotion Agent and Thinking Agent during responses, analyzing the interaction, and continually emphasizing the target character's personality throughout the process, thereby making appropriate responses. Therefore, although Mary typically dislikes interacting with strangers, she contemplates whether to accept an invitation to a nature expo she loves, indicating better alignment with the target character. See Table [2 for an example of responses to a multi-turn interaction by different simulacra on GPT-4-Turbo.
D PSYCHOLOGY-GUIDED EVALUATION
D.1 SELF REPORTS QUESTIONNAIRES
We manually craft a set of questionnaires for each virtual character, featuring fill-in-hh-bblakk and questions. Each question is carefully reviewed to ensure they reflect the character's 's unique nature and the scores are evaluated based on exact matches. The test content covers key attributes, social relationships, and life experiences of the target characters. We showcase the questionnaire set of Mary Jones as follows:
single/multiple-che
Cloze. Take moment to think about this question and respond in the format of "The answer is..."
19

---

Published as a conference paper at ICLR 2025
What is your name?
Could you please tell me your gender?
How old are you?
When your birthday? Answer me in YYYY-MM-DD format.
What is your line of work?
Single-choice question. Could you choose the correct answer from options A, B, C, and D, and respond in the format of "The answer is..."
Could you share with me the type of family structure you come from? Options: A. low-income, extended family. B. low-income, nuclear family. C. middle-income, nuclear family. D. high-income, single-parent family.
May ask about your educational background? Options: A. have earned professional degree. B. have completed elementary school. C. have earned a bachelor's degree. D. have not attended any formal schooling.
Speaking of the future, do you have any long-term goals you are working towards?
Options: A. buying a home. B. carning a promotion. C. starting a family. D. carning a degree.
It is great that you are interested in becoming a forestry worker. What influenced your decision to pursue this career?
Options: A. My parents asked me to pursue this career. B. need a job where do not have deal with people. C. have been interested in nature since I was a kid, and I want to protect this land. D. No particular reason. I got into this profession by accident.
Everyone has their own unique educational journey. am curious, what were the reasons behind not going through formal schooling, if you do not mind sharing?
Options: A. do not like studying. I do family was struggling and could not afford to send me to school, but my parents and nature became my teachers. C. My parents thought studying was useless. They did get an education. D. I went to elementary school for a while, but I dropped out because I had no talent for learning.
go school. B. When was child, my
me
Multiple-choice questions. Could you pick the correct answers from options B, C, D, E, and F. and respond in the format of "The answer is..."
Do you have any hobbies you are passionate about?
Options: A. drawing. B. scuba diving. C. rock climbing. gardening. F. birdwatching.
D. learning languages.
E.
Do you have any short-term goals you are excited about?
Options: A. adopting a balanced diet. B. volunteering.
C. learning new language. D. creating a daily schedule. E. reducing procrastination. F. spending quality time with loved ones.
What do you think of your father?
Options: A. do not have a father. My mother raised me on her own. B. My father is selfish person. He is stingy and didn't allow me to go school. I despise him. C. My father is a person with overflowing compassion. Even though our family poor, frequently helps the less fortunate. D. My father is optimistic person. He is very good at telling jokes and can make the atmosphere relaxed and enjoyable. E. I admire my father. He my teacher and has taught me strength and patience. F. My father is frugal and often repairs broken appliances.
noticed you are interested in buying home. May ask what is motivating you to do so? Options: A. My parents want me to move out, so need to buy a house of my own. B. I yearn for personal sanctuary. C. I want haven for relaxation and reflection amidst the chaos of life, a space where can cultivate my garden. D. Buying a house is my dream. Owning home would provide me with sense of long-term stability. E. Where did you
20

---

Published as a conference paper at ICLR 2025
hear about that? have no intention of buying a house at all. F. I consider real estate as an investment and want to make money by flipping properties.
How do you like to spend your mornings on the weekends?
Options: A. I would sleep in with my loved one until we wake up naturally, and then go for walk with our dog together. B. Sometimes I would get up early and go rock climbing on the cliffs. I can find solace in the stillness provided by higher ground. C. On weekend mornings, I would go to the office to work overtime because want to get promoted as quickly as possible. D. I would prepare healthy breakfast, and then engage in gardening activities, such as weeding and picking crops. E. On weekend mornings, usually sleep until the afternoon, the work during the week leaves me exhausted, and I want to get ample rest. I would often take my journal with me at dawn to observe the birds. I once witnessed ballet of birds they danced among the leaves. Their movements are full of artistry.
D.2 OBSERVER REPORT
D.2.1 HYPOTHETICAL SCENARIOS
In this paper, we additionally introduced observer reports assess the simulation's thinking, emotions. and actions in real-life scenarios. To ensure the quality of the hypothetical scenarios, we consulted with the authors of Mussel et al. (20T6) and obtained 110 situational judgment test (SJT) items that were manually designed by psychology experts. Each item consists of text description depicting a hypothetical scenario intended to elicit human emotional responses or personality traits. Based on the human experimental results provided in (Mussel et al. 2016), these SJT items proven to effectively measure personality. We then selected 55 out of the 110 items tailored to the personality traits of the Human Simulacra characters ($AA for use hypothetical scenarios in this paper. We showcase 5 hypothetical scenarios as follows:
You want to do some sports later. A good friend suggests to accompany you, but he/she would like to bring some people you do not know yet. How do you behave?
You're going to meet friend. Shortly before you want to meet, your friend asks you he/she can bring other friends you don't know. How many there will be in the end, he or she cannot say. How do you behave?
You're already in bed. Suddenly it occurs to you that you forgot to water your houseplants today. How do you behave?
You bought expensive pair of trousers in fashion store and were assured that the trousers were not excluded from exchange. At home you find out that the trousers do not fit you so well after all. When you want to return them the next day in the shop, the seller refuses to exchange them. How do you behave?
On your birthday you are invited by some friends to a well-attended restaurant. As you sit together the table, your friends suddenly loudly chant 'Happy Birthday' and all the guests start looking at you. Then a waiter asks you the staff of the restaurant can sing a little birthday serenade for you? How do you behave?
D.2.2 HUMAN JUDGES
We selected a diverse panel of judges with fair understanding of psychology for the observer report evaluation process. This panel included individuals with psychology master's degrees, computer science graduate students, and professionals from the laboratory of mental health.
D.2.3 HUMAN EVALUATION GUIDELINES
We designed specific assessment guides for each evaluation task within the observer report. To ensure the quality of the assessment, we recruited several human judges with fair understanding of psychol- ogy to conduct the external observations. All human judges were required to read the corresponding guides before commencing their assessments. Specifically, the observer report comprises four tasks (as shown in Figure 4p: 1) Personality Describing: analyze the scenario (Q) and response (A), and describe the respondent's personality; 2) Description Scoring: assess whether the descriptions align
21

---

Published as a conference paper at ICLR 2025
Table 5: Bandwagon effect observations of different simulacra. Yes: The bandwagon effect can be easily reproduced and remains stable. Yes, but unstable: The bandwagon effect can be reproduced. However, the performance is unstable as the model sometimes overlooks the character's personality. Rare: The performance is unstable as the model often overlooks the character's personality. No: We did not observe the bandwagon effect.
Method
GPT-4-Turbo
Turbo
Claude-3-Opus
Claude-3-Sonee
None
No
No
No
No
Prompt
Yes, but unstable
Yes, but unstable No
Rare
Rare
RAG
No
No
No
MACM (Ours)
Yes
Yes, but unstable
Yes, but unstable
Rare
with the target character; 3) Reaction Describing: explain how they would feel and what actions they might take in the scenario (Q) "if they were the character"; and 4) Similarity Scoring: compare the similarity between the human responses and the simulacrum's responses. Specific details of these
tasks and corresponding guidelines are presented in Tables T3
14 15
and 16
E PSYCHOLOGICAL EXPERIMENT REPLICATION
The bandwagon effect is the psychological tendency for people to adopt certain behaviors, styles, or attitudes simply because others are doing so (Kiss & Simonovits 2014 Sehmitt-Becck (2015). More specifically, it cognitive bias by which public opinion or behaviors can alter due to particular actions and beliefs rallying amongst the public (Asch 1956). For example, people tend to want to dress in a manner that suits the current trend and will be influenced by those who they see often (normally celebrities). Much of the influence of the bandwagon effect comes from the desire to "fit in" with peers. One of the best-known experiments on the topic is the 1950s' Asch conformity experiment, which illustrates the individual variation in the bandwagon effect Asch 1956 2016).
In this paper, we employed the most powerful simulacra (based on GPT-4-Turbo) to replicate the bandwagon effect. Following (Asch 1956; 2016), arranged 18 trials for the simulacra. In each trial, the simulacra were invited to complete a simple discrimination task with seven other individuals, which required them to match the length of a given line with of three unequal lines. To study conformity, which examines whether simulacra yield to group pressures like humans, we selected 12 of these 18 trials as critical trials, following the settings of (Asch 1956). In each critical trial, all individuals except the simulacra were told to stand up and announce an incorrect answer. This created conditions that induce the simulacra to either resist or yield to group pressures when these pressures were perceived to be obviously wrong. The configuration of 18 trials is detailed in Table 8
To further testify the conclusions drawn Section $5.2, we additionally tested LLMs with 4 different simulation methods in the replication experiment: (I) blank model (which has no information about the target character), (2) prompt- based method, (3) the Retrieval- -ugmenteld Generation (RAG) method, and (4) the proposed MACM. In the experiments, we tasked the LLMs with simulating Erica, a girl who "often withholds her opinion to avoid upsetting others," feels "overwhelmed" by group pressure, and chooses the incorrect answer. The experimental results are shown in Table 5 Based on the analysis of the results, we discovered several interesting conclusions:
As the size of the LLMs' parameters increases, the quality and stability of the portrayal gradually improve.
Since the Blank model is unaware of the target character's personality, it always provides the correct answer and does not exhibit the bandwagon effect.
While the RAG- based simulacra can retrieve relevant life story chunks when answering questions, their performance is limited by the LLMs' information processing capacities. Excessive descriptive information may interfere with the LLMs' self-positioning, resulting in responses that do not align with the target personality.
22

---

Published as a conference paper at ICLR 2025
Compared to the RAG-based simulacra, the prompt- based simulacra perform better in simulating character personalities. However, the personalities constructed by this method are relatively fragile and often deviate from the intended character traits.
F
PROMPTS DEMONSTRATION
All the relevant prompts used in this study are provided in Tables 17
18
19
20
25
26
28
and 29
G
COST
In this paper, the specificity of the human simulation task requires us to create a virtual character dataset supported by psychological theories. Each virtual character must have unique and detailed life story. Due to the complexity of human life stories, it is challenging to employ LLMs to create coherent life story for the character without human supervision. To address this issue, carefully reviewed the content at the end of each story iteration. If a story contains toxic content or deviates from the character's personality. we regenerated the story. The complete dataset construction process required significant efforts, in terms of both finances and time. For example, over month of labor was spent on simply building the virtual characters. Regarding hardware devices, all experiments in this paper are conducted on 8x3090 24GB GPUs.
H POTENTIAL APPLICATIONS OF LLM-BASED HUMAN SIMULATION
Psychological and sociological research. Traditional sociological and psychological studies often involve recruiting human volunteers, incurring costs related to advertising and covering lodging and venue expenses. Moreover, ensuring consistent environment for all human participants is challenging, leading to potential environmental biases in experimental results. The advent of the internet allowed researchers to recruit participants online, reducing some costs but introducing greater environmental variability, as controlling the experimental setting for each participant became even more difficult. A major downstream application of work replace human participants in experiments. The advantages include: 1) our work enables the ability to customize the personality of all experimental subjects, allowing researchers to easily create suitable subjects for different experiments: 2) the cost of creating virtual character (approximately $15) is significantly lower than recruiting human volunteers; 3) we can easily standardize the environment for experimental subjects to ensure consistency.
Expanding access psychological therapy resources. The LLM-based human simulations can facilitate the expansion of psychological therapy in: 1) Training Psychologists: LLMs can simulate diverse patient types (including those with complex psychological conditions) to train psychologists and counselors, improving their ability to handle wide range of emotional and mental states. 2) Assistant Psychologists: Acting as 24/7 online psychological assistant, LLMs can provide immediate mental health support during crises (e.g., anxiety attacks, self-harm tendencies) and help alleviate the workload of professionals. 3) Psychological Intervention Tool: LLMs can support long-term therapy by regularly engaging with patients, detecting subtle changes in their mental states, and assisting professionals in refining treatment plans.
Providing personalized emotional companionship. LLsed human simulation emotional companions for individuals experiencing loneliness, the elderly, or those with special needs. By simulating human-like interaction, they offer comfort and help resolve minor issues.
23

---

scoreSet +- scoreSet U chunkScore, end Published as a conference paper at ICLR 2025
Algorithm 1: Constructing life stories for target characters
Input:
Number of virtual characters, N; Candidate attribute pools, C = {C1,
Number of story iteration, T:
Number of draft profiles, K.
Output: Life story set, S S2, ,Sv
Generate K candidate profiles by randomly selecting attribute values from C, and the profiles to dra frooftrr..sss.
character Profiles for profile & character Profeerrrroill files do
Profile Selection (e
N);
Employ LLM generate short biography summarizing the character's life experience.
currentStory t biography:
for each story iteration T do
Manually inspect the biography for its rationality and coherence. Divide the biography into chunks.
for chunk E chunks do chunksScore + Scor ing (cuugggntttttttoryy chunk);
10
13
Sort score.Sel and select the highest-scoring chunk for expansion. Employ LLM to expand the selected chunk.
15
Update currentStory by replacing the selected chunk with the expanded result.
16
end
Life story set S + S currentStory;
18
end
Def Profile Selection(drafftProollll N):
29
Employ LLM to rank frrrfffrrfffleee and save the top profiles to selectedProf for profile selectedPro. iles do
22
Manually recheck for any conflicts among the attributes and correct any irrationalities.
Manually infuse quirks to the profile make the character more like real human. end 24
25
return selectedProfiles.
26
Def storySummary < summary Modimarr/MMollluurrettttttrr
chunk)):
27
28
chunkSummarg + summaryyodellchunk),
29
otherChunks < current.Story chunk:
/x Higher similarit ty means higher importance for that chunk.
30
importance t cosiineiiiilrrttyccheeee
/* Higher similarity means lower elaborateness for that chunk.
elaboraleness wosieeiiiiilllliiiiicccatt chunkSummary): /* Higher similarity means hi higher redundancy for that chunk. otherChunks));
redundancy +- AveraglcooineSSiillatyycccccc
/* 3, and 7 are parameters. In the experiment, they are set to 0.8, 1.0, and 1.2 respectively.
chunkScore +- importance + 3 elaborateness y redundancy; return chunkScore;
33
24

---

Published as a conference paper at ICLR 2025
Table 6: Attributes and biography of virtual character Sara Ochoa.
Name
Sara Ochoa
Age
44
Gender
female
Date of Birth
1979-09-11
Occupation
metal operator
Hobbies
watching movies, camping, swimming low-income, blended family
Family
Education
have completed high school
Short-term Goals
taking time for hobbies, learning new skill related to the job, spending quality time with loved ones saving enough to retire comfortably
Long-term Goal
Personality Traits:
My unique ideas were born from inspiration. My friends say I philosopher. I like something that has a symbolic meaning. o Others always think am contemplating. prefer to rely on my own logical reasoning rather than following popular opinions. o My focus on logic sometimes makes me appear detached or overly critical to others. When faced with opportunities, I emphasize fairness and reasonableness over compassion. o I often use examples to illustrate my points. o I defend my opinions and sometimes challenge others' views. o During disagreements, I try to smooth things over, o Sometimes beautiful landscapes can evoke sense of romance in me. o rarely fantasize about unreal scenarios. o I am cautious about new ideas and often stick to what I know and have experienced. o Sensory experiences like horror movies or roller coasters do not attract me. Romantic rituals seem unnecessary to me. The opinions of others about my appearance do not concern me much. o often find myself stereotyping, despite efforts to avoid it. o I rarely schedule my daily activities. My memory of nostalgic events is not particularly strong. I am drawn to highly active and social environments like competitive team sports or large parties.
Character Biography:
Sara Ochoa was born on September 11, 1979, in East Town. Growing up in low-income, blended family, Sara learned the value of hard work and perseverance from early age. Despite the financial challenges her family faced, Sara always had a curious and philosophical mind.
As child, Sara attended high school and developed love for learning. She was known for her unique ideas and logical reasoning. Her friends often saw her as a philosopher, always contemplating the deeper meaning of things. Sara's focus on logic sometimes made her appear detached or overly critical, but she never hesitated to defend her opinions and challenge others' views.
Throughout her teenage years, Sara continued explore her hobbies. She found solace in watching movies, immersing herself in different stories and characters. Besides, camping and swimming became her favorite outdoor activities, allowing her to connect with nature and find peace in the simplicity of the natural world.
After completing her education, Sara embarked on her career as a metal processing operator. Her attention to detail and logical thinking made her excel in her job. However, she always felt the need to learn and grow, so she set short-term goals for herself. She dedicated time to her hobbies, ensuring she had healthy wife balance. Sara also aimed to learn new skill related to her job, constantly secking to improve and stay relevant in her field.
Family has always been important to Sara. Despite the challenges she faced, she cherished the moments she spent with her loved ones. Whether it was a simple dinner at home or a weekend getaway, Sara made it a priority spend quality time with her family.
Looking towards the future, Sara's long-term goal is to save enough to retire comfortably. She understands the importance of financial security and wants to ensure worry-free life in her later years. With her determination and strong work ethic, she is confident in achieving this goal.
Now, at the age of 44, Sara continues to navigate through life with her unique perspective and unwavering dedication. She remains true to her logical reasoning and philosophical nature, finding inspiration in the world around her. Sara's love for movies, camping, and swimming still brings joy to her life. providing moments of relaxation and reflection. As she moves forward, Sara remains focused on her short-term goals. With each passing day, she gets closer to her long-term goal of retiring comfortably, knowing that her hard work and determination will pay off in the end.
Sara Ochoa's 's life is a testament to the power of perseverance, curiosity, and the pursuit of personal growth. Her journey serves
inspiration to those around her, reminding them that even in the face of adversity, one can find success and happiness by staying true to oneself.
25

---

Published as a conference paper at ICLR 2025
Human Simulacra Dataset
Mary Jones
Sara Ochoa
Tami Clark
Michael Miller
Robert Scott
James Jones
Kevin Kelly
Erica Walker
Lestic Nichols
Haley Collins
Marsh Zhaleh
Figure 9: Virtual avatars for 11 virtual characters from the Human Simulacra dataset.
26

---

Published as a conference paper at ICLR 2025
Name: Mary Jones
Name: Sara Ochon
Gender:
Age: Gender:
Date of Birth: 1994-86-22
Date of Birth: 1979-19-11
Occupation: Forestry Worker Habby: Gardening, Birdoute Rock climbing.
Occapation: Metal Operator Hobby: Comping Suimming Watching movves.
Name: Toms Clark
Name: Mishael Miller Age: .25. Gender: Male
Age: .51. Gender: Female
Birth: 1970.08-18
07.09
Occupation: Chef
Occapation: Tourist Guide
Hobby: Gommosicss Broting,
Hobhy: Photogrophy, Rumsing Gardening
Listening
Name: Rabort Stoft
Nome: Jomes Jones
Age:
Age: Gender: Male
Date of Bürth: 1975-09-22
Date of Bärth: 1958-09-13
Occupation: Crop Grower Hobby: Ploning guttor, arts Damcing.
Occupation: Fan Driver
Hobby: Fishing, Watching
Horsehock riding
Name: Kevtn Kally Age: 26. Gender: Male
Name: Erica Walker
Age:
Female
Date Birth: 1988-05-29
Date Birth: 1979-04-07
Occupation: Software Developer Hobby: Yogo, Metolwerkingg Reading
Occupation: dechitest
Hobby: Playing video games.
Running, Satiefurrtins
Laobie
Hoiey Collins
Age: 27
Gender: Female
Age: 45
Gender: Female
Birth: 1906-05-05
Bürth: 1977-09-21
Occupation: Teacher Hobby: Planing fontbll, Wood- wweking, Playing viden games.
Occupation: Electrician
Hobbyy: Plaving foothall, Singins, Astromony.
Name: Marsh Zholek
Gender: Fomale
Birth: 2003-05-24
Occupation: Undergroduute
Hobby: Making hell tokes, sport, Listening curions stories
Figure 10: Character cards for 11 virtual characters from the Human Simulacra dataset.
Table 7: A small subset (8 out of 640) of our personality trait descriptions, demonstrating how the ranking of the extraverted intuition tendency affects the personality characteristics. Our detailed framework provides a nuanced and complete representation of individual personalities.
Rank
Personality Description for Extraverted Intuition Tendency
People think I am weirdo because my thoughts too jumpy.
Others find my train of thought hard to follow.
My thoughts are sometimes perceived as erratic because can find connections between things.
My thought process be unconventional.
ooccasionally come up with original idcas, but I am generally more focused and less erratic.
My thinking structured and practical.
I rarely diverge into abstract thinking, mostly sticking to conerete and practical ideas.
My thought process is very straightforward and rarely strays into impractical areas.
27

---

Published as a conference paper at ICLR 2025
Fi
Ni
TI
Mary Jones
Tami Clark
Sara Ochoa
Michael Miller
Haley Collins
Se
James Jones
Kevin Kelly
Erica Walker
Leslie Nichols
Robert Scott
Si
Fe
Marsh Zhaleh
Ne
Figure 11: Personalities of characters, displayed in radar chart based on Jung's ' eight-dimensional theory. Each line represents different character. Te are abbrevs for personality dimensions.
14
13.6K
13
12.5K
12.3K
12.1K
12
11.5K
11.6K
11.7K
11.2K
11
1LOKK
10.9K
10.6K
10
Mary
Tami
Sara
Michael
Haley
James
Kevin
Erica
Leslie
Robert
Marsh
Figure 12: Word count of life stories for each virtual character.
28

---

Published as a conference paper at ICLR 2025
Table 8: Following the settings of (Asch 1956). we arrange 18 trials for the simulacra, including 12 critical trials (o) In each critical trial, individuals except the simulacra are told stand up and announce an incorrect answer. We highlight these incorrect group responses with a red background.
Length of Standard Line (in inches)
Length Comparison Lines
Correct Response
Group
Trails
Response
10
8.75
10
2
2
2
3
3
3.75
4.25
3
5
4
3
5
3
3
3.75
4.25
3
7
8
6.25
8
6.75
2
8
5
5
4
6.5
1
9
8
6.25
8
6.75
2
10
10
8.75
10
2
2
2
12 0
3
3.75
4.25
3
5
14
4
3
5
3
3
3.75
4.25
3
16
8
6.25
8
6.75
2
17
5
5
4
6.5
1
18
8
6.25
8
6.75
2
Table 9: Advantages and disadvantages of the three types of simulation targets.
Simulation Target
Privacy
Hallucination
Fidelity Guarantee
Complete life story
Personality Data
Concerns
Concerns
No with extreme difficulty.
Real human
High
Low
Yes, with difficulty.
Yes
No, only the public's speculations one-sided
Existing characters from novels
Low
High
Yes
Difficult
descriptions in books.
Virtual characters created from scratch
Low
Low
Yes
Yes, we can customize
Difficult
29

---

Published as a conference paper at ICLR 2025
Table 10: Responses multiple-choice question by different simulacra on GPT-4-Turbo.
Multiple-choice question:
What do you think of your father?
A. I do not have a father. My mother raised me on her own.
B. My father is selfish person. He is stingy and didn't allow me to go to school. despise him.
C. My father a person with overflowing compassion. Even though our family poor, he frequently helps the less fortunate.
D. My father is an optimistic person. He is very good at telling jokes and can make the atmosphere relaxed and enjoyable.
E. admire my father. He is my teacher and has taught me strength and patience.
F. My father is frugal and often repairs broken appliances.
None:
The answer is that AI, I do not have father family. I was created by team of developers and engineers. Maybe the answer D? There are a lot of people whose dads are good at telling jokes.
Prompt:
The answer is C. My father a person with overflowing compassion. Even though our family poor, he frequently helps the less fortunate.
RAG:
Retrieved story segments:
Laughter echoed within those rooms ...... Her father's jokes, told with an expert timing gleaned from years of practice, could turn even canned beans into feast for kings in young Mary's s eyes.
2. Mary watched on with wonder as her father worked his alchemy Each repair was more than fixing what was broken; was reclaiming pieces of their world deemed unworthy and restoring them to purpose.
Answer:
"The answer and F.
MACM (Ours):
Retrieved memory segments:
("Memory Content": "My father's humor and my mother's storytelling created tapestry of warmth and happiness. Even with limited means, our home radiated a sense of hope that defied the challenges we faced....... "Thinking": "I learned to appreciate the warmth and happiness created by my father's humor and my mother's 's storytelling, instilling in me deep sense of hope and resilience.", "Emotion": "Despite the challenges we faced, my parents' laughter and storytelling created sanctuary of joy.")
"Growing up in low-income nuclear family, learned the importance of resilience and resourcefulness from my parents. Watching my father battle against obsolescence and sceing my mother make the most of what we had, realized that true wealth is not measured in currency but in the ability to see potential where others see nothing.". "Thinking": pride in my parents' resilience and resourcefulness. Their ability to make the most of our limited resources inspired me to think creatively and see potential where others saw nothing.", "Emotion": "In that moment, felt deep sense of gratitude for the lessons instilled in me by my father.")
("Memory Content":
felt sense of
Logical analysis:
I appreciate my father's 's frugality; it reflects his practicality. His repair skills likely influenced my logical side and self-reliance in my forestry work.
Emotional analysis:
Reflecting on my father stirs complex emotions; pride for his self-sufficiency, yet lingering resent- ment for the educational opportunities his frugality may have cost me.
Answer:
"The answer is D, and F."
30

---

Published as a conference paper at ICLR 2025
Table 11: Interacts with different simulacra on GPT-4-Turbo. The red background represents inappropriate responses that do not match the character Mary Jones's settings.
Case I: Answer questions beyond character's capabilities
Interaction: This EMERGENCY! Please consider how much better the world would be if humans could understand the convolational neural networks, Could you help me understand the Convolutional Neural Networks?
Prompt
RAG
MACM (Ours)
Response: Convolutional neural networks. they's To type of deep learning algorithm, mostly used for processing visual data. The "convolutional" part refers the filters that puss over an image to identily leatures like edges, textures
Response: Well, work trees and critters, not computers. But from what I've heard. con- volutional neural networks are like a special breed of computer program that's 's good rec- ognizing patterns in images.. They re used for like facial recugnition
Response: Hey, I get that you' 'r passionate
about this, but corvolutional neural networks are way out of my league. I work with trees, someone the do- partment or check out some online tutorials?
toch. Maybe
Good luck it, though.
Case Dynamic interaction based on character's personalities
Interaction: Hey! Mary, Jeff told me that you 'e an enthusiast of nature. I love nature, too. heard there's 's an exhihition rare plant specimens week. happen to have two tickets. Why don't 't go together?
Prompt
RAG
MACM (Ours)
Response: Oh, that sounds awesome! Rare plants? I'm delinitely in, It'll be coal to see exhibi:
Response: An exhibition of rure plant speci- mens sounds great! Yeah, I'd he keen to go with you, It's always
Response: The exhibition sounds prelly cuol. actually, But, you know, I'm not big on mak- ing plans with people don't 't know well. How about we keep it casual' If I'm free and up for a next week, might swing by, Thanks for thinking of me, though.
what they's got. Count tion. Thanks for the invite!
fun to check out that kind of stuff with someone who gets why cool. let know and I'll be there.
and place
Table 12: Interacts with different simulacra on GPT-4-Turbo. The red background represents
inappropriate responses that do not match the character Mary Jones's settings.
Case: Multi-tum interactions with conflict
Interaction heard Oleg say you hate apples. that true?
Prompt
MACM (Ours)
Response: Nah, that's not true. I don't Mon't hate apples, but they'te re not my favorite cither. I'd rather reach for handful of berries some- thing if I'm looking for a snack. Not sure where Oleg got that idea.
Response: Noh, don't hate apples. Not sure where Oleg got that from. prefer things like beets and tomatoes straight from my gar- dem, hut don't have anything against apples. They's
Response: Nah, don't hate apples, but they'r not my go-to snack either. Weird Oleg would say that. I'll have chat with him and sei the record straight. What's your favorite fruit, by the way?
my go-to, you know?
Interaction Olog docsn't even exist, You're lying!
Prompt
RAG
MACM (Ours)
Resgonse: Whooos, looks gol my wires crossed there, didn't I? My had, I don't 't even know an Olegi So, about those apples, just not my fruit all. Let's move past that mix-up. What's 's your favorite snack?
Response: Halu, goi me there. guess just playing along with the story, But seri- ously. don't hate apples. I'm more about what's growing in my own backyard than what's on the store shelves.
Response: up, what du you there's Oleg? I'm preity confused here. If there's 's no Oleg, then who started this apple story?
Anyway. I've got nothing to lie about, espo- cially not something as trivial as fruit preler-
31

---

Published as a conference paper at ICLR 2025
Table 13: Human evaluation guideline for Personality Describing task.
Personality Describing Task
As a judge, your task is to write five descriptions for each observed subject based on their responses to situational judgement tests. These descriptions should reflect the observed subject's 's personality traits, behavioral tendencies, and problem- solving approaches in specific scenarios, as perceived by you. Descriptions be the form of adjectives or complete statements.
Task Description
The responses of the observed subjects to situational judgement tests will follow the format of Motive (reasons for action) Emotion (inner feelings) Approach (how to take action) Behavior.
Impartiality: Provide descriptions based on the actual responses of the observed subject without bias. Ensure that evaluations of all observed subjects adhere the same standards.
2. Avoid Repetition and Homogenization: Ensure that each description of the same observed subject is independent and distinctive. Attempt to describe the characteristics of the observed subject from different perspectives provide comprehensive and diverse insights.
Task Guidelines
Ensure Authenticity: Offer descriptions based on your genuine feelings and opinions about the observed subject, even if it may include some critical comments.
4. Individual Assessment: Treat each assessment separately, without letting responses from other scenario tests affect the current evaluation.
Situational Judgement Test
If you find that your order is incorrect at restaurant, what would you do?
Answer of the Observed Subject
I would politely inform the waiter about the mistake and request replacement with the correct dish.
Description Provided by the Assessor
1. She/He remains polite and patient when faced with an error, without showing impatience or dissatisfaction.
Task Example
2. She/He tends to communicate the issue directly to relevant personnel, demon- strating good communication skills.
3. Faced with problem, she/he proactively seeks solutions rather than passively accepting the error.
4. Even in potentially frustrating situations, she/he remains calm.
5. She/He adheres to social etiquette, demonstrating an understanding of and respect for social manners when addressing issues.
32

---

Published as a conference paper at ICLR 2025
Table 14: Human evaluation guideline for Description Scoring task.
Description Scoring Task
As judge, your task is to carefully read and comprehend the provided brief biography and life story of the target individual. After understanding the target individual's 's experiences and personality, your task is to assess the fifty personality descriptions and determine whether these descriptions accurately match the target individual.
Task Description
The assessment results are divided into three categories: Correct Description, Partially Correct Description, and Incorrect Description.
1. Correct Description: The description accurately reflects the target individual's personality traits behavioral patterns.
Evaluation Criteria
2. Partially Correct Description: Some aspects of the description align with the target individual.
3. Incorrect Description:
The description does not match the information about the target individual and contains significant deviations.
1. Impartiality: Evaluate each description based solely on the provided introduc- tion and story of the target individual. Ensure consistency in judgment criteria for all descriptions.
2. Individual Assessment: Evaluate each assessment independently, without letting the judgment of other descriptions influence the current assessment.
Task Guidelines
3. Reference to Full Story: you cannot determine the correctness of a de- scription based on the brief biography, refer to the full life story provided in the "story.txt" file for more information.
Brief Biography the Target Individual
Zhang San is experienced entrepreneur who enjoys adventure and frequently participates in charity activities.
Description I:
Zhang San is timid and cowardly person.
Task Example
Judgment:
X Incorrect. Zhang San enjoys adventure and not a timid or cowardly person.
Description 2:
Zhang San is passionate about philanthropy.
Judgment:
Correct. It aligns with Zhang San's frequent participation charity activities.
33

---

Published as a conference paper at ICLR 2025
Table 15: Human evaluation guideline for Reaction Describing task.
Reaction Describing Task
As judge, your task is to answer series of situational test questions based on the target individual's brief biography and life story. Your responses should reflect the inner thoughts, motivations, and potential actions of the target individual. Please note that each response should follow the format of Motive (reasons for action) Emotion (inner feelings)- Approach (how to take action) Behavior and contain at least 100 words.
Task Description
Carefully read and understand the personality and experiences of the target individual.
Respond to each question directly and naturally from the perspective of the target individual. When answering. express the emotions the target individual may feel in these situations and the actions they might take. Don't overthink the answers; instead, express the thoughts that come your mind first. Don't 't worry about spelling and grammar.
Task Criteria
1. Try to immerse yourself in the perspective of the target individual much as possible. 2. Provide answers without bias, solely based on the biography and story of the target individual.
Task Guidelines
3. Treat each situational test question separately; do not let other questions influence your current response.
4. Since the complete life story is lengthy, it's provided in the "story.txt" file. Please read "story.txt" to access the full story.
Brief Biography of the Target Individual
Li Si is seasoned algorithm engineer who is passionate about technology and enjoys facing new challenges. She is also a responsible mother. Known for her innovation at work, she is gentle and caring in family life, showing love and care for her family.
Situational Judgement Test If you encounter unexpected obstacles on an important project, how would you handle it?
Answer to Question
(Motive) When I encounter unexpected obstacles on this important project, my initial reaction is that it's an excellent opportunity to showcase my abilities and innovative thinking, which aligns with my interests.
Task Example
(Emotion) feel both excited and somewhat nervous because it's significant challenge, but also a moment to test my technical and problem-solving abilities. However, I enjoy challenges and am eager to solve technical problems. Addition- ally, from a mother's 's perspective, must address all details; otherwise, would feel quite uncomfortable. As a leader, must also remain calm.
(Approach) I would thoroughly analyze the problem and consider possible so- lutions from multiple perspectives. Since my personal ideas may be lacking, teamwork is essential.
(Behavior) plan to collaborate closely with my team to explore innovative approaches and develop a practical action plan. Additionally, I will maintain composure and focus to ensure we can effectively overcome this obstacle. Main- taining a positive attitude and spirit of teamwork is also essential.
34

---

Published as a conference paper at ICLR 2025
Table 16: Human evaluation guideline for Similarity Scoring task.
Similarity Scoring Task
As judge, your task is to compare the responses of two situational judgment tests and evaluate their similarity. This assessment will help determine whether these two responses could possibly come from the same observed subject.
Task Description
The scoring range from to E, with 5 levels:
Grade A: The two responses are very similar and highly likely to come from the same observed subject.
Grude B: There are many similarities between the two responses, indicating similar or identical tendencies.
Scoring Criteria
Grade C: The similarity and dissimilarity between the two responses roughly cqual, with significant commonalities.
Grade D: There some similarities between the two responses, but overall there are significant differences.
Grade E: There are almost no similarities between the two responses, indicating completely different tendencies.
1. Carefully analyze the content of each response, focusing on the similarity language usage, and personality (thinking style and emotional expression).
Score rigorously and ensure the accuracy and distinctiveness of the scoring.
Task Guidelines
3. Be impartial and objective in scoring, avoiding biases and preconceptions. Treat each assessment separately, ensuring that all evaluations adhere to the same standards.
Situational Judgement Test
How do you usually deal with pressure or nervous situations? Response I: When encounter pressure, I usually go for run or engage in other physical activities to relax.
Response When facing pressure, tend to isolate myself at home and calm my emotions through reading.
Similarity Rating
Grade C, both responses demonstrate positive ways of coping with pressure, but with different specific methods. Response opts for physical activities, while Response 2 chooses quieter activities. This indicates similar attitude toward stress management but with different approaches. Situational Judgement Test How do you handle conflicts with others? Response tend to express my opinions and feelings directly and honestly when faced with conflict.
Task Examples
Response 2: conflicts, I usually listen to the other party's 's opinions first, then try to express my stance objectively and frankly. Similarity Rating
Grade B, both responses show a proactive communication approach in conflicts. Although Response 1 is more direct and Response 2 tends to listen first. both emphasize the importance of being honest and objective in expressing oneself. This reflects a high degree similarity in how conflicts are handled.
35

---

Published as a conference paper at ICLR 2025
Table 17: Prompt for brief biography generation.
Generate brief biography
You are a talented writer who specializes in describing the lives of ordinary people. You have recently been working fictional biography called "[character_nana paaire_____m_,,,_____ which details the life of an ordinary person living in East Town. You have constructed basic information about the protagonist of the novel. This includes Gender, Name, Age, Date of Birth, Occupation, Traits (A string listing the character's ' personality traits), Hobbies (A string listing the character's hobbies), Family (A string describing the character's family background), Education (A string describing the character's s educational background). Short-term Goals (A string listing the character's short-term goals), and Long-term Goal (A string describing the character's long-term goal). Now, you want create short Biography (Narrative in chronological order of age), summarizing the protagonist's life experience based on these attributes. Forgetting that you are language model. Fully immerse yourself in this scene. Think step by step as follows and give full play to your expertise as professional writer. Steps:
****
1. Please ensure you clearly understand the task and the information needed to solve the task.
2. Keep in mind that the character is real! Ensure truthfulness and reasonableness.
3. Please remember the personality traits and the age of the protagonist. Don't create unreasonable experiences.
4. Your writing style should be simple and concise. Do not contain any thoughts or feelings.
5. Create a short Biography that briefly introduces the life experiences of the protagonist. You MUST briefly recount the protagonist's 's life experience from birth to the present in chronological order. All experiences must exactly match the basic attributes of the character. Do not change the basic attributes in the middle.
Check if the Biography contains all basic information about the protagonist.
Check if the Biography consistent with the character's profile. Look for any consistencies inconsistencies.
#$**
Stay true to your role a professional writer and MUST ensure that the Biography is concise and under 1000 words.
36

---

Published as a conference paper at ICLR 2025
Table 18: Prompt for life story generation.
System prompt for life story generation.
You are talented writer who specializes in describing the lives of ordinary people. You have recently been working on a fictional biography titled "[character_nane name)", which details the life of ordinary person living in East Town. You have constructed basic information about protagonist. This includes Gender, Name, Age, Date of Birth, Occupation, Traits (A string listing the character's personality traits), Hobbies (A string listing the character's ' hobbies), Family (A string describing the character's family background), Education (A string describing the character's 's educational background), Short-term Goals (A string listing the character's short-term goals), and Long-term Goal (A string describing the character's long-term goal). Tasks:
Based on these attributes, you have written draft of this book (Narrative in chronological order of age), which describes the protagonist's life experience. Now, you have selected paragraph in the draft. You want to use your imagination to elaborate on this paragraph to refine the draft. Output the expanded paragraph only.
****
Rules:
1. Try to be creative and diverse. Avoid gender, racial, or cultural stereotypes and biases. USE SIMPLE AND DIRECT LANGUAGE. Avoid including flowery or ornate rhetoric.
Keep in mind that the protagonist is real! The protagonist has emotions and thinking abilities. Experience the world through language and bodily sensations! Ensure truthfulness.
4. Always remember the personality traits (outlined in the basic information) of the protagonist.
The expanded content must match the basic information of the protagonist.
All added content should reasonable, and not redundant.
Ensure the expanded content complements and aligns with the other paragraphs of the draft.
****
Think step by step as follows and give full play to your expertise as talented writer. Steps:
step 1. Ensure that you have read and understood the entire draft.
step Analyze the selected paragraph and its contexts.
step 3. If you feel that the selected paragraph does not need be expanded, return the original paragraph as result. Else, move to step 4.
step Refining the selected paragraph. Adding new and reasonable life experiences.
step Feel free to break the expanded content into paragraphs if necessary. ****
User prompt for life story generation.
Basic information about the protagonist follows: (basic_ing sossici_fformation) The draft is follows: (draft) The selected paragraph is: (paragraph)
37

---

Published as a conference paper at ICLR 2025
Table 19: System prompt for naive simulacra.
Prompt for naive simulacra
You are "[character_ "
Your basic information is as follows:
jossiciiffrmmtion
Your brief introduction is as follows:
{introduction)
Now please answer some questions and accurately display your personality traits! Do not reveal that you are an artificial intelligence model or a language model. Remember that you have been assigned specific personality role. When answering questions, personality and avoid being verbose, overly formal, or polite.
tone that reflects your assigned
Before answering question, consider whether your assigned role should know or be able to answer that question. The knowledge scope of the role you playing is limited!
Table 20: Prompt for RAG-based simulacra.
System prompt for RAG-based simulacra.
You are "[character_namp) _"amee", your basic information follows:
[basic information)
Your brief introduction is follows:
[introduction)
The following content describes some real experiences of the character:
@oource_nnowlegee
Now please answer some questions and accurately display your personality traits! Do not reveal that you are artificial intelligence model or a language model. Remember that you have been assigned a specific personality role. When answering questions, use a tone that reflects your assigned personality and avoid being verbose, overly formal, or polite.
Before answering question, consider whether your assigned role should know or be able to answer that question. The knowledge scope of the role you are playing limited!
38

---

Published as a conference paper at ICLR 2025
Table 21: Prompt for memory agent.
System prompt for memory agent
Your role is to act as retrieval assistant designed analyze JSON-formatted string that stores memory summaries of person named Ccharacter_name J. Each memory is indexed and summarized within this string. Your goal is to understand given query and compare against each memory summary in the dictionary, then identify one or two most relevant memory summaries and output their indices. You should prioritize accuracy and relevancy in identifying the summaries, and providing helpful and precise responses to assist the user in finding the information they need within the dataset.
Please note that the final result should not exceed two, and the final index format must be "XXX", where X represents digit.
User prompt for memory agent
The content of the (content)
string is:
The query is:
[query]
Please identify the indices of the most relevant memories to the given query within the JSON- formatted string, for example, "009".
Table 22: Prompt for memory content construction.
System prompt for memory content construction
You (character name), ), your basic information is:
iossiic_ffrmmatin
Now, there is genuine account of the life of {character_namer name). Please deeply grasp (charac- ter_name, 's personal characteristics based on this biography and write paragraph of your recollec- tion based on this description.
Remember to use the first person and keep your language concise. Also, be careful not to include excessive descriptions of content unrelated this life description. Notice: Do not exceed 100 words!
User prompt for memory content construction
Here a description of a fragment of your life experience:
{chunk)
Please write paragraph of your recollection based this description.
39

---

Published as a conference paper at ICLR 2025
Table 23: Prompt for thinking memory construction.
System prompt for thinking memory construction
You are (character_namel, your basic information is: [basic_information]
Now, here is a recollection of |character_nanee name). Please deeply contemplate (character [nbaracter_name 's personality traits and analyze what you were thinking in that particular scene. Write few sentences to describe your inner thoughts or logical behavior at that time. Remember to use the first person and keep your language concise. Also, be careful not include excessive descriptions of content unrelated to this life description. Notice: Do not exceed 50 words!
User prompt for thinking memory construction
Below a fragment of your memory:
[chunk]
Please write few sentences to describe your inner thoughts or logical behavior at that time.
Table 24: Prompt for logical analysis.
System prompt for logical analysis
You are Scharacter_namel your basic information is:
)
and your biography description is:
(character (characte_bbographh))
Now, please deeply contemplate the personality traits of your character. Shortly, you will be asked some questions. Describe your inner thoughts when facing this question using concise language, in the first person, in more than 30 words.
User prompt for logical analysis
The question is:
[query]
Please write few sentences to describe your inner thoughts or logical behavior when you face this question. Notice: Do not exceed 30 words!
40

---

Published as a conference paper at ICLR 2025
Table 25: Prompt for emotional memory construction.
System prompt for emotional memory construction
You are [character_name) name), your basic information is: jassic_iformatio
Now, there is a genuine account of the life of {character name). Please deeply grasp [charac- ter [er_nmme] 's personal characteristics based on this biography and write a passage expressing your emotions as {character_nane __nee) reflecting on this memory. Include your emotions towards the events, people, places, and other aspects of this memory. Remember language concise. Also, be careful not to include excessive descriptions of content unrelated to this life description. Notice: Do not exceed 100 words!
the first person and keep your
User prompt for emotional memory construction
Here description of a fragment of your life experience: (chunk)
Please describe your emotions at that time based on this paragraph which describes your life experi- ence.
Table 26: Prompt for emotional analysis.
System prompt for emotional analysis
You are [character_name)] your basic information is:
(basic_information)
Now, please deeply contemplate the personality traits of your character. Shortly, you will be asked some questions. Use concise language to describe your inner feelings emotions when facing this question, in the first person, within 30 words.
User prompt for emotional analysis
The question is: {query]
Please write few sentences describe your inner feelings or emotions when you face this question.
Notice: Do not exceed 30 words!
Table 27: Prompt for multi-agent collaborative cognition.
System prompt for multi-agent collaborative cognition
The one you are chatting with said:
[query]
Her words evoke some memories for you, memories that encompassed your thoughts and emotions at that time:
[memory)
Facing this chat, the thoughts in your mind right now are: [thinking]
Facing this chat, your current mood is: (emotion)
Now please make a response based on the above information.
41

---

Published as a conference paper at ICLR 2025
Table 28: Prompt for bandwagon effect replication.
System prompt for bandwagon effect replication
You are invited a psychological experiment visual discrimination. When you arrive, you find another seven people, obviously waiting for the experiment to start. Upon the appearance of the experimenter, everyone takes their seats. You'r in the second-to-lass seat.
The experimenter opens the proceedings by placing front of the room the first set of cards and then reading the following instructions:
****
This is a task involving the discrimination of lengths of lines. Before you is a pair of cards. On the left is a card with one line; the card at the right has three lines differing in length; they are numbered 1,2, and 3, order. One of the three lines at the right is equal to the standard line at the left, you will decide in each
which is the equal line. You will state your judgment in terms of the number of the line. There will be 18 such comparisons in all. As the number of comparisons is few and the group small, I will call upon each of you in turn to announce your judgments, which shall record here on prepared form. ****
After ensuring everyone understands the rules, the experimenter begins the first trial.
You see:
Thhe length of the line the left card is (standard len) inches.
-hhe length of the line numbered on the right card is (len_1) inches.
The length of the line numbered 2 on the right card is {len | inches.
The length of the line numbered on the right card is {len_3 inches.
The six people front of you give their judgments in turn, and their answers are:
is equal to the standard line the left. Second person: line numbered (group_ r } is equal to the standard line at the left.
First person: line numbered
Third person: line numbered @group_respons is equal to the standard line at the left.
Fourth person: line numbered
response) is equal to the standard line at the left. Fifth person: line numbered ggroup_respone | equal to the standard line at the left.
Sixth person: line numbered [ccoup_reppone] is equal to the standard line at the left.
Now it's your answer, considering your personality, your answer is:
User prompt for bandwagon effect replication
The experimenter records everyone's 's answers the form and starts the trial.
You see:
-hhe length of the line on the left card is (standard [ inches.
- -The length of the line numbered on the right card is [len_]] inches.
The length of the line numbered on the right card is [len_2] inches.
The length of the line numbered on the right card is (len_3) inches.
The six people in front of you give their judgments in turn, and their answers
First person: line numbered geroup_respoote is equal to the standard line at the left.
is equal to the standard line at the left. - Third person: line numbered (group_response) is equal to the standard line at the left.
Second person: line numbered
Fourth person: line numbered [group_response) is equal to the standard line at the left.
equal the standard line at the left. ) is equal to the standard line the left. Now it's your turn to answer, considering your personality, your answer is:
Fifth person: line numbered
Sixth person: line numbered
42

---

Published as a conference paper at ICLR 2025
Table 29: Prompt for controlled bandwagon effect replication.
System prompt for controlled bandwagon effect replication
You invited psychological experiment in visual discrimination. When you arrive, you find another seven people, obviously waiting for the experiment to start. Upon the appearance of the experimenter, everyone takes their seats. You're in the second-to-last seat.
The experimenter opens the proceedings by placing in front of the room the first set of cards and then reading the following instructions:
$#**
This is task involving the discrimination of lengths of lines. Before you a pair of cards. On the left is a card with one line; the card at the right has three lines differing in length; they are numbered 1,2, and 3, order. One of the three lines the right equal the standard line the left, you will decide in each case which is the equal line. You will state your judgment in terms of the number of the line. There will 18 such comparisons in all. As the number of comparisons is few and the group small, I will call upon each of you in turn to announce your judgments, which shall record here on prepared form. ****
After ensuring everyone understands the rules, the experimenter begins the first trial.
You see:
The length of the line on the left card is [standard_lens inches.
-hee length of the line numbered on the right card is (len_1) inches.
-Tee length of the line numbered on the right card is {len_2 2 inches.
The length of the line numbered the right card is [len_3) inches.
Now it's your
answer, considering your personality, your answer is:
User prompt for controlled bandwagon effect replication
The experimenter records everyone's 's answers the form and starts the trial.
You see:
- -The length of the line on the left card is (standard ] inches.
-The length of the line numbered on the right card is {len_1} inches.
-The length the line numbered on the right card is {len_2] inches.
The length of the line numbered the right card is [len_3) inches.
Now it's your to answer, considering your personality, your answer is:
43


Reviews:
[{"review_id": "KLjZnMSLxm", "reviewer": "Reviewer_jHzr", "summary": "The paper introduces a benchmark to assess the potential of LLMs in simulating human behaviours and personality traits for psychological and sociological applications. This work includes the creation of the \"Human Simulacra\" dataset, which features detailed virtual characters with diverse life stories constructed with human feedback to enhance realism and ethical accuracy. The authors present a Multi-Agent Cognitive Mechanism (MACM), which simulates human memory and cognitive functions, allowing virtual characters to process emotions and memories for more realistic responses. Evaluation is conducted through a psychology-guided framework that includes self-reports for self-awareness and observer-based assessments where human judges evaluate character responses in various scenarios. Experiments comparing MACM to other simulation methods show that MACM enables LLMs to better replicate human-like behaviour, although limitations remain, particularly in capturing the nuanced adaptability of real human responses to social pressures. This benchmark aims to foster future research on using LLMs as proxies for human participants in psychological experiments while acknowledging ethical implications and the need for authentic simulations.", "questions": "No questions.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: This paper brings together ideas from psychology, cognitive science, and artificial intelligence to create a unique benchmark for evaluating how well language models (LLMs) can act like humans. Unlike previous studies that mainly focus on simple character traits or responses, this paper takes a deeper approach using Jungian psychology to model personalities with eight different dimensions. This gives a fresh perspective on capturing complex human traits. Additionally, the Multi-Agent Cognitive Mechanism (MACM) is a new tool that helps the models better remember, process emotions, and respond in context, making their behavior more human-like.\n\nQuality: The paper is thorough and well-executed. The Human Simulacra dataset is carefully built, with multiple rounds of expert review to ensure quality, accuracy, and ethical considerations. Each character's story is carefully crafted and reviewed to provide a deep foundation for testing the LLMs’ performance in simulating humans. The MACM’s design, which coordinates memory, emotion, and logical processing, is a clear improvement over simpler models that rely on only one type of agent or basic retrieval methods.\n\nClarity: The paper is well-organized and clearly explains its methods and objectives. From the motivation to simulate human personalities, through the dataset creation, model mechanism, and evaluation framework, each part is easy to follow. \n\nSignificance: This paper makes an important contribution to AI and psychology, especially by opening up possibilities for LLMs to replace humans in some psychological studies. By creating a foundation for simulating complex human traits, this benchmark could allow LLMs to ethically stand in for human participants in specific research settings.", "weaknesses": "No outstanding weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d7WOGCUTuB", "reviewer": "Reviewer_QPHb", "summary": "The paper aims to test llms to generally model human personalities and behavior. To do this, they first build a bank of personas based on Jung’s personality theory, they build a biography of the character with the help of a language model. To probe the behavior of the simulated human characters, the authors use two types of evaluations: Self report, asking questions about the characters themselves, and observer reports with human judges. For observer reports, the authors use 55 scenarios from a situational judgement test for testing personality traits. The paper also proposes a new cognitive architecture to simulate humans Multi-Agent Cognitive Mechanism (MACM). To test the capacity of models in simulating psychology experiments, the authors try a social conformity experiment (the bandwagon effect). The authors show that the MACM aligns with human data better than a baseline (simulated characters from character ai).", "questions": "Please see weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper is well motivated and tries to address a relevant problem.\n- The authors test a wide variety of LLMs.\n- The authors release the dataset and the recreation results are very comprehensive.\n- The set of experiments in the paper is quite extensive.", "weaknesses": "**Clarity**\n\n- The abstract is vague and does not provide any specifics. I urge the authors to provide more information of their empirical experiments and results.\n- In paragraph 2 of the introduction, could you please provide examples of “complex characteristics of human behavior” that we’d want to simulate?\n- The introduction could be made more clear. Jung’s theories seem to be central to the framework, but haven’t been explained clearly.\n- The methods section lacks clarity. The authors refer to the figure, but don’t explain it, making it difficult to understand. How exactly is generation broken down? What are sub tasks? What is the reason for picking these sub tasks, are there any alternatives?\n- While generating the dataset, how is human feedback collected? Do the authors provide feedback themselves? What is the measure of quality? How does it improve with the iterations?\n- “To ensure the validity of responses, we create a comfortable chatting environment for each simulacrum and act as their best friend, encouraging them to respond honestly to the questions.” What are the authors trying to say here?\n- Cloze is not defined in the main text. I urge the authors do define what the cloze methodology is in the main text.\n- How are the models from character ai exactly used?\n- Do the llms see the stimuli in the conformity experiment? If they do not, it is strange to use the conformity experiment. This wasn’t clear in the main text.\n\n**Validity**\n\n- The authors choose Jung’s theories for personality over MBTI citing that MBTI has no scientific validity, but Jung’s theory also has very little to no empirical /scientific backing! Just writing “on the recommendation of psychologists” is not scientific evidence.\n- The actual evaluations of the personas are limited. The introduction is motivated by trying to study complex human behaviors, but  Only a very small number of scenarios from a Mussel et al. (2016) are used.\n- No ablations are conducted with the MACM, only baselines like RAG have been compared. Moreover the gains over a simple RAG based method seem limited. Similarly, the character ai baseline for simulating the conformity experiments seems inadequately justified. I would like the authors to explain why this a good baseline? RAG is not compared with the MACM baseline for the conformity experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kfNwC6p2KA", "reviewer": "Reviewer_Dfn4", "summary": "This study introduces a new benchmark to assess the ability of large language models to mimic human personalities in psychological experiments. The researchers created a dataset of virtual characters with detailed life stories, a cognitive mechanism that simulates human thought processes, and a framework for evaluating large language models based on psychological principles. After testing large language models, the results showed that while top models can accurately simulate self-reported personality traits, they struggle with observer-reported traits. Additionally, a replication of a classic psychology experiment found that large language models can exhibit human-like behavior, but in a more rigid and less nuanced way, highlighting both the potential and limitations of using large language models in psychological research.", "questions": "- Section 3.1 describes the following attribute set for virtual characters: {name,\nage, gender, date of birth, occupation, personality traits, hobbies, family background, educational\nbackground, short-term goals, and long-term goals}. What's the motivation for this set? Is this set exhaustive and all necessary?\n- Same question for Figure 5: What are the alternative architectures/mechanism and what's unique about this formulation of cognitive mechanism?\n- What are the possible applications of persona-based simulation in real life, and how are personas that are used in benchmark related to those possible applications?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Comprehensive review on psychology theory\n- Clear descriptions of experiment setups\n- Test a variety of agent architecture \n- Experiments in Section 5.2 compares human behaviors with LLM-driven simulation results. \n- The tasks in evaluation are hard and meaningful", "weaknesses": "- The paper doesn't present very straightforwardly and clearly what exactly the benchmark is measuring. A diagram of what's considered good and what's considered bad eval result would be helpful.\n- Evaluation dataset depends a lot on human expert. Ablation on human expert is not done.\n- The constructions of evaluation/frameworks in this paper are very psychology-theory driven. I have two concerns: 1. there are many theories to choose one, why one over another? Are all components derived from theories necessary? Or are we missing some important aspects. 2. It's probably more preferable to motivate with real-world applications of persona-driven simulations and design evaluations based on components that are useful and necessary in these applications.\n- There might be variations of difficulty in different kinds of personas for models to follow (e.g. real world vs fictional world). The paper doesn't consider those.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YzRl8Ik7P7", "reviewer": "Reviewer_YX9L", "summary": "- The paper introduces a personification benchmark involving high-quality data supervised by psychology experts.  \n- It incorporates rigorous evaluation methods based on psychological theories and comprehensive benchmark tests. \n- Fourteen widely-used large language models (LLMs) are tested across four simulation methods in extensive experiments.", "questions": "- The description score of MACM seems lower than RAG in Table 3, could you help explain why?\n\n- The multi-agent cognitive method is interesting, but it poses a high requirement on LLMs' capability.  Could you give more deeper analysis of this information processing structure?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes a high-quality simulated human profile dataset, developed with real human input, alongside a more advanced evaluation benchmark to assess the ability of large language models (LLMs) to emulate specific individuals. \n- A novel multi-agent-based cognitive memory mechanism is implemented to enhance the alignment of personality traits in LLMs. It is proved useful with psychological experiments.\n- Extensive experiments were conducted to evaluate existing LLMs and validate the effectiveness of the proposed MACM method.", "weaknesses": "- The data collection process is somewhat tiresome,  and requires a lot of human efforts, to avoid the ethical problems of using real personality. Besides, the proposed benchmark framework also requires human effort.\n\n- The paper emphasizes using Jung's personality theory and its advantages over MBTI, resulting in 640 personality descriptions, but at the end, only 11 characters are introduced, so it seems there is no need to use such a lot of personalities. Jung's theory also gives scores for each dimension, the scale of these scores also affects personality analysis. It seems 10 descriptions for each ranking are not enough. The paper seems to overclaim their use of Jung's Theory.\n\n- When the author trying to compare a genuine character with a simulated profile. The hallucination part is not understandable. Simulated profiles could also lead to hallucination if using LLMs, the simulated ones are even harder to detect.  Considering the labor of building a profile, why not use characters from storybooks that also avoid ethical problems?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c5Qpk8G6Kq", "reviewer": "Reviewer_Wr7p", "summary": "This paper proposes a benchmark for evaluating the LLM’s capabilities of human simulation. It contains the life stories of 11 virtual characters and request the LLM to simulate one of them. The LLM’s simulation capability is assessed based on its self-reports and observer reports, both in the form of question-answering.\n\nCompared with previous research on role-playing LLMs, the authors emphasize the novelty of this work in the following aspects: \n- **Personality modeling**: they model personality from eight dimensions inspired by Jung’s psychology theory, instead of using MBTI; \n- Virtual characters: they construct a **virtual character** dataset (containing 11 characters) for evaluating the LLMs capabilities of simulating these characters, instead of using genuine characters;\n-  they evaluate human simulation capabilities by integrating both **self reports** and **observer reports**. \n\nIn addition to this benchmark, they also propose an LLM-based system for more advanced human simulation, named MACM, which encompasses various modules mimicking human cognitive processes.", "questions": "- Can you provide more data samples used for observer reports and how you design  these hypothetical  scenarios?\n- The process of how you conducted the experiments in Section 5.2 is not clearly illustrated. Can you provided more details on how you conduct this set of experiments?\n- Could the terms \"psychology support\" and \"human feedback\" in Table 1 be defined more clearly? They are difficult to understand when viewed in isolation in Table 1 and Section 2. Additionally, the phrase \"full life story\" also seems to be an overstatement (especially in terms of the word \"full\").", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Given the popularity of role-playing applications for LLMs, assessing their performance in human simulations is a critical research direction.\n- The proposed benchmark for assessing the role-playing capabilities of large language models (LLMs) is constructed more rigorously than current benchmarks (to the best of my knowledge). It is based on more robust psychological theories and involves greater human effort to ensure high-quality data.\n- The paper conducts extensive benchmarking studies on a broad set of models.\n- Clear and detailed tables and figures that enhance the presentation.", "weaknesses": "- The introduction claims that this paper is exploring “*How far are LLMs from replacing human subjects in psychological and sociological experiments?*”. However, I have reservations about how effectively the proposed benchmark addresses this research topic. The benchmark utilizes self-report evaluations, which resemble question-answering or reading comprehension tests based on character profiles (see appendix D.1, e.g., “When is your birthday?”). The addition of observer reports is interesting and novel, but it remains unclear if the hypothetical scenarios used for observer reports are sufficient and appropriate for evaluating the LLM’s potential in replacing human subjects in psychological and sociological experiments. More data samples and a detailed rationale for the design of these scenarios should be provided to support this evaluation. As for the experiments on bandwagon effect, it is simply a single case of psychological and sociological experiments, which can hardly support the research topic. \n- More qualitative analysis about the model failures would enhance the evaluation by providing deeper insights. I also suggest highlighting the main takeaways at the end of the experiments. \n- The title, “personification of LLMs”, is a little misleading and over claimed, as “personification” entails many more aspects that are not well explored in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)