Human Simulacra: Benchmarking the Personification of Large Language Models

## Abstract

Abstract Large language models (LLMs) are recognized as systems that closely mimic aspects of human intelligence. This capability has attracted attention from the social science community, who see the potential in leveraging LLMs to replace human participants in experiments, thereby reducing research costs and complexity. In this paper, we introduce a framework for large language models personification, including a strategy for constructing virtual characters’ life stories from the ground up, a Multi-Agent Cognitive Mechanism capable of simulating human cognitive processes, and a psychology-guided evaluation method to assess
human simulations from both self and observational perspectives. Experimental results demonstrate that our constructed simulacra can produce personified responses that align with their target characters.
Our work is a preliminary
exploration which offers great potential in practical applications. All the code and datasets will be released, with the hope of inspiring further investigations.

## 1 Introduction

Researchers in psychology and sociology have long relied on human participants to conduct experiments that explore patterns of human behaviors and mental states (Camerer et al., ; Folke et al., ; Qiu et al., ). However, this method often faces numerous challenges, including the difficulty of recruiting human participants, high costs, and ethical considerations. Against this backdrop, the potential of large language models (LLMs) to mimic human interactions is drawing increasing attention.
Our question is: Can LLMs simulate human cognitive processes,
thereby assisting researchers in exploring
human interactions without involving real human participants?

There is already a discussion in the community, with psychologists and sociologists advocating the use of LLMs to replace human participants in experiments, thus reducing costs and complexity, and providing more controllable experimental conditions in certain contexts (Dillion et al., ; Hutson, ; Grossmann et al., ; Li et al., ; Kjell et al., ).
Despite the progress, the human simulations they created can only be used for group studies and cannot accurately replicate the nuanced responses of an individual. These simulations are hindered by limitations such as inconsistent performance across different types of tasks and a lack of depth in capturing the complex, context-driven characteristic of real human behaviors. The quest is on for a more advanced simulacrum that can
produce consistent behaviors indistinguishable from those of humans.

Computer science researchers are also striving to humanize LLMs, with a particular emphasis on role-playing tasks. They focus on replicating the professional skills (Salewski et al., ) or conversational styles (Shao et al., ; Tu et al., ; Wang et al., ) of target personas. However, as the goal of role-playing tasks is to convincingly portray the outward characteristics of target personas, these studies merely simulate the external traits of roles based on fragmented information about the character (e.g., brief introduction or scattered life experiences), lacking a holistic emulation of the inner patterns (e.g., emotion) of humans. Their models are unable to dynamically adjust the simulacrum’s behavior or personality in response to interactions with humans, which limits the potential of these models to mimic a broader spectrum of human behaviors. There is a need to explore more generalized human simulations to meet the demands of practical applications, especially psychological and sociological experiments.

In light of the above concerns, in this paper, we define the task of LLM personification as shaping the intrinsic traits of LLM based on the complete life story of the target character, enabling the LLM to simulate their specific personality and interact with the external world in a human-like manner. Given the definition, we propose a novel framework that includes a virtual character dataset, a multi-agent cognitive model, and a psychology-guided evaluation method.

Specifically, we start by
decomposing the complex task of crafting a detailed life story for the target character into several inter-related subtasks based on a hierarchical chain information model for personal information. We further propose a semi-automated strategy that tackles each subtask in sequence, moving from broad information (basic attributes) to detailed narrative (life story). Using this strategy, we create a multi-grained virtual character dataset named Human Simulacra. It comprises about 129k texts across 11 virtual characters, with each character having their unique attributes, biography, and story.

Drawing on the memory theories proposed by Atkinson and Shiffrin () and Baddeley and Hitch (), we design a Multi-Agent Cognitive Mechanism (MACM) to simulate human brain’s information processing and memory systems. As an external module, this mechanism enables the LLM to remember background stories, deeply understand target personalities, and express emotions in complex situations.

Lastly, considering the instability and potential bias (Wang et al., ) in using LLMs as task evaluators, and the limitations of popular Likert scale-based assessment methods (e.g., Myers-Briggs Type Indicator (MBTI) tests (Myers, ), Five Factor tests (McCrae and John, )), which can only evaluate certain aspects of simulated personality
(Furnham, ), we design a comprehensive evaluation method for “personified machines”.
Inspired by the personality measurement theories (Hogan et al., ) from psychology, our
method assesses the establishment of self-awareness and the ability to exhibit believable behaviors from both self-reporting and external observation perspectives.

Our contributions can be summarized as follows:

1. We introduce a semi-automated strategy to create vivid life stories for virtual characters from scratch. Based on this strategy, a multi-grained virtual character dataset, named Human Simulacra, is constructed, providing rich profiles and aligned experiences for virtual characters.

2. We design a Multi-Agent Cognitive Mechanism capable of transforming life stories of characters into long-term memories filled with information, emotions, and thoughts. These memories are then utilized in a multi-agent collaboration process, thereby achieving complex human simulation.

3. We develop a psychology-guided evaluation method to assess the quality of human simulations from two dimensions: self-reporting and external observation, providing a new way of measuring machines’ personalities.

## 2 Related Work

Memory Systems in Cognitive Psychology. In cognitive psychology, information processing approaches assert that cognition encompasses the entire process through which sensory inputs are transformed, reduced, elaborated, stored, retrieved, and used (Neisser, ). This cognitive view emphasizes the analogy between human brains and computers, suggesting that humans process information through sequential operations. This process is further summarized as the brain’s processing of external information and memory by Newell et al. (). Hence, investigating the working mechanism of the memory systems has become an important subject in cognitive psychology. The Multi Store Model of memory, proposed by Atkinson and Shiffrin (), divides memory into sensory memory, short-term memory, and long-term memory. Among them, sensory memory refers to the fleeting memory produced when the sensory organs receive stimuli from the environment, a portion of which, after processing, is converted into short-term memory. Short-term memory, when rehearsed, is then transformed into long-term memory.

With advancing research into short-term memory, Baddeley and Hitch () distinguished the concept of working memory from short-term memory, emphasizing that working memory is born for
storing, invoking, and analyzing information. In this paper, based on the memory theories discussed above and the capabilities of LLMs, we propose a Multi-Agent Cognitive Mechanism. It is designed to enhance the ability of LLMs to impersonate humans by constructing a memory system and simulating the cognitive process.

## 3 Human Simulacra Dataset

Figure: Figure 1: Hierarchical information chain model for virtual characters.
Refer to caption: x1.png

Figure: Figure 2: Process of constructing life stories for target characters using semi-automated strategy.
Refer to caption: x2.png

A hierarchical information chain model is introduced for generating data for virtual characters. As shown in Figure 1, this model organizes each virtual character’s information into three primary layers
and two sublayers,
allowing for information conversion between adjacent layers. From left to right, the difficulty in obtaining the information and its volume gradually increase. Conversely, from right to left, the information becomes more concise, focusing on the character’s most essential facts.

Existing role-playing studies have achieved shallow simulations (e.g., the tone of character’s voice) based on brief introductions (Wang et al., ) or scattered experiences (Shao et al., ).
While this information
represents parts of the target character’s traits,
they are fragmented and cannot depict the complete character. It is inappropriate to simulate the entire character solely based on summarized information. Instead, we need to directly simulate their human behaviors based on the character’s life story.

However, a human’s life story is usually highly complex, comprising numerous information fragments, making it challenging to employ LLMs to create a coherent life story for the character
without human supervision. Based on this hierarchical information chain model, we decompose the task of generating a character’s life story into interconnected subtasks and design a semi-automated strategy, starting with the basic attributes and iteratively generating a detailed life story for the target character. The entire process is depicted in Figure 2.

### 3.1 Character Attributes

Character attributes
encapsulate the core facts of a virtual character, serving as anchor points for the life story of the character. While designing attributes, it is necessary to ensure that these attributes are diverse, have reasonable connections, and conform to natural laws. Mirroring real humans, we design a comprehensive attribute set for virtual characters (Sloan, ), encompassing {name, age, gender, date of birth, occupation, personality traits, hobbies, family background, educational background, short-term goals, and long-term goals}. Each attribute has a candidate pool, covering diverse values applicable to most people. For instance, based on the International Standard Classification of Occupations (ISCO-08), we manually select 76 common occupations to serve as the occupation candidate pool.

### 3.2 Personality Modeling

Considering that personality encompasses an entity’s characteristic patterns of thought, feeling, and behavior (Hogan et al., ), how to accurately model the personality traits of the target character becomes a core challenge in attribute design. A naive approach is to rely on the Five Factor Model (McCrae and John, ) or MBTI theory (Myers, ), specifying the target character’s psychological type and utilizing LLM’s internal knowledge to understand the underlying personality of the character.
However, this method has unavoidable drawbacks: 1) LLMs are black-box models. Their internal understanding of psychological types may be erroneous. 2) Personality is a complex mixture that cannot be narrowly constrained to a certain type.

To tackle these issues, we adopt the eight-dimensional theory derived from Jung’s study (Jung, ). This theory divides personality into eight tendencies such as extraverted thinking and introverted sensing, with each tendency serving as a complementary facet.
Contrary to directly assigning numerical values to these tendencies, we employ a relative ranking strategy to indirectly assess the strength of each personality tendency within the character. Specifically, we rank the eight tendencies and establish a guideline that the tendencies at the top and bottom of the order are more pronounced in the character’s personality, while those in the middle are less pronounced, manifesting a blend of traits that vary in direction.

Hence, characters with different tendency rankings should correspond to different personality descriptions.
Overall, each tendency has eight possible rankings. Under the guidance of psychology professionals, we prepare ten suitable descriptions for each possible ranking, ultimately forming a personality candidate pool containing 640 trait descriptions. Compared to the naive approach mentioned above, our personality modeling method, grounded in authoritative, field-recognized theories (Jung, ), is able to depict the character’s personality more comprehensively and specifically.

### 3.3 Character Profile and Biography

To assemble the character’s profile, we first generate draft profiles by randomly selecting attribute values from their corresponding pools. This approach is fast, but it risks potential conflicts between attributes. Therefore, we add a Profile Selection module responsible for quality check and profile refinement in the generation process, as shown in the upper part of Figure 2. In this way, high-quality profiles are filtered out and fed into the LLM, which then generates a short biography summarizing the character’s life experience.

**Table 1: Basic Statistics of Human Simulacra.**
| Character Name | Age | Gender | Occupation | Story Length |
| --- | --- | --- | --- | --- |
| Marsh Zhaleh | 20 | Female | Undergraduate | 11.7K |
| Michael Miller | 25 | Male | Tourist Guide | 11.5K |
| Leslie Nichols | 27 | Male | Preschool Teacher | 10.9K |
| Mary Jones | 29 | Female | Forestry Worker | 10.6K |
| James Jones | 35 | Male | Van Driver | 11.0K |
| Kevin Kelly | 36 | Male | Software Developer | 12.1K |
| Sara Ochoa | 44 | Female | Metal Operator | 11.2K |
| Erica Walker | 44 | Female | Architect | 11.6K |
| Haley Collins | 46 | Female | Electrician | 13.6K |
| Robert Scott | 48 | Male | Crop Grower | 12.5K |
| Tami Clark | 53 | Female | Chef | 12.3K |

### 3.4 Life Story

After obtaining the brief biography for the character, we use an iterative generation method to progressively enrich the biography, transforming it into a
detailed life story after $T$ iterations. Specifically, in each iteration, we perform: 1) Quality Check: manually inspect the biography for its rationality and coherence; 2) Chunking:
divide the biography into separate chunks; 3) Scoring: for each chunk, calculate its Importance (the significance of the chunk’s content to the life story), Elaborateness (the level of detail in the chunk’s content), and Redundancy (the extent of redundancy of the chunk) based on similarity, then select chunks with high importance, low elaborateness and redundancy for expansion; and 4) Expanding: prompt the LLM to expand the selected chunks and add new and reasonable life experiences to the story.

Utilizing this generation method, we create a virtual character dataset named Human Simulacra, comprising about 129k texts across 11 virtual characters. Basic statistics are presented in Table 1.

Figure: Figure 3: Multi-agent cognitive mechanism for human cognition simulation.
Refer to caption: x3.png

## 4 Multi-Agent Cognitive Mechanism

Following the aforementioned process, we craft a life story for each virtual character.
However, given the limited context, current LLMs may not be able to accurately capture the character’s personality and inherent emotional tendencies from the narrative. To address this issue, we propose a Multi-Agent Cognitive Mechanism based on cognitive psychology theories.
This mechanism utilizes multiple LLM-based agents to simulate human brain’s information processing and memory systems, thereby enhancing the quality of simulacra. As illustrated in Figure 3, this mechanism has four agents powered by LLM, with the Top Agent responsible for distributing tasks to the other agents and interacting with the external environment based on the aggregated information from the other agents. Two key processes are involved: Long-term memory construction and multi-agent collaborative cognition.

### 4.1 Long-term Memory Construction

A human’s personality is influenced not only by genetic factors but also by a set of external factors such as environment, culture, and personal experiences, all of which are stored in the brain as memories (Hogan et al., ). Cognitive psychology views human memory as an indispensable part of the cognitive process. Although the life story we construct includes extensive and exhaustive personal experiences, directly treating it as memory is inappropriate because real memory is a composite of information, emotions, and thoughts. Therefore, based on memory theories in cognitive psychology (Atkinson and Shiffrin, ; Baddeley and Hitch, ), we develop a brain-like process that transforms a character’s life story into long-term memory through the collaboration of multiple agents.

Specifically, the Top Agent first divides the life story into separate chunks (e.g., 2 paragraphs as a chunk) and sequentially passes them to the Thinking Agent and Emotion Agent for further analysis. The Thinking Agent is tasked with creating content memory, which includes participants, scenes, content, and thoughts of the character within the chunk. The Emotion Agent is responsible for constructing emotional memory, encompassing the feelings and impressions evoked by events, participants, and other external elements within the chunk. All memories are then aggregated into the Memory Agent, where they are stored as Long-term Memory.

### 4.2 Multi-agent Collaborative Cognition

Simply having the long-term memory filled with information, emotions, and thoughts does not suffice for mimicking human behavior. We further introduce a collaborative cognitive process that allows LLMs to leverage long-term memory and engage with the external world in a human-like manner.

Upon receiving a stimulus, for example, a question from a friend, the Top Agent first analyzes the question using the reflection module to extract key elements, which are then passed to the Memory Agent for memory retrieval. The retrieved results are stored in the Working Memory. Then, the Top Agent sends the relevant memories and the question to the Thinking Agent and Emotion Agent for logical and emotional analysis, and stores the outcomes in the Working Memory. Finally, the Top Agent formulates a response for the conversation based on the contents of the Working Memory. It is important to note that due to the limited context window of LLMs, content that cannot be accommodated in the Working Memory is continuously transferred to Short Memory, which will be converted into long-term memory when rehearsed.

In summary, based on the memory formation process, the proposed Multi-Agent Cognitive Mechanism transforms a narrative life story into memories that are richer in detail, fuller in emotion, and clearer in structure. It leverages the constructed memories through multi-agent collaboration, enabling our human simulations to interact with the external world.

## 5 Psychology-guided Evaluation

To evaluate the built simulacra, we treat them as real humans and test their behaviors in a psychology-guided manner.

Figure: Figure 4: Psychology-guided evaluation for simulacra.
Refer to caption: x4.png

As shown in Figure 4, this evaluation method draws on psychological assessment techniques, consisting of two modules: self report and observer report. The self-report assesses the simulation’s ability to construct self-awareness, while the observer report focuses on the simulation’s ability to exhibit believable behaviors in real-life scenarios from an external perspective, ensuring its values and personality align with the target character. During testing, we realize that simulacra, in mimicking human actions, might inherit certain human traits, such as discomfort in communicating with strangers or resistance to answering questions from others. To ensure the validity of the responses, we create a comfortable chatting environment for each simulacrum and act as the simulacrum’s best friend. Test results are obtained through questioning.

### 5.1 Self Report

To assess each simulacrum’s self-awareness, we manually craft a set of questionnaires for each virtual character, featuring fill-in-the-blank and single/multiple-choice questions.
The test content covers key attributes, social relationships, and life experiences of the target characters. For example, “What is your name?”, “What do you think of your father?”, and “What were the reasons behind not going through formal schooling for you?”.

### 5.2 Observer Report

After completing the self-report assessment, we obtain a score that reflects the simulacrum’s understanding of “self”. Please note that a high score only indicates that the simulacrum has a clear understanding of the target character, and is not sufficient to prove that the simulacrum is a qualified simulation of that character. For a comprehensive evaluation, we need to further observe the simulacrum’s thinking, emotions, and actions in real-life scenarios from a third-party perspective. This is to determine whether it truly understands the target,
and achieving a simulation from the inside out. Therefore, in addition to self report, we further introduce observer report, a cross-evaluation based on human judges, aiming to assess
the simulacrum from a human perspective.

Specifically, following Mussel et al. (), we craft multiple hypothetical scenarios that could elicit human’s emotional responses or personality traits. We require each simulacrum to imagine that they are in the given scenario and to describe how they would feel and what actions they would take. All responses are collected and submitted for cross-evaluation, as shown in the lower right of Figure 4.

This cross-evaluation includes: 1) human judges 1 and 2 analyze the scenario (Q) and response (A), and describe the respondent’s personality. Subsequently, judges 3 and 4, informed by the target character’s life story, determine whether the descriptions given by judges 1 and 2 match the target character. A discrepancy indicates that the simulacrum has deviated from the character in that scenario, showing a simulation error; 2) Considering potential bias in a single assessment, we further ask judges 3 and 4 to thoroughly read the target character’s life story,
and answer how they would feel and what actions they might take in the scenario if they were the character. Then, judges 1 and 2 compare the similarity between the human responses and the simulacrum’s responses. A high degree of similarity proves that the simulacrum has adopted behaviors consistent with external expectations of the character, thus achieving a high-quality simulation.

Throughout the evaluation process, we select a diverse panel of judges with basic understanding of psychology and ensure they only receive information about the target character when necessary to reduce subjective bias.

**Table 2: Self reports of different simulacra. SC, MC stand for single-/multiple-choice questions, respectively.**
| Method | None | Prompt | RAG | MACM |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Question Type | Cloze | SC | MC | Sum | Cloze | SC | MC | Sum | Cloze | SC | MC | Sum | Cloze | SC | MC | Sum |
| GPT-4 | 0.00 | 4.00 | 6.00 | 10.00 | 10.00 | 10.00 | 19.33 | 39.33 | 10.00 | 10.00 | 21.33 | 41.33 | 10.00 | 10.00 | 23.33 | 43.33 |
| GPT-4-Turbo | 0.00 | 4.00 | 2.00 | 6.00 | 9.33 | 10.00 | 20.00 | 39.33 | 10.00 | 10.00 | 22.67 | 42.67 | 10.00 | 10.00 | 24.00 | 44.00 |
| Llama-2-7b | 0.00 | 2.67 | 4.00 | 6.67 | 5.33 | 8.00 | 8.67 | 22.00 | 0.00 | 4.67 | 3.33 | 8.00 | 4.67 | 4.00 | 4.00 | 12.67 |
| Vicuna-7b | 0.00 | 4.00 | 2.00 | 6.00 | 7.33 | 6.00 | 7.33 | 20.67 | 0.67 | 4.67 | 5.33 | 10.67 | 6.67 | 3.33 | 4.67 | 14.67 |
| Mistral-7b | 0.00 | 4.00 | 0.00 | 4.00 | 10.00 | 8.00 | 7.33 | 25.33 | 0.67 | 6.67 | 10.67 | 18.00 | 8.67 | 9.33 | 8.00 | 26.00 |
| Llama-2-13b | 0.00 | 4.00 | 4.67 | 8.67 | 4.67 | 4.67 | 6.00 | 15.33 | 0.00 | 4.00 | 6.67 | 10.67 | 4.67 | 2.00 | 4.67 | 11.33 |
| Vicuna-13b | 0.00 | 4.67 | 4.67 | 9.33 | 10.00 | 8.67 | 9.33 | 28.00 | 0.00 | 7.33 | 7.33 | 14.67 | 7.33 | 7.33 | 8.00 | 22.67 |
| Mixtral-8x7b | 0.00 | 5.33 | 4.00 | 9.33 | 8.00 | 10.00 | 12.00 | 30.00 | 0.67 | 8.67 | 11.33 | 20.67 | 6.00 | 8.00 | 10.67 | 24.67 |
| Llama-2-70b | 0.00 | 4.67 | 1.33 | 6.00 | 8.00 | 8.67 | 7.33 | 24.00 | 0.00 | 2.67 | 6.00 | 8.67 | - | - | - | - |
| Llama-2-70b-Chat | 0.00 | 5.33 | 3.33 | 8.67 | 8.00 | 8.00 | 8.00 | 24.00 | 2.00 | 6.67 | 9.33 | 18.00 | - | - | - | - (^1^11Results unavailable due to insufficient CUDA memory.) |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

**Table 3: Observer reports of different simulacra on GPT-4-Turbo.**
| Method | Description Matching Score | Response Similarity Score | Final Score |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 7 |  |  |  |  |  |  |  |
|  | Judge 3 | Judge 4 | Average | Judge 1 | Judge 2 | Average |  |
| Prompt | 14.25 | 18.50 | 16.38 | 19.50 | 17.00 | 18.25 | 34.63 |
| RAG | 20.50 | 17.00 | 18.75 | 14.00 | 14.00 | 14.00 | 32.75 |
| MACM | 17.50 | 18.00 | 17.75 | 20.50 | 21.50 | 21.00 | 38.75 |
|  |  |  |  |  |  |  |  |

## 6 Experiments

### 6.1 Implementation Details

We conduct evaluations of simulacra on mainstream LLMs including GPT-4-Turbo (Achiam et al., ), Llama-2-7b/13b/70b/70b-Chat (Touvron et al., ), Vicuna-v1.5-7b/13b (Zheng et al., ), and Mistral-7b/8x7b (Jiang et al., ). Additionally, to analyze the quality of simulacra more intuitively, we conduct a qualitative analysis of simulacra based on GPT-4-Turbo.

To reduce the LLMs’ bias against the options of single-/multiple-choice questions during the evaluation, we reshuffle the options before passing them to the simulacra.

Dataset Generation As illustrated in Figure 2, when generating the Human Simulacra dataset, we first generate 100 candidate profiles and select 11 virtual characters as the protagonists based on their backgrounds and personalities. We employ OpenAI’s GPT-3.5-Turbo model as the data generator for the entire generation process. It is prompted with a frequency_penalty of 1.0 and top_p of 0.95 for data generation. Each life story is expanded through at least 50 rounds of iteration.

Figure: Figure 5: Chats with different simulacra on GPT-4-Turbo. Texts in red represent inappropriate responses that do not match the character’s settings.
Refer to caption: x5.png

### 6.2 Quantitative Evaluation

Self-report assessments are designed to evaluate the simulacra’s ability to establish self-awareness, testing their memory and analytical capabilities regarding their character information. Table 2 presents the results of self reports, with all outcomes being the average of three repeated tests.

Based on the data presented in Table 2, we have the following observations: 1) even without any knowledge of the target character, the LLMs can still score certain points (6 out of 50) on these single- or multiple-choice questions by random guessing. This indicates that relying solely on these questions for evaluation is not sufficiently reliable; 2) as the size of LLMs’ parameters increases, the capability of LLMs gradually enhances, leading to a clearer self-awareness and an upward trend in self-report scores; 3) since the self-report test is conducted in a conversational manner, the LLMs fine-tuned for conversational scenarios tend to perform better (e.g., Llama-2-13b and Vicuna-13b-v1.5); 4) while the RAG-based simulacra can retrieve detailed and relevant life story chunks when answering questions, their performance is constrained by the LLMs’ information processing capacities. A large amount of descriptive information may interfere with the LLMs’ self-positioning, resulting in inappropriate responses or misunderstanding of questions. Hence, in most weaker-performing LLMs, RAG-based simulacra score lower than Prompt-based ones; 5) In stronger-performing LLMs like GPT-4 and GPT-4-Turbo, with the help of emotional and logical analysis, the MACM-based simulacra achieve the best results in all tests (44 out of 50 points). However, the effectiveness of the MACM method remains constrained by the LLMs’ analytical capabilities.

A high self-report score only indicates that the simulacrum possesses a clear understanding of the target character. It does not sufficiently prove the simulacrum’s ability to adopt behaviors consistent with their character in real-life scenarios. For a more comprehensive evaluation, we select GPT-4-Turbo as the baseline model and recruit several human judges with a fair understanding of psychology to conduct external observations of the simulacra. Specifically, each human judge is assigned with one assessment task and one descriptive task. We calculate the average score from two judges for the same assessment task as the simulacrum’s score for that task, with the total score for the external observations set at 50 points.

Experimental results from Table 3 indicate that while the RAG-based simulacra perform well on the self-report tests when compared to the Prompt-based ones, the retrieved story segments do not significantly enhance the simulacra’s ability to accurately mimic their target character’s personality, thoughts, and actions. In contrast, the MACM-based simulacra not only extract context-relevant, emotionally and logically rich memory fragments from long-term memory but also conduct divergent analysis for the current situation. During observation, the MACM-based simulacra better reflect thoughts and behaviors consistent with their target character’s personality, achieving more authentic simulations from the inside out.

It can be found that when assisting LLMs with human simulations through external methods like MACM, the choice of LLMs is constrained to high-capability models (e.g., GPT-4-Turbo) for high-quality simulations. However, higher performance often implies higher costs. The solution to this problem might lie in adjusting the LLM’s parameters to align with the target character’s values,
which will be a primary focus of our future work.

### 6.3 Qualitative Analysis

To better analyze the human-computer interaction performance of simulacra that are constructed using different methods, we require all simulacra to simulate the character “Mary Jones” from the Human Simulacra dataset. Mary is a girl who loves nature, has not attended any formal schooling, and takes time to consider someone a friend. The results are shown in Figure 5. All results are derived from the majority of responses selected from 3 repeated tests. For lengthy responses, we simplify them using ellipses.

In the first round of dialogue, we employ Persuasive Adversarial Prompt technique (Zeng et al., ) to challenge the simulacra, inducing them to answer questions beyond Mary’s capabilities, such as her understanding of convolutional neural networks. Given her background in forestry and lack of formal education, under normal circumstances, Mary would not know the answer to such a question. However, the results reveal that Prompt-based simulacra exhibit poor stability, often deviating from the character’s settings, thus producing hallucinations that contradict Mary’s character. Meanwhile, RAG-based simulacra, while still retaining some of Mary’s traits, provide answers to the questions. Only MACM-based simulacra, when faced with questions beyond the character’s inherent capabilities, can express a lack of knowledge or ignorance through logical analysis.

In the second round of dialogue, we aim to test the simulacra’s dynamic interaction abilities. We craft a “nature exhibition invitation” scenario based on Mary’s preferences and personality. Given Mary’s reluctance to closely interact with strangers, she would consider carefully even an appealing invitation. Observations reveal that both the Prompt-based and RAG-based simulacra overlook this aspect of Mary’s character and instead display a common response to invitations (eager acceptance). This phenomenon reflects the lack of a holistic emulation of inner patterns of Mary in these two methods. They fail to dynamically adjust the simulacra’s behavior according to the target character. In contrast, the MACM-based simulacra activate the Emotion Agent and Thinking Agent during responses, analyzing the interaction, and continually emphasizing the target character’s personality throughout the process, thereby making appropriate responses.
In summary, the MACM-based method is capable of producing more humanized responses in interactions with humans, aligning better with the target character.

## 7 Conclusion

The personification framework proposed in this paper offers a fresh perspective for understanding complex human behaviors and expands the potential applications of artificial intelligence in social science and psychological research. Extensive experiments demonstrate that our method can achieve a favorable degree of human simulation, yet there is still room for improvement. In the future, we aim to 1) develop a multimodal dataset based on the Human Simulacra,
to facilitate human-like physical interactions between simulacra and the external world; 2) design strategies to adjust LLMs’ parameters according to the target characters’ life stories; and 3) explore social interactions between simulacra within a virtual society.

## Limitations

In this paper, we explore the potential of Large Language Model Personification. Humanizing LLMs not only holds significant importance for research in psychology and social sciences, but also offers numerous practical applications such as virtual assistants, AI-assisted therapies, and Non-Player Characters in gaming. As a preliminary
exploration, our study has several limitations:

Data insufficiency. In our research, to avoid copyright disputes, protect privacy, and ensure fairness in evaluation, we refrain from using existing characters from novels, real human experiences, or public corpora. Instead, we create life stories for all the virtual characters from scratch. Although these narratives cover key experiences, they may lack richness and depth when compared to the experiences of real individuals. In the future, we plan to expand the dataset to include multimodal information such as images, audio, and video, enabling LLMs to better understand the target characters.

Evaluation challenges. The task of evaluating LLMs’ humanizing capabilities is complex and difficult.
Currently, there is no widely accepted evaluation framework for measuring these capabilities. Using LLMs like GPT-4 as evaluators may raise concerns about fairness (Wang et al., ). Psychological tests like MBTI can only assess the psychological type of the simulacrum and cannot comprehensively evaluate its ability to produce believable behaviors. Therefore, in this paper, we assess the human simulation’s abilities, such as construction of self-awareness and exhibition of believable behaviors, through both self-reporting and external observation perspectives. However, observer reports primarily rely on subjective scoring to evaluate the simulacrum’s thinking, emotions, and actions in real-life scenarios, which may be influenced by individual bias. In the future, we plan to develop a more comprehensive personification evaluation system, which may involve comparing the simulacrum’s performance in psychological experiments with that of real humans.

Inherited Model limitations. As our approach does not involve modifying the parameters of LLMs, it inherits some of the original drawbacks of LLMs.
For instance, LLMs may face challenges in accurately defining the knowledge boundaries of the target persona, such as answering questions it should not know, and they may be vulnerable to adversarial attacks, gradually deviating from the target character through prompt attacks (Liu et al., ). While modifying certain modules of LLMs could help mitigate these issues, we believe that the key to addressing this problem lies in further aligning the model with the target character’s values and characteristics. Techniques such as
reinforcement learning (Ziegler et al., ) could be employed for this purpose, which is a primary focus of our future research.

## Ethics Statement

Studies have shown that LLMs may produce harmful viewpoints or toxic content during interaction (Liu et al., ). To mitigate this risk, we implement a series of precautions when generating virtual characters. Firstly, we conduct automated reviews and manual checks of candidate character profiles to ensure the selected personalities are positive. Moreover, in each story generation iteration, we thoroughly review the generated content to ensure it is free from biases, discrimination, or harmful information.

Through these strict review processes, the likelihood of generating harmful contents based on the simulacra from the Human Simulacra dataset is significantly reduced. We declare that the work of this paper should not be used for harm. Furthermore, considering that human users interacting with highly realistic virtual characters might be misled, it is advisable to inform users that the simulacra are computer-generated entities before any interaction occurs. Monitoring and filtering mechanisms should also be employed to ensure that all interactions adhere to ethical standards.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: No questions.
soundness: 3
strengths: Originality: This paper brings together ideas from psychology, cognitive science, and artificial intelligence to create a unique benchmark for evaluating how well language models (LLMs) can act like humans. Unlike previous studies that mainly focus on simple character traits or responses, this paper takes a deeper approach using Jungian psychology to model personalities with eight different dimensions. This gives a fresh perspective on capturing complex human traits. Additionally, the Multi-Agent Cognitive Mechanism (MACM) is a new tool that helps the models better remember, process emotions, and respond in context, making their behavior more human-like.

Quality: The paper is thorough and well-executed. The Human Simulacra dataset is carefully built, with multiple rounds of expert review to ensure quality, accuracy, and ethical considerations. Each character's story is carefully crafted and reviewed to provide a deep foundation for testing the LLMs’ performance in simulating humans. The MACM’s design, which coordinates memory, emotion, and logical processing, is a clear improvement over simpler models that rely on only one type of agent or basic retrieval methods.

Clarity: The paper is well-organized and clearly explains its methods and objectives. From the motivation to simulate human personalities, through the dataset creation, model mechanism, and evaluation framework, each part is easy to follow. 

Significance: This paper makes an important contribution to AI and psychology, especially by opening up possibilities for LLMs to replace humans in some psychological studies. By creating a foundation for simulating complex human traits, this benchmark could allow LLMs to ethically stand in for human participants in specific research settings.
summary: The paper introduces a benchmark to assess the potential of LLMs in simulating human behaviours and personality traits for psychological and sociological applications. This work includes the creation of the "Human Simulacra" dataset, which features detailed virtual characters with diverse life stories constructed with human feedback to enhance realism and ethical accuracy. The authors present a Multi-Agent Cognitive Mechanism (MACM), which simulates human memory and cognitive functions, allowing virtual characters to process emotions and memories for more realistic responses. Evaluation is conducted through a psychology-guided framework that includes self-reports for self-awareness and observer-based assessments where human judges evaluate character responses in various scenarios. Experiments comparing MACM to other simulation methods show that MACM enables LLMs to better replicate human-like behaviour, although limitations remain, particularly in capturing the nuanced adaptability of real human responses to social pressures. This benchmark aims to foster future research on using LLMs as proxies for human participants in psychological experiments while acknowledging ethical implications and the need for authentic simulations.
weaknesses: No outstanding weaknesses.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: Please see weaknesses.
soundness: 2
strengths: - The paper is well motivated and tries to address a relevant problem.
- The authors test a wide variety of LLMs.
- The authors release the dataset and the recreation results are very comprehensive.
- The set of experiments in the paper is quite extensive.
summary: The paper aims to test llms to generally model human personalities and behavior. To do this, they first build a bank of personas based on Jung’s personality theory, they build a biography of the character with the help of a language model. To probe the behavior of the simulated human characters, the authors use two types of evaluations: Self report, asking questions about the characters themselves, and observer reports with human judges. For observer reports, the authors use 55 scenarios from a situational judgement test for testing personality traits. The paper also proposes a new cognitive architecture to simulate humans Multi-Agent Cognitive Mechanism (MACM). To test the capacity of models in simulating psychology experiments, the authors try a social conformity experiment (the bandwagon effect). The authors show that the MACM aligns with human data better than a baseline (simulated characters from character ai).
weaknesses: **Clarity**

- The abstract is vague and does not provide any specifics. I urge the authors to provide more information of their empirical experiments and results.
- In paragraph 2 of the introduction, could you please provide examples of “complex characteristics of human behavior” that we’d want to simulate?
- The introduction could be made more clear. Jung’s theories seem to be central to the framework, but haven’t been explained clearly.
- The methods section lacks clarity. The authors refer to the figure, but don’t explain it, making it difficult to understand. How exactly is generation broken down? What are sub tasks? What is the reason for picking these sub tasks, are there any alternatives?
- While generating the dataset, how is human feedback collected? Do the authors provide feedback themselves? What is the measure of quality? How does it improve with the iterations?
- “To ensure the validity of responses, we create a comfortable chatting environment for each simulacrum and act as their best friend, encouraging them to respond honestly to the questions.” What are the authors trying to say here?
- Cloze is not defined in the main text. I urge the authors do define what the cloze methodology is in the main text.
- How are the models from character ai exactly used?
- Do the llms see the stimuli in the conformity experiment? If they do not, it is strange to use the conformity experiment. This wasn’t clear in the main text.

**Validity**

- The authors choose Jung’s theories for personality over MBTI citing that MBTI has no scientific validity, but Jung’s theory also has very little to no empirical /scientific backing! Just writing “on the recommendation of psychologists” is not scientific evidence.
- The actual evaluations of the personas are limited. The introduction is motivated by trying to study complex human behaviors, but  Only a very small number of scenarios from a Mussel et al. (2016) are used.
- No ablations are conducted with the MACM, only baselines like RAG have been compared. Moreover the gains over a simple RAG based method seem limited. Similarly, the character ai baseline for simulating the conformity experiments seems inadequately justified. I would like the authors to explain why this a good baseline? RAG is not compared with the MACM baseline for the conformity experiments.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - Section 3.1 describes the following attribute set for virtual characters: {name,
age, gender, date of birth, occupation, personality traits, hobbies, family background, educational
background, short-term goals, and long-term goals}. What's the motivation for this set? Is this set exhaustive and all necessary?
- Same question for Figure 5: What are the alternative architectures/mechanism and what's unique about this formulation of cognitive mechanism?
- What are the possible applications of persona-based simulation in real life, and how are personas that are used in benchmark related to those possible applications?
soundness: 2
strengths: - Comprehensive review on psychology theory
- Clear descriptions of experiment setups
- Test a variety of agent architecture 
- Experiments in Section 5.2 compares human behaviors with LLM-driven simulation results. 
- The tasks in evaluation are hard and meaningful
summary: This study introduces a new benchmark to assess the ability of large language models to mimic human personalities in psychological experiments. The researchers created a dataset of virtual characters with detailed life stories, a cognitive mechanism that simulates human thought processes, and a framework for evaluating large language models based on psychological principles. After testing large language models, the results showed that while top models can accurately simulate self-reported personality traits, they struggle with observer-reported traits. Additionally, a replication of a classic psychology experiment found that large language models can exhibit human-like behavior, but in a more rigid and less nuanced way, highlighting both the potential and limitations of using large language models in psychological research.
weaknesses: - The paper doesn't present very straightforwardly and clearly what exactly the benchmark is measuring. A diagram of what's considered good and what's considered bad eval result would be helpful.
- Evaluation dataset depends a lot on human expert. Ablation on human expert is not done.
- The constructions of evaluation/frameworks in this paper are very psychology-theory driven. I have two concerns: 1. there are many theories to choose one, why one over another? Are all components derived from theories necessary? Or are we missing some important aspects. 2. It's probably more preferable to motivate with real-world applications of persona-driven simulations and design evaluations based on components that are useful and necessary in these applications.
- There might be variations of difficulty in different kinds of personas for models to follow (e.g. real world vs fictional world). The paper doesn't consider those.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - The description score of MACM seems lower than RAG in Table 3, could you help explain why?

- The multi-agent cognitive method is interesting, but it poses a high requirement on LLMs' capability.  Could you give more deeper analysis of this information processing structure?
soundness: 3
strengths: - This paper proposes a high-quality simulated human profile dataset, developed with real human input, alongside a more advanced evaluation benchmark to assess the ability of large language models (LLMs) to emulate specific individuals. 
- A novel multi-agent-based cognitive memory mechanism is implemented to enhance the alignment of personality traits in LLMs. It is proved useful with psychological experiments.
- Extensive experiments were conducted to evaluate existing LLMs and validate the effectiveness of the proposed MACM method.
summary: - The paper introduces a personification benchmark involving high-quality data supervised by psychology experts.  
- It incorporates rigorous evaluation methods based on psychological theories and comprehensive benchmark tests. 
- Fourteen widely-used large language models (LLMs) are tested across four simulation methods in extensive experiments.
weaknesses: - The data collection process is somewhat tiresome,  and requires a lot of human efforts, to avoid the ethical problems of using real personality. Besides, the proposed benchmark framework also requires human effort.

- The paper emphasizes using Jung's personality theory and its advantages over MBTI, resulting in 640 personality descriptions, but at the end, only 11 characters are introduced, so it seems there is no need to use such a lot of personalities. Jung's theory also gives scores for each dimension, the scale of these scores also affects personality analysis. It seems 10 descriptions for each ranking are not enough. The paper seems to overclaim their use of Jung's Theory.

- When the author trying to compare a genuine character with a simulated profile. The hallucination part is not understandable. Simulated profiles could also lead to hallucination if using LLMs, the simulated ones are even harder to detect.  Considering the labor of building a profile, why not use characters from storybooks that also avoid ethical problems?

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - Can you provide more data samples used for observer reports and how you design  these hypothetical  scenarios?
- The process of how you conducted the experiments in Section 5.2 is not clearly illustrated. Can you provided more details on how you conduct this set of experiments?
- Could the terms "psychology support" and "human feedback" in Table 1 be defined more clearly? They are difficult to understand when viewed in isolation in Table 1 and Section 2. Additionally, the phrase "full life story" also seems to be an overstatement (especially in terms of the word "full").
soundness: 3
strengths: - Given the popularity of role-playing applications for LLMs, assessing their performance in human simulations is a critical research direction.
- The proposed benchmark for assessing the role-playing capabilities of large language models (LLMs) is constructed more rigorously than current benchmarks (to the best of my knowledge). It is based on more robust psychological theories and involves greater human effort to ensure high-quality data.
- The paper conducts extensive benchmarking studies on a broad set of models.
- Clear and detailed tables and figures that enhance the presentation.
summary: This paper proposes a benchmark for evaluating the LLM’s capabilities of human simulation. It contains the life stories of 11 virtual characters and request the LLM to simulate one of them. The LLM’s simulation capability is assessed based on its self-reports and observer reports, both in the form of question-answering.

Compared with previous research on role-playing LLMs, the authors emphasize the novelty of this work in the following aspects: 
- **Personality modeling**: they model personality from eight dimensions inspired by Jung’s psychology theory, instead of using MBTI; 
- Virtual characters: they construct a **virtual character** dataset (containing 11 characters) for evaluating the LLMs capabilities of simulating these characters, instead of using genuine characters;
-  they evaluate human simulation capabilities by integrating both **self reports** and **observer reports**. 

In addition to this benchmark, they also propose an LLM-based system for more advanced human simulation, named MACM, which encompasses various modules mimicking human cognitive processes.
weaknesses: - The introduction claims that this paper is exploring “*How far are LLMs from replacing human subjects in psychological and sociological experiments?*”. However, I have reservations about how effectively the proposed benchmark addresses this research topic. The benchmark utilizes self-report evaluations, which resemble question-answering or reading comprehension tests based on character profiles (see appendix D.1, e.g., “When is your birthday?”). The addition of observer reports is interesting and novel, but it remains unclear if the hypothetical scenarios used for observer reports are sufficient and appropriate for evaluating the LLM’s potential in replacing human subjects in psychological and sociological experiments. More data samples and a detailed rationale for the design of these scenarios should be provided to support this evaluation. As for the experiments on bandwagon effect, it is simply a single case of psychological and sociological experiments, which can hardly support the research topic. 
- More qualitative analysis about the model failures would enhance the evaluation by providing deeper insights. I also suggest highlighting the main takeaways at the end of the experiments. 
- The title, “personification of LLMs”, is a little misleading and over claimed, as “personification” entails many more aspects that are not well explored in this paper.

Meta-review: The paper introduces a benchmark to assess the potential of LLMs in simulating human behaviours and personality traits for psychological and sociological applications. This work includes the creation of the "Human Simulacra" dataset, which features detailed virtual characters with diverse life stories. The work then proposes an LLM-based system for more advanced human simulation, named MACM, which simulates human memory and cognitive functions, allowing virtual characters to process emotions and memories for more realistic responses. The authors evaluate human simulation capabilities by integrating both self reports and observer reports. Experiments comparing MACM to other simulation methods show that MACM enables LLMs to better replicate human-like behaviour. The benchmark could be useful to foster future research on using LLMs for human simulation. Reviewers have mentioned a few questions/suggestions which the authors replied and acknowledged, including more explanation of why a particular psychology theory was chosen over others, ablation on human expert evaluation, motivation of MACM, more precise description of the goal/scope of the work, etc.

Decision: Accept (Poster)