Human Simulacra: Benchmarking the Personification of Large Language Models

Authors: Qiujie Xie, Qiming Feng, Tianqi Zhang, Qingqiu Li, Linyi Yang, Yuejie Zhang, Rui Feng, Liang He, Shang Gao, Yue Zhang
Primary area: datasets and benchmarks
Keywords: Large Language Models, Human simulation

Abstract:
Large Language Models (LLMs) are recognized as systems that closely mimic aspects of human intelligence. This capability has attracted the attention of the social science community, who see the potential in leveraging LLMs to replace human participants in experiments, thereby reducing research costs and complexity. In this paper, we introduce a benchmark for LLMs personification, including a strategy for constructing virtual characters' life stories from the ground up, a Multi-Agent Cognitive Mechanism capable of simulating human cognitive processes, and a psychology-guided evaluation method to assess human simulations from both self and observational perspectives. Experimental results demonstrate that our constructed simulacra can produce personified responses that align with their target characters.  We hope this work will serve as a benchmark in the field of human simulation, paving the way for future research.

Decision: accept (poster)
Reviewer scores: 8, 3, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730626312610

thread_type:
official_review

<thread_object>
creator: reviewer_wr7p

<content>
summary:
This paper proposes a benchmark for evaluating the LLM’s capabilities of human simulation. It contains the life stories of 11 virtual characters and request the LLM to simulate one of them. The LLM’s simulation capability is assessed based on its self-reports and observer reports, both in the form of question-answering.

Compared with previous research on role-playing LLMs, the authors emphasize the novelty of this work in the following aspects: 
- **Personality modeling**: they model personality from eight dimensions inspired by Jung’s psychology theory, instead of using MBTI; 
- Virtual characters: they construct a **virtual character** dataset (containing 11 characters) for evaluating the LLMs capabilities of simulating these characters, instead of using genuine characters;
-  they evaluate human simulation capabilities by integrating both **self reports** and **observer reports**. 

In addition to this benchmark, they also propose an LLM-based system for more advanced human simulation, named MACM, which encompasses various modules mimicking human cognitive processes.

strengths:
- Given the popularity of role-playing applications for LLMs, assessing their performance in human simulations is a critical research direction.
- The proposed benchmark for assessing the role-playing capabilities of large language models (LLMs) is constructed more rigorously than current benchmarks (to the best of my knowledge). It is based on more robust psychological theories and involves greater human effort to ensure high-quality data.
- The paper conducts extensive benchmarking studies on a broad set of models.
- Clear and detailed tables and figures that enhance the presentation.

weaknesses:
- The introduction claims that this paper is exploring “*How far are LLMs from replacing human subjects in psychological and sociological experiments?*”. However, I have reservations about how effectively the proposed benchmark addresses this research topic. The benchmark utilizes self-report evaluations, which resemble question-answering or reading comprehension tests based on character profiles (see appendix D.1, e.g., “When is your birthday?”). The addition of observer reports is interesting and novel, but it remains unclear if the hypothetical scenarios used for observer reports are sufficient and appropriate for evaluating the LLM’s potential in replacing human subjects in psychological and sociological experiments. More data samples and a detailed rationale for the design of these scenarios should be provided to support this evaluation. As for the experiments on bandwagon effect, it is simply a single case of psychological and sociological experiments, which can hardly support the research topic. 
- More qualitative analysis about the model failures would enhance the evaluation by providing deeper insights. I also suggest highlighting the main takeaways at the end of the experiments. 
- The title, “personification of LLMs”, is a little misleading and over claimed, as “personification” entails many more aspects that are not well explored in this paper.

questions:
- Can you provide more data samples used for observer reports and how you design  these hypothetical  scenarios?
- The process of how you conducted the experiments in Section 5.2 is not clearly illustrated. Can you provided more details on how you conduct this set of experiments?
- Could the terms "psychology support" and "human feedback" in Table 1 be defined more clearly? They are difficult to understand when viewed in isolation in Table 1 and Section 2. Additionally, the phrase "full life story" also seems to be an overstatement (especially in terms of the word "full").
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648817730

thread_type:
official_review

<thread_object>
creator: reviewer_yx9l

<content>
summary:
- The paper introduces a personification benchmark involving high-quality data supervised by psychology experts.  
- It incorporates rigorous evaluation methods based on psychological theories and comprehensive benchmark tests. 
- Fourteen widely-used large language models (LLMs) are tested across four simulation methods in extensive experiments.

strengths:
- This paper proposes a high-quality simulated human profile dataset, developed with real human input, alongside a more advanced evaluation benchmark to assess the ability of large language models (LLMs) to emulate specific individuals. 
- A novel multi-agent-based cognitive memory mechanism is implemented to enhance the alignment of personality traits in LLMs. It is proved useful with psychological experiments.
- Extensive experiments were conducted to evaluate existing LLMs and validate the effectiveness of the proposed MACM method.

weaknesses:
- The data collection process is somewhat tiresome,  and requires a lot of human efforts, to avoid the ethical problems of using real personality. Besides, the proposed benchmark framework also requires human effort.

- The paper emphasizes using Jung's personality theory and its advantages over MBTI, resulting in 640 personality descriptions, but at the end, only 11 characters are introduced, so it seems there is no need to use such a lot of personalities. Jung's theory also gives scores for each dimension, the scale of these scores also affects personality analysis. It seems 10 descriptions for each ranking are not enough. The paper seems to overclaim their use of Jung's Theory.

- When the author trying to compare a genuine character with a simulated profile. The hallucination part is not understandable. Simulated profiles could also lead to hallucination if using LLMs, the simulated ones are even harder to detect.  Considering the labor of building a profile, why not use characters from storybooks that also avoid ethical problems?

questions:
- The description score of MACM seems lower than RAG in Table 3, could you help explain why?

- The multi-agent cognitive method is interesting, but it poses a high requirement on LLMs' capability.  Could you give more deeper analysis of this information processing structure?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683782321

thread_type:
official_review

<thread_object>
creator: reviewer_dfn4

<content>
summary:
This study introduces a new benchmark to assess the ability of large language models to mimic human personalities in psychological experiments. The researchers created a dataset of virtual characters with detailed life stories, a cognitive mechanism that simulates human thought processes, and a framework for evaluating large language models based on psychological principles. After testing large language models, the results showed that while top models can accurately simulate self-reported personality traits, they struggle with observer-reported traits. Additionally, a replication of a classic psychology experiment found that large language models can exhibit human-like behavior, but in a more rigid and less nuanced way, highlighting both the potential and limitations of using large language models in psychological research.

strengths:
- Comprehensive review on psychology theory
- Clear descriptions of experiment setups
- Test a variety of agent architecture 
- Experiments in Section 5.2 compares human behaviors with LLM-driven simulation results. 
- The tasks in evaluation are hard and meaningful

weaknesses:
- The paper doesn't present very straightforwardly and clearly what exactly the benchmark is measuring. A diagram of what's considered good and what's considered bad eval result would be helpful.
- Evaluation dataset depends a lot on human expert. Ablation on human expert is not done.
- The constructions of evaluation/frameworks in this paper are very psychology-theory driven. I have two concerns: 1. there are many theories to choose one, why one over another? Are all components derived from theories necessary? Or are we missing some important aspects. 2. It's probably more preferable to motivate with real-world applications of persona-driven simulations and design evaluations based on components that are useful and necessary in these applications.
- There might be variations of difficulty in different kinds of personas for models to follow (e.g. real world vs fictional world). The paper doesn't consider those.

questions:
- Section 3.1 describes the following attribute set for virtual characters: {name,
age, gender, date of birth, occupation, personality traits, hobbies, family background, educational
background, short-term goals, and long-term goals}. What's the motivation for this set? Is this set exhaustive and all necessary?
- Same question for Figure 5: What are the alternative architectures/mechanism and what's unique about this formulation of cognitive mechanism?
- What are the possible applications of persona-based simulation in real life, and how are personas that are used in benchmark related to those possible applications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690591272

thread_type:
official_review

<thread_object>
creator: reviewer_qphb

<content>
summary:
The paper aims to test llms to generally model human personalities and behavior. To do this, they first build a bank of personas based on Jung’s personality theory, they build a biography of the character with the help of a language model. To probe the behavior of the simulated human characters, the authors use two types of evaluations: Self report, asking questions about the characters themselves, and observer reports with human judges. For observer reports, the authors use 55 scenarios from a situational judgement test for testing personality traits. The paper also proposes a new cognitive architecture to simulate humans Multi-Agent Cognitive Mechanism (MACM). To test the capacity of models in simulating psychology experiments, the authors try a social conformity experiment (the bandwagon effect). The authors show that the MACM aligns with human data better than a baseline (simulated characters from character ai).

strengths:
- The paper is well motivated and tries to address a relevant problem.
- The authors test a wide variety of LLMs.
- The authors release the dataset and the recreation results are very comprehensive.
- The set of experiments in the paper is quite extensive.

weaknesses:
**Clarity**

- The abstract is vague and does not provide any specifics. I urge the authors to provide more information of their empirical experiments and results.
- In paragraph 2 of the introduction, could you please provide examples of “complex characteristics of human behavior” that we’d want to simulate?
- The introduction could be made more clear. Jung’s theories seem to be central to the framework, but haven’t been explained clearly.
- The methods section lacks clarity. The authors refer to the figure, but don’t explain it, making it difficult to understand. How exactly is generation broken down? What are sub tasks? What is the reason for picking these sub tasks, are there any alternatives?
- While generating the dataset, how is human feedback collected? Do the authors provide feedback themselves? What is the measure of quality? How does it improve with the iterations?
- “To ensure the validity of responses, we create a comfortable chatting environment for each simulacrum and act as their best friend, encouraging them to respond honestly to the questions.” What are the authors trying to say here?
- Cloze is not defined in the main text. I urge the authors do define what the cloze methodology is in the main text.
- How are the models from character ai exactly used?
- Do the llms see the stimuli in the conformity experiment? If they do not, it is strange to use the conformity experiment. This wasn’t clear in the main text.

**Validity**

- The authors choose Jung’s theories for personality over MBTI citing that MBTI has no scientific validity, but Jung’s theory also has very little to no empirical /scientific backing! Just writing “on the recommendation of psychologists” is not scientific evidence.
- The actual evaluations of the personas are limited. The introduction is motivated by trying to study complex human behaviors, but  Only a very small number of scenarios from a Mussel et al. (2016) are used.
- No ablations are conducted with the MACM, only baselines like RAG have been compared. Moreover the gains over a simple RAG based method seem limited. Similarly, the character ai baseline for simulating the conformity experiments seems inadequately justified. I would like the authors to explain why this a good baseline? RAG is not compared with the MACM baseline for the conformity experiments.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730892110291

thread_type:
official_review

<thread_object>
creator: reviewer_jhzr

<content>
summary:
The paper introduces a benchmark to assess the potential of LLMs in simulating human behaviours and personality traits for psychological and sociological applications. This work includes the creation of the "Human Simulacra" dataset, which features detailed virtual characters with diverse life stories constructed with human feedback to enhance realism and ethical accuracy. The authors present a Multi-Agent Cognitive Mechanism (MACM), which simulates human memory and cognitive functions, allowing virtual characters to process emotions and memories for more realistic responses. Evaluation is conducted through a psychology-guided framework that includes self-reports for self-awareness and observer-based assessments where human judges evaluate character responses in various scenarios. Experiments comparing MACM to other simulation methods show that MACM enables LLMs to better replicate human-like behaviour, although limitations remain, particularly in capturing the nuanced adaptability of real human responses to social pressures. This benchmark aims to foster future research on using LLMs as proxies for human participants in psychological experiments while acknowledging ethical implications and the need for authentic simulations.

strengths:
Originality: This paper brings together ideas from psychology, cognitive science, and artificial intelligence to create a unique benchmark for evaluating how well language models (LLMs) can act like humans. Unlike previous studies that mainly focus on simple character traits or responses, this paper takes a deeper approach using Jungian psychology to model personalities with eight different dimensions. This gives a fresh perspective on capturing complex human traits. Additionally, the Multi-Agent Cognitive Mechanism (MACM) is a new tool that helps the models better remember, process emotions, and respond in context, making their behavior more human-like.

Quality: The paper is thorough and well-executed. The Human Simulacra dataset is carefully built, with multiple rounds of expert review to ensure quality, accuracy, and ethical considerations. Each character's story is carefully crafted and reviewed to provide a deep foundation for testing the LLMs’ performance in simulating humans. The MACM’s design, which coordinates memory, emotion, and logical processing, is a clear improvement over simpler models that rely on only one type of agent or basic retrieval methods.

Clarity: The paper is well-organized and clearly explains its methods and objectives. From the motivation to simulate human personalities, through the dataset creation, model mechanism, and evaluation framework, each part is easy to follow. 

Significance: This paper makes an important contribution to AI and psychology, especially by opening up possibilities for LLMs to replace humans in some psychological studies. By creating a foundation for simulating complex human traits, this benchmark could allow LLMs to ethically stand in for human participants in specific research settings.

weaknesses:
No outstanding weaknesses.

questions:
No questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732566670605

thread_type:
official_comment

<thread_object>
creator: area_chair_upwe

<content>
comment:
Dear Reviewers, 

The rebuttal discussion period is coming to a close and the paper currently has a mix of positive and negative reviewers. The authors have spent a lot of time responding to each concern -- can you take a look at the author responses and let them know any remaining concerns you have?
 
Best,  
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1733282407442

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses and Summary of Revisions

comment:
We sincerely thank all reviewers for their feedback, which has helped us significantly improve our work. We are encouraged by the reviewers' recognition of several key strengths:

- "_This paper makes an important contribution to AI and psychology, especially by opening up possibilities for LLMs to replace humans in some psychological studies._" (Reviewer jHzr)
- "_This paper proposes a high-quality simulated human profile dataset, developed with real human input, alongside a more advanced evaluation benchmark to assess the ability of large language models (LLMs) to emulate specific individuals._" (Reviewer YX9L)
- "_This study introduces a new benchmark to assess the ability of large language models to mimic human personalities in psychological experiments...highlighting both the potential and limitations of using large language models in psychological research._" (Reviewer Dfn4)
- "_The proposed benchmark for assessing the role-playing capabilities of large language models (LLMs) is constructed more rigorously than current benchmarks (to the best of my knowledge). It is based on more robust psychological theories and involves greater human effort to ensure high-quality data._" (Reviewer Wr7p)

Based on their constructive comments, we have made substantial improvements to strengthen our paper's contributions and address the limitations. The major enhancements include:
-  **Extended Discussion.** We added comprehensive analysis of MACM's structure (Section 4.3), the rationale for using Jung's theory (Section 6, Appendix A.5), and potential applications of LLM-based human simulation (Appendix H). 
-  **Additional Experimental Results.** We provided additional experimental results highlighting the contribution of each agent within MACM (Appendix B.3) and observations of the bandwagon effect of different simulacra (Appendix E). These results further validate the conclusions drawn in our paper.
-  **More Implementation details.** We supplemented the paper with additional details on character attribute design (Section 3.1), human involvement (Section 3, Appendix D.2.2), hypothetical scenarios (Appendix D.2.1), and the bandwagon effect replication (Section 5.2, Appendix E). 

We thank the reviewers again for their valuable input that has helped us improve the paper. We believe these additions have fully realized the potential of our research contribution to the field of LLM-based human simulation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a benchmark to assess the potential of LLMs in simulating human behaviours and personality traits for psychological and sociological applications. This work includes the creation of the "Human Simulacra" dataset, which features detailed virtual characters with diverse life stories. The work then proposes an LLM-based system for more advanced human simulation, named MACM, which simulates human memory and cognitive functions, allowing virtual characters to process emotions and memories for more realistic responses. The authors evaluate human simulation capabilities by integrating both self reports and observer reports. Experiments comparing MACM to other simulation methods show that MACM enables LLMs to better replicate human-like behaviour. The benchmark could be useful to foster future research on using LLMs for human simulation. Reviewers have mentioned a few questions/suggestions which the authors replied and acknowledged, including more explanation of why a particular psychology theory was chosen over others, ablation on human expert evaluation, motivation of MACM, more precise description of the goal/scope of the work, etc.