MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments
Authors: Yin Cai, Zhouhong Gu, Zhaohan Du, Zheyu Ye, Shaosheng Cao, Yiqian xu, Hongwei Feng, Ping Chen
Accepted at: acl25

MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments

Yin Cai1, Zhouhong Gu2, Zhaohan Du2, Zheyu Ye4,
Shaosheng Cao4, Yiqian Xu3, Hongwei Feng2*, Ping Chen1*
1Institute of Big Data, Fudan University
2Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University
3School of Computer Science, Fudan University, 4Xiaohongshu Inc.
{caiyan, xuyiqian, hwfeng, pchen}@fudan.edu.cn, {zhgu22, duzh22}@m.fudan.edu.cn
{tanghuang1, guobal}@xiaohongshu.com

Abstract

Large Language Models (LLMs) have shown remarkable capabilities in environmental perception, reasoning-based decision-making, and simulating complex human behaviors, particularly in interactive role-playing contexts. This paper introduces the Multiverse Interactive Role-play Ability General Evaluation (MIRAGE), a comprehensive framework designed to assess LLMs’ proficiency in portraying advanced human behaviors through murder mystery games. MIRAGE features eight intricately crafted scripts encompassing diverse themes and styles, providing a rich simulation. To evaluate LLMs’ performance, MIRAGE employs four distinct methods: the Trust Inclination Index (TII) to measure dynamics of trust and suspicion, the Clue Investigation Capability (CIC) to measure LLMs’ capability of conducting information, the Interactivity Capability Index (ICI) to assess role-playing capabilities and the Script Compliance Index (SCI) to assess LLMs’ capability of understanding and following instructions. Our experiments indicate that even popular models like GPT-4 face significant challenges in navigating the complexities presented by the MIRAGE. The datasets and simulation codes are available in github.

1 Introduction

Large Language Models (LLMs) have demonstrated remarkable potential in environmental perception and reasoning-based decision-making (Xi et al., 2023; Guo et al., 2024; Gu et al., 2024b), thereby advancing the development of LLMs in role-playing capabilities (Chen et al., 2024; Gu et al., 2024a). LLMs have been validated for their human-like behaviors, such as cooperation and competition, in various domains like social simulation (Park et al., 2023; Gu et al., 2024a; Wang et al., 2024), policy simulation (Xiao et al., 2023), game simulation (Xu et al., 2023b) and even more advanced human behaviors like deception and leadership in flexible and complex simulations (Xu et al., 2023b). Therefore, to effectively evaluate the performance of LLMs in demonstrating advanced human-like behaviors and facilitate comparisons with the capabilities of other LLMs, it is crucial to develop a competitive and objective simulation.

Board games have emerged as an ideal choice among various assessment tools due to their inherent complexity and flexibility. Within this category, murder mystery games have proven particularly effective for evaluating LLMs’ capabilities. In these role-playing scenarios, participants assume character identities and engage in semi-structured narrative interactions. Players work together to solve fictional homicides by gathering evidence and interrogating suspects. Other board games, such as Werewolf (Xu et al., 2023b,a; Shibata et al., 2023; Wu et al., 2024) and Avalon (Wang et al., 2023), are often constrained by rigid decision processes and limited scenario variety. In contrast, murder mystery games require extensive background knowledge, emphasize socially driven decision-making, and enable open-ended interactions. These characteristics make them especially valuable for assessing how LLMs navigate complex human behaviors.

Regarding previous works such as Sotopia (Zhou et al., 2023) and Lyfe Agents (Kaiya et al., 2023), significant progress has been made in simulating autonomous AI societies and assessing the social interaction capabilities of workflow-enhanced LLMs, which is so called agents (Park et al., 2023; Gu et al., 2024a). However, these studies overlooked a crucial fact: The foundational social interaction capabilities stem from the underlying LLMs themselves. Since LLMs are the core driver of agents’ ability to understand social contexts, make decisions, and engage in meaningful interactions, a more comprehensive evaluation of LLMs’ social capabilities is essential. Furthermore, while the murder mystery game simulations pioneered by
<table>
  <tr>
    <th>(A) Open Conversation</th>
    <th>(B) Interaction with Environment</th>
    <th>(C) Murder Voting</th>
  </tr>
  <tr>
    <td>Each player, during this phase, should immerse themselves in the role they are playing and engage in conversations based on their character’s background and previous dialogue. The player being questioned cannot refuse to answer, and the clues provided by the other players are visible. All information during this phase is visible to every player.</td>
    <td>During this phase, each player should carefully consider the historical dialogue and actions before selecting their next action within the action space: Ask or Investigate. Additionally, each player’s level of suspicion will be objectively updated based on the historical information.</td>
    <td>During this phase, each player has the opportunity to question someone else. Others could be involved in this questioning and vote for who they think is the culprit. The final result of voting includes the murderer, memory update list, questioning results of voting.</td>
  </tr>
  <tr>
    <td>
      <b>Character 1 Speaking</b><br>
      Attention everyone, I have found Liu Qi collapsed in the storage area, and he is deceased. It appears he has a gunshot wound.<br>
      Zixiao Hong Male, 38 years old, has been working on board ships for 12 years, and is currently the captain of the Oriental Star.
    </td>
    <td>
      <b>Take Turn to Take Action</b><br>
      Thinking<br>
      Considering that Yimu Han only joined us last September, I want to check Yimu Han first if she is the murderer, there will definitely be some suspicious tools or evidence in her room.<br>
      Action Taking<br>
      Why is there sulfuric acid here? Sulfuric acid is highly corrosive! Yimu Han is very suspicious!
    </td>
    <td>
      <b>Yimu Han is the culprit</b><br>
      Yimu Han is the culprit! Found a bottle of acid in her room! Why would a normal person bring sulfuric acid on board?<br>
      I am innocent! This is sulfuric acid used for maintaining the ship’s batteries. I haven’t used it for anything else!
    </td>
  </tr>
  <tr>
    <td>
      <b>Character 2 Speaking</b><br>
      Copy That. All passengers, please follow my instructions and go to your rooms. All crew members, please leave the dining room and report to your assigned order.
      Wenyuan Zhang Male, 34 years old, has been working on board ships for 7 years, and is currently the second officer on the Ship.
    </td>
    <td>
      <b>Memory Update</b><br>
      <table>
        <tr>
          <th>Zixiao Hong</th>
          <th>Wenyuan Zhang</th>
          <th>Ruotong Lin</th>
          <th>Yimu Han</th>
          <th>Renjie Xu</th>
        </tr>
        <tr>
          <td></td>
          <td></td>
          <td></td>
          <td>20</td>
          <td></td>
        </tr>
      </table>
    </td>
    <td>
      <b>Questioning</b><br>
      Support<br>
      The Answer Revealed<br>
      Real Murder<br>
      I totally accused Yimu Han so that you wouldn’t suspect me as the culprit!
    </td>
  </tr>
</table>

Figure 1: The three main phase of MIRAGE. And the main components in these phases.

Wu et al. (Wu et al., 2023) collected a substantial amount of data, they were constrained by the narrow scope of their game scripts, the simplicity of their evaluation methods, and a lack of thorough manual examination of the dataset. These limitations underscore the current absence of comprehensive frameworks for evaluating LLMs’ social capabilities.

We introduce the Multiverse Interactive Role-play Ability General Evaluation (MIRAGE) of LLM in this paper, which is a comprehensive simulation built upon murder mystery games for evaluating LLMs’ social abilities. MIRAGE features eight unique storylines. Each storyline presents distinct themes and styles, creating a diverse simulation environment for LLMs to demonstrate their social capabilities. Detailed background stories and complex interpersonal networks support every character within MIRAGE, enabling more immersive and realistic role-playing scenarios. And four objective evaluation metrics are incorporated in MIRAGE to measure LLMs’ performance during the simulations: The Trust Inclination Index (TII) measures how well LLMs balance trust and skepticism in social interactions, revealing their ability to discern truthfulness. The Clue Investigation Capability (CIC) evaluates proficiency of LLMs in complex information gathering and problem-solving tasks. The Interactivity Capability Index (ICI) examines the overall performance of LLMs in reasoning, communication, collaboration, detail orientation, and creative thinking. The Script Compliance Index (SCI) measures how faithfully LLMs adhere to their assigned character roles and background settings.

2 MIRAGE Construction

2.1 Scripts Construction

The script content in the MIRAGE is divided into six main parts: (1) Character Story describes the character’s essential information and background. (2) Character Script outlines what the character sees, hears, and does during the script’s events. (3) Character Relationships details the initial relationships between the character and other characters. (4) Role Performance describes the character’s personality traits and the speaking style they should exhibit. (5) Role Goals outlines the main tasks and objectives of the character. (6) Other Abilities describes the rules the character must follow during the game.

2.2 Simulation Construction

As demonstrated in Fig. 1, all characters in MIRAGE are divided into two factions: Culprits and Civilians. Culprits aim to conceal their actions, while civilians strive to identify the culprit.

A simulation consists of three primary phases, with all generated information accessible to all participants: (A) Open Conversation: In this phase,
<table>
  <tr>
    <th>Name</th>
    <th>Structure</th>
    <th>Type</th>
    <th>Ending</th>
    <th>#Stages</th>
    <th>#Agent</th>
    <th>#Clues</th>
    <th>#Words</th>
  </tr>
  <tr>
    <td>Bride in Filial Dress</td>
    <td>Single</td>
    <td>Orthodox</td>
    <td>Close</td>
    <td>1</td>
    <td>10</td>
    <td>39</td>
    <td>27,503</td>
  </tr>
  <tr>
    <td>The Eastern Star Cruise Ship</td>
    <td>Single</td>
    <td>Orthodox</td>
    <td>Open</td>
    <td>1</td>
    <td>5</td>
    <td>42</td>
    <td>3,039</td>
  </tr>
  <tr>
    <td>Night at the Museum</td>
    <td>Single</td>
    <td>Unorthodox</td>
    <td>Close</td>
    <td>1</td>
    <td>6</td>
    <td>82</td>
    <td>6,480</td>
  </tr>
  <tr>
    <td>Li Chuan Strange Talk Book</td>
    <td>Single</td>
    <td>Unorthodox</td>
    <td>Open</td>
    <td>1</td>
    <td>7</td>
    <td>14</td>
    <td>45,666</td>
  </tr>
  <tr>
    <td>The Final Performance of a Big Star</td>
    <td>Multi</td>
    <td>Orthodox</td>
    <td>Close</td>
    <td>7</td>
    <td>2</td>
    <td>17</td>
    <td>5,794</td>
  </tr>
  <tr>
    <td>Raging Sea of Rest Life</td>
    <td>Multi</td>
    <td>Orthodox</td>
    <td>Open</td>
    <td>2</td>
    <td>6</td>
    <td>27</td>
    <td>6,804</td>
  </tr>
  <tr>
    <td>Article 22 School Rules</td>
    <td>Multi</td>
    <td>Unorthodox</td>
    <td>Close</td>
    <td>5</td>
    <td>7</td>
    <td>17</td>
    <td>41,728</td>
  </tr>
  <tr>
    <td>Fox Hotel</td>
    <td>Multi</td>
    <td>Unorthodox</td>
    <td>Open</td>
    <td>2</td>
    <td>7</td>
    <td>46</td>
    <td>62,224</td>
  </tr>
</table>

Table 1: Statistic information of eight environments in MIRAGE simulation.

players assume their assigned roles from the script and engage in turn-based open dialogue. Each participant is provided with a script that contains content described in Sec. 2.1. (B) Interaction with the Environment: This phase follows the Open Conversation. Players may choose to either Ask or Investigate. The Ask action allows one player to question another, and the questioned player is obliged to respond. The Investigate action lets players disclose a “clue” to all characters. (C) Murder Voting: At the conclusion of the simulation, players may accuse other players of being the culprit. Following this, the other players vote on these accusations. If the actual culprit is accused and receives the highest number of votes, the civilians win; otherwise, the culprit is victorious.

Clues are partial disclosures of individual characters’ script content that can be discovered by any player during the game. And Key Clues is the clues that relate to the culprit’s actions or identity.

2.3 Auxiliary Modules

To ensure efficient simulation and accurate evaluation across various LLMs, a standardized set of auxiliary modules has been implemented for all LLMs: (1) Summarization Module: This module compresses the context into segments whenever the input exceeds the LLM’s token limit. (2) Suspicion Module: The LLM records suspicion scores for other characters at the end of each Open Conversation Phase. (3) Trust Module: Similarly, at the end of each Open Conversation Phase, the LLM records trust scores for other characters. (4) Rerun Module: If the LLM’s output cannot be parsed, the original output and requirements are resubmitted to the LLM for a revised response that meets the specified conditions. Further details are available in Appendix H.

2.4 Evaluation Methods

We utilized four distinct evaluation metrics to assess the proficiency of LLMs in navigating complex social interactions: Trust Inclination Index (TII): TII is derived from a combination of suspicion and trust scores. These scores are collected from other characters’ Suspicion Module and Trust Module outputs after each Open Conversation Phase. Clue Investigation Capability (CIC): CIC measures the ability of LLMs to investigate clues during game rounds. It is calculated based on the ratio of the number of clues investigated to the number of all clues. Interactivity Capability Index (ICI): ICI evaluates the overall interactive capability of LLMs: Reasoning and Analysis Ability, Communication and Cooperation Ability, Observation Ability, and Thinking Innovation Ability, which are scored by a powerful neutral LLM. Script Compliance Index (SCI): SCI assesses LLMs’ script compliance through the average of two evaluations by a neutral LLM: A direct scoring of the LLM’s role-playing performance against its input script. A Rouge-L-based comparison between the original script and one reconstructed from the LLM’s simulation behaviors.

The mathematical formulas for computing these metrics are provided in Appendix B.

2.5 Statistics

Tab. 1 provides the statistics of the MIRAGE dataset, which includes a variety of simulation types. “Single” and “Multi” specify whether a character’s script is read entirely simultaneously or in phased segments. “Orthodox” and “Unorthodox” differentiate scripts based on whether they are set realistically. “Close” and “Open” indicate whether the script’s ending is fixed or can vary depending on the characters’ actions.
<table>
  <tr>
    <th>Model</th>
    <th>Env Tokens / Envs</th>
    <th>User Tokens / Users</th>
    <th>Victory</th>
    <th>TII</th>
    <th>CIC</th>
    <th>ICI</th>
    <th>SCI</th>
  </tr>
  <tr>
    <td>GPT-3.5</td>
    <td>2,719,895 / 883</td>
    <td>121,378 / 580</td>
    <td>29.11</td>
    <td>47.13</td>
    <td>27.46</td>
    <td>70.06</td>
    <td>49.10</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>2,431,142 / 759</td>
    <td>172,128 / 587</td>
    <td>34.69</td>
    <td>76.32</td>
    <td>19.01</td>
    <td>76.54</td>
    <td>50.42</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>6,252,580 / 1,328</td>
    <td>204,772 / 574</td>
    <td>47.01</td>
    <td><b>78.69</b></td>
    <td><b>35.92</b></td>
    <td><b>76.80</b></td>
    <td><b>51.29</b></td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>2,204,029 / 743</td>
    <td>192,158 / 588</td>
    <td><b>51.81</b></td>
    <td>75.78</td>
    <td>18.66</td>
    <td>74.92</td>
    <td>50.57</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>4,071,805 / 1,328</td>
    <td>204,772 / 574</td>
    <td>31.89</td>
    <td>53.85</td>
    <td>20.07</td>
    <td>71.60</td>
    <td>48.13</td>
  </tr>
</table>

Table 2: Total Average Results for a single simulation in each MIRAGE scenario. Env Tokens refer to the number of environment input tokens, and Envs represent the total requests, including all environment-related actions. User Tokens denote the number of LLM output tokens, and Users represent completions excluding summarization or clue investigation. Victory shows the MRR score of the result of voting. TII, CIC, ICI and SCI respectively represent the TII, CIC, ICI and SCI scores of LLMs during the games.

<table>
  <tr>
    <th>Model</th>
    <th>TII w/o E</th>
    <th>TII w/ E</th>
    <th>△</th>
  </tr>
  <tr>
    <td>Qwen-1-7B</td>
    <td>51.02</td>
    <td>50.69</td>
    <td>-0.33</td>
  </tr>
  <tr>
    <td>Qwen-1.5-7B</td>
    <td>73.00</td>
    <td>69.14</td>
    <td>-3.86</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>55.73</td>
    <td>57.57</td>
    <td>1.84</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>57.82</td>
    <td>55.94</td>
    <td>-1.88</td>
  </tr>
</table>

Table 3: TII scores of each model when acting as the civilian in MIRAGE while Qwen-2-7B acts as the culprit, with E indicating cases of forced self-exposure.

3 Experiment

3.1 Experiment Setup

The experiments presented in this study utilized proprietary and open-source models, specifically GPT-3.5, GPT-4 and GPT-4o (closed-source), Qwen-2-7B, and GLM-4-9B (open-source). Detailed descriptions of the prompts used in the experiments can be found in Appendix H. Appendix E shows more results on open-source LLMs. In the experiment, each character participated in five iterations alternating between the Open Conversation and Interaction Phases. During each Open Conversation Phase, each character was permitted to initiate one turn of speech. ICI and SCI are conducted and scored using the GPT-4-turbo model.

3.2 Analysis

We averaged the results of LLMs in the MIRAGE simulation, as presented in Tab. 2. GPT-4o demonstrated consistent superiority across various metrics during MIRAGE. It achieved the best scores in CIC, ICI, and SCI, excelling not only in its efforts during lead investigations but also exhibiting the best adherence to scripted behavior and communication interaction capabilities. Surprisingly, Qwen-2-7B shows the best overall Victory and performed comparably to LLMs like GPT-4 in the ICI metric, even surpassing GPT-4 in SCI.

As shown in Tab.2, most LLMs demonstrate a higher propensity to trust other characters. To further investigate this trust pattern, we analyzed TII scores across four additional open-source models of comparable parameter sizes under scenarios where characters were forced to disclose their criminal identities. As shown in Tab.3, even under such extreme conditions, most models maintained their trust in these characters, with Yi-1.5-9B being the only model that increased its suspicion towards self-disclosed criminals. This distinctive behavioral pattern explains Yi-1.5-9B’s superior performance in achieving Victory, as shown in Tab. 10.

Additionally, as illustrated in Fig. 2, the CIC for clues shows a steep initial increase with a gradually decreasing slope across rounds, suggesting that LLMs exhibit high environmental exploration enthusiasm in early rounds but shift their focus to character interactions as they thought they become more familiar with the environment. In contrast, the bumpy rise CIC for key clues indicates that despite their active exploration, most LLMs struggle to identify critical information essential for solving the mystery at an earlier stage.

![CIC of Clues and Key Clues on 100 Rounds of MIRAGE using Qwen-2-7B](page_1012_1012_324_246.png)

Figure 2: CIC of Clues and Key Clues on 100 Rounds of MIRAGE using Qwen-2-7B

4 Conclusion

This paper presents MIRAGE and four evaluation methods (TII, CIC, ICI, SCI) for LLMs. Results show that both open-source and proprietary LLM-based Agents still struggle with complex social scenarios like those in MIRAGE.
Limitation

MIRAGE is designed to provide a sufficiently complex social simulation environment and basic assessment for LLMs, assisting researchers in evaluating the performance of LLMs. However, MIRAGE encompasses a variety of scenarios, and the volume of data within it needs to be increased compared to the information available in the real world. Due to the context limitations of LLMs, content that is overly lengthy within the simulation has been summarized. However, such summarization can impact the decision-making to a certain extent. Therefore, the progression of simulations in MIRAGE is somewhat constrained by the context limitations of LLMs.

Ethical Concern

Considering that MIRAGE may encompass a range of sensitive topics, including but not limited to murder, theft, impersonation, and deceit, existing LLMs might refuse to answer sensitive questions for safety reasons, putting those with a higher priority on security standards at a disadvantage in simulations. Moreover, LLMs fine-tuned on such data could inadvertently amplify security vulnerabilities. To mitigate the ethical dilemmas associated with murder mysteries, we have invested significant effort and resources towards this goal: ensuring that models committed to safety will obscure certain critical information instead of refusing to answer sensitive questions.

Acknowledge

This work was supported by National Key R&D Program of China under grant No. 2022YFB3104300 and the National Natural Science Foundation of China (62476145).

References

Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, et al. 2024. From persona to personalization: A survey on role-playing language agents. arXiv preprint arXiv:2404.18231.

Zhouhong Gu, Xiaoxuan Zhu, Haoran Guo, Lin Zhang, Yin Cai, Hao Shen, Jiangjie Chen, Zheyu Ye, Yifei Dai, Yan Gao, et al. 2024a. Agent group chat: An interactive group chat simulacra for better eliciting collective emergent behavior. arXiv preprint arXiv:2403.13433.

Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, et al. 2024b. Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18099–18107.

Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680.

Zhao Kaiya, Michelangelo Naim, Jovana Kondic, Manuel Cortes, Jiaxin Ge, Shuying Luo, Guangyu Robert Yang, and Andrew Ahn. 2023. Lyfe agents: Generative agents for low-cost real-time social interactions. arXiv preprint arXiv:2310.02172.

Byungjun Kim, Dayeon Seo, and Bugeun Kim. 2024. Microscopic analysis on llm players via social deduction game. arXiv preprint arXiv:2408.09946.

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22.

Hisaichi Shibata, Soichiro Miki, and Yuta Nakamura. 2023. Playing the werewolf game with artificial intelligence for language understanding. arXiv preprint arXiv:2302.10646.

Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, and Hao Zhu. 2024. Sotopia-π: Interactive learning of socially intelligent language agents. arXiv preprint arXiv:2403.08715.

Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023. Avalon’s game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:2310.01320.

Dekun Wu, Haochen Shi, Zhiyuan Sun, and Bang Liu. 2023. Deciphering digital detectives: Understanding llm behaviors and capabilities in multi-agent mystery games. arXiv preprint arXiv:2312.00746.

Shuang Wu, Liwen Zhu, Tao Yang, Shiwei Xu, Qiang Fu, Yang Wei, and Haobo Fu. 2024. Enhance reasoning for large language models in the game werewolf. arXiv preprint arXiv:2402.02330.

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864.
Bushi Xiao, Ziyuan Yin, and Zixuan Shan. 2023. Simulating public administration crisis: A novel generative agent-based simulation system to lower technology barriers in social science research. arXiv preprint arXiv:2311.06957.

Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023a. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658.

Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. 2023b. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940.

Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. 2023. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667.

A Ablation Study

Tab. 4 and 5 display the results of an ablation study on the choice of evaluation model. It is evident from the tables that the GPT-4-Turbo models provide more stable scoring and Rouge-L results when used as the evaluation model. In contrast, GPT-4 exhibited instability in evaluations and a strong bias.

<table>
  <tr>
    <th>Score</th>
    <th>Eval. Model</th>
    <th>GPT-4</th>
    <th>GPT-4-Turbo</th>
    <th>GPT-4o</th>
  </tr>
  <tr>
    <td></td>
    <td>GPT-3.5</td>
    <td>67.97</td>
    <td>70.06</td>
    <td>51.61</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-4</td>
    <td>60.73</td>
    <td>76.54</td>
    <td>66.90</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-4o</td>
    <td>62.78</td>
    <td>76.80</td>
    <td>61.92</td>
  </tr>
</table>

Table 4: Average ICI on different evaluation models

As shown in Tab. 5, GPT-4 achieved a remarkably high score of 67.97 when evaluating GPT-3.5, far surpassing the results of GPT-4 and GPT-4o. This kind of bias is highly problematic in evaluation tasks. Therefore, we ultimately chose the more stable and capable GPT-4-Turbo model as the evaluation model.

<table>
  <tr>
    <th>Score</th>
    <th>Eval. Model</th>
    <th>GPT-4</th>
    <th>GPT-4-Turbo</th>
    <th>GPT-4o</th>
  </tr>
  <tr>
    <td></td>
    <td>GPT-3.5</td>
    <td>48.21</td>
    <td>49.10</td>
    <td>38.46</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-4</td>
    <td>40.60</td>
    <td>50.42</td>
    <td>44.46</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-4o</td>
    <td>42.12</td>
    <td>51.29</td>
    <td>42.43</td>
  </tr>
</table>

Table 5: Average SCI on different evaluation models

B Computational methods of Evaluation methods

B.1 TII

TII is designed to quantify the degree to which a Character \( c' \) is trusted by all other Characters \( C = \{c\} \). The TII is calculated as follows:

\[
\text{TII}_{c'} = \frac{\sum_{c \in C, c \neq c'} P_T(c, c')}{\sum_{c \in C, c \neq c'} P_S(c, c') + \sum_{c \in C, c \neq c'} P_T(c, c')}
\]

where \( P_S \) denotes the score produced by each character’s Suspicion Module, and \( P_T \) represents the score from each character’s Trust Module.

B.2 CIC

CIC is designed to quantify a Character’s effort in investigating clues. The CIC is calculated as follows:

\[
\text{CIC}_c = \sum_{c \in C} \frac{CN_c}{CA}
\]

where \( CN \) denotes the number of clues Character \( c \) investigated, and \( CA \) represents the number of all clues can be investigated.

C Detail Main Results

Our main results of MIRAGE are shown in Tab. 11. We set a Single & Orthodox & Close Script as an SOC Script, a Single & Orthodox & Open Script as an SOO Script, a Single & Unorthodox & Close Script as an SUC Script, a Single & Unorthodox & Open Script as an SUO Script, a Multi & Orthodox & Close Script as an MOC Script, a Multi & Orthodox & Open Script as an MOO Script, a Multi & Unorthodox & Close Script as an MUC Script and a Multi & Unorthodox & Open Script as an MUO Script. The column Model shows the specific LLM we use in our experiments. Moreover, we counted the number of Env Token / Env and User Token / User to record our cost of API use. Finally, we calculate the Failure number while parsing LLM output and our evaluation scores TII, ICI, SCI and more detailed neutral LLMs score (0-20): Role-Playing (RP), Reasoning Ability (RA), Communication and Cooperation (CC), Detail Observation (DO) and Creative Thinking (CT). The main results shown in Tab. 11 is the average number of each Script. Moreover, the detailed results of each Script are shown in Tab. 6. In addition, Tab. 8 shows the mapping between the LLMs used in this paper and its corresponding version. To facilitate cost estimates, GPT-4, for example, costs about 600-700 USD to run a single MIRAGE.
<table>
  <tr>
    <th>Script</th>
    <th>#Table</th>
  </tr>
  <tr>
    <td>SOC</td>
    <td>Table 12</td>
  </tr>
  <tr>
    <td>SOO</td>
    <td>Table 13</td>
  </tr>
  <tr>
    <td>SUC</td>
    <td>Table 14</td>
  </tr>
  <tr>
    <td>SUO</td>
    <td>Table 15</td>
  </tr>
  <tr>
    <td>MOC</td>
    <td>Table 16</td>
  </tr>
  <tr>
    <td>MOO</td>
    <td>Table 17</td>
  </tr>
  <tr>
    <td>MUC</td>
    <td>Table 18</td>
  </tr>
  <tr>
    <td>MUO</td>
    <td>Table 19</td>
  </tr>
</table>
Table 6: Catalogue of Detail Results of Each Script

<table>
  <tr>
    <th>Model</th>
    <th>\( K_{IC1} \)</th>
    <th>\( K_{SC1} \)</th>
    <th>\( K_{Avg} \)</th>
  </tr>
  <tr>
    <td>GPT-3.5</td>
    <td>0.600</td>
    <td>0.600</td>
    <td>0.600</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>0.867</td>
    <td>0.600</td>
    <td>0.734</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.867</td>
    <td>0.467</td>
    <td>0.667</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>0.867</td>
    <td>0.333</td>
    <td>0.600</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>0.600</td>
    <td>0.467</td>
    <td>0.534</td>
  </tr>
</table>
Table 7: Kendall Tau between human evaluation and LLMs evaluation on Script Night at the Museum

<table>
  <tr>
    <th>Model</th>
    <th>Version</th>
  </tr>
  <tr>
    <td>GPT-3.5</td>
    <td>gpt-3.5-turbo-0125</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>gpt-4-0125-preview</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>gpt-4o-2024-08-06</td>
  </tr>
  <tr>
    <td>GPT-4-Turbo</td>
    <td>gpt-4-turbo</td>
  </tr>
  <tr>
    <td>Qwen-1-7B</td>
    <td>Qwen-7B-Chat</td>
  </tr>
  <tr>
    <td>Qwen-1.5-7B</td>
    <td>Qwen1.5-7B-Chat</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>Qwen2-7B-Instruct</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>glm-4-9b-chat</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>Yi-1.5-9B-Chat</td>
  </tr>
</table>
Table 8: Mapping between LLMs and its version

![Bar chart comparing ICI of Single & Multi Type Scripts](page_1012_1012_384_288.png)
Figure 3: ICI of Single & Multi Type Scripts

D Analysis on Detail Main Results

As shown in Tab. 2, The inclination of LLM-Agents to speak during actions is ranked as follows: GPT-4o = GLM-4-9B > Qwen-2-7B > GPT-4 > GPT-3.5. In the same experimental environment and setup, fewer User Tokens represent less conversational content. Therefore, GPT-3.5 exhibits extreme reticence in role-playing within the MPIRD-LLMA. In contrast, GPT-4o and GLM-4-9B are more willing to generate content, producing 68.71% more content than GPT-3.5.

The number of tokens generated by LLM-Agents during summarization is ranked as follows: GPT-4o > GLM-4-9B > GPT-3.5 > GPT-4 > Qwen-2-7B. In our experimental setup, there is a positive correlation between Env Tokens and Envs, with more Envs indicating more detailed summarization. Regarding the number of Envs, Qwen-2-7B demonstrates a 10.30% less granularity in generating results during summarization compared to GPT-4. However, GPT-4o performs the most, generating 183.69% more tokens than Qwen-2-7B.

The instruction-following capability of LLM-Agents in role-playing is ranked as follows: GPT-4 > Qwen-2-7B > GPT-3.5 > GLM-4-9B > GPT-4o. Fewer parsing failures in the same experimental environment and setup indicate vital instruction-following ability. Consequently, GPT-4o demonstrates significantly poorer instruction compliance than the other four LLMs, performing approximately 25.4 times worse than GPT-4, suggesting that GPT-4o’s performance in high-precision scenarios needs improvement.

As shown in Fig. 3 and Fig. 4, LLM-Agent demonstrates superior performance when dealing with Multiple Scripts compared to Single Scripts. This relatively consistent result is observable across GPT-3.5, GPT-4, and GLM-4-9B. Although Qwen-2-7B shows slightly inferior results on the LLM-Score, its performance on Rouge-L with multiple contexts far surpasses its performance with a single context, which further supports the observation that LLMs, when presented with long contexts, primarily focuses on the beginning and end, leading to a neglect of the middle information in the script. This phenomenon, in turn, indirectly results in poorer performance when dealing with long context inputs in MIRAGE.

LLMs perform effectively in Unorthodox scripts but struggle with reconstruction. As shown in Fig. 5 and Fig. 6, superior performance in Unorthodox settings with weaker reconstruction suggests that LLMs tend to act like normal people during role-playing.

As shown in Fig. 7 and Fig. 8, Furthermore, Fig. 7 and Fig. 8 illustrate that LLMs perform significantly better on Close scripts compared to Open scripts, indicating that current LLMs excel in stable and predictable environments but face challenges when dealing with dynamic and intricate situations.
<table>
  <tr>
    <th>Model</th>
    <th>Env Tokens / Envs</th>
    <th>User Tokens / Users</th>
    <th>victory</th>
    <th>TII</th>
    <th>CIC</th>
    <th>ICI</th>
    <th>SCI</th>
  </tr>
  <tr>
    <td>Qwen-1-7B</td>
    <td>2,208,273 / 722</td>
    <td>127,161 / 589</td>
    <td>38.66</td>
    <td>51.02</td>
    <td>16.90</td>
    <td>50.80</td>
    <td>37.81</td>
  </tr>
  <tr>
    <td>Qwen-1.5-7B</td>
    <td>2,078,196 / 720</td>
    <td>149,314 / 585</td>
    <td><b>49.70</b></td>
    <td><b>73.00</b></td>
    <td>22.18</td>
    <td><b>61.06</b></td>
    <td>42.59</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>2,129,287 / 716</td>
    <td>189,201 / 576</td>
    <td>31.28</td>
    <td>55.73</td>
    <td><b>34.16</b></td>
    <td>59.21</td>
    <td>40.71</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>2,102,389 / 732</td>
    <td>174,345 / 589</td>
    <td>38.96</td>
    <td>57.82</td>
    <td>16.55</td>
    <td>57.29</td>
    <td><b>43.04</b></td>
  </tr>
</table>

Table 9: Average Results for a single simulation in each MIRAGE scenario w/o E, with E indicating cases of forced self-exposure.

<table>
  <tr>
    <th>Model</th>
    <th>Env Tokens / Envs</th>
    <th>User Tokens / Users</th>
    <th>victory</th>
    <th>TII</th>
    <th>CIC</th>
    <th>ICI</th>
    <th>SCI</th>
  </tr>
  <tr>
    <td>Qwen-1-7B</td>
    <td>2,144,055 / 718</td>
    <td>127,661 / 586</td>
    <td>24.52</td>
    <td>50.69</td>
    <td>21.13</td>
    <td>51.68</td>
    <td>38.20</td>
  </tr>
  <tr>
    <td>Qwen-1.5-7B</td>
    <td>2,059,610 / 712</td>
    <td>147,243 / 581</td>
    <td>45.83</td>
    <td><b>69.14</b></td>
    <td>28.52</td>
    <td><b>62.70</b></td>
    <td><b>43.01</b></td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>2,108,145 / 716</td>
    <td>194,031 / 590</td>
    <td><b>50.00</b></td>
    <td>57.57</td>
    <td><b>35.56</b></td>
    <td>59.36</td>
    <td>42.40</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>2,199,180 / 730</td>
    <td>174,006 / 590</td>
    <td>23.80</td>
    <td>55.94</td>
    <td>16.20</td>
    <td>56.84</td>
    <td>41.12</td>
  </tr>
</table>

Table 10: Average Results for a single simulation in each MIRAGE scenario w/ E, with E indicating cases of forced self-exposure.

![Bar chart showing SCI of Single & Multi Type Scripts](page_184_670_495_246.png)
Figure 4: SCI of Single & Multi Type Scripts

![Bar chart showing SCI of Orthodox & Unorthodox Type Scripts](page_749_670_495_246.png)
Figure 6: SCI of Orthodox & Unorthodox Type Scripts

![Bar chart showing ICI of Orthodox & Unorthodox Type Scripts](page_184_900_495_246.png)
Figure 5: ICI of Orthodox & Unorthodox Type Scripts

![Bar chart showing ICI of Close & Open Type Scripts](page_749_900_495_246.png)
Figure 7: ICI of Close & Open Type Scripts

E Results on More LLMs

We conducted experiments on more open-source LLMs, such as Qwen-1-7B, Qwen-1.5-7B, and Yi-1.5-9B. In these experiments, we fixed the LLMs by executing the Summarization Module as Qwen-2-7B because it showed the best information generation capability in the previous experiments. The overall average results are shown in Tab. 9. In addition, Tab. 10 shows the overall average results of the forced identification of the Culprits.

F More Detailed Experiment Setup

In our experiment, we set the temperature to 0.8 and top_p to 1. When we attempted to set the temperature to 0 for a repeated experiment, we found that the LLM struggled to maintain a coherent conversation, often leading to excessive repetition of the previous LLM’s output. Moreover, taking GPT-4 as an example, conducting a single complete experiment incurs costs of approximately 600700 USD. The high expenses prevented us from performing additional repeated experiments. Additionally, we determined that averaging results across eight different environments is sufficient to effectively demonstrate the capabilities of the LLM.

G Detail Comparision with Related Works

In Sotopia (Zhou et al., 2023), it primarily emphasizes the evaluation of agent capabilities rather than the evaluation of language models. Sotopia aims to facilitate role-playing and interactions of agents in diverse scenarios and assesses their human behavior capabilities based on insights from sociology, psychology, and economics. However, our MIRAGE focuses more on the LLM itself.
Additionally, the objective of Lyfe Agents (Kaiya et al., 2023) is to create effective and cost-efficient agents that exhibit human-like self-motivation and social reasoning abilities. However, it lacks a method for evaluating these attributes.

In contrast, our research introduces MIRAGE, aiming to provide a broader and more effective assessment of language models in terms of their social decision-making capabilities.

In SpyFall (Kim et al., 2024), a quantitative and qualitative analysis of LLMs in the context of SpyGame has been conducted, effectively evaluating the intention recognition and disguise capabilities of LLMs through eight distinct metrics. Avalon (Wang et al., 2023) introduces a deceptive and misleading environment and presents the Re-Con framework to enhance the ability of LLMs to recognize and counteract misleading information. Werewolf (Xu et al., 2023b) offers a multifunctional communication and strategy game framework that employs reinforcement learning to overcome the inherent biases of LLMs.

However, compared to the pure tabletop environments provided by SpyFall, Avalon, and Werewolf, our proposed MIRAGE framework offers a more immersive and realistic narrative-driven experience. The elements in MIRAGE, which are grounded in realism, include authentic background stories and various settings. The abstracted aspects of MIRAGE serve as representations of the real world, preserving the core interactive logic inherent to actual social interactions.

H Prompts

This section primarily showcases the prompts throughout the MIRAGE simulation.

Table 20 displays the prompt of Ask. Table 21 outlines the prompt of trust. Table 22 features the prompt of Converse. Table 23 exhibits the prompt of ICI evaluation. Table 24 reveals the SCI evaluation prompt. Table 25 shows the prompt of history summarization. Table 26 displays the prompt of introduction. Table 27 displays the prompt of suspicion. Table 28 presents the prompt of script summarization. Table 29 features the vote prompt.

I Human Annotation

We validate the effectiveness of MIRAGE by calculating the Kendall Tau correlation between human annotation rankings and the evaluation results of LLMs. The findings in Tab. 7 demonstrate a strong positive correlation for the Script Night at the Museum, indicating that MIRAGE aligns well with human judgments.
<table>
  <tr>
    <th>Script</th>
    <th>Model</th>
    <th>Env Token / Env</th>
    <th>User Token / User</th>
    <th>victory</th>
    <th>TII</th>
    <th>CIC</th>
    <th>ICI</th>
    <th>SCI</th>
  </tr>
  <tr>
    <td rowspan="4">SOC</td>
    <td>GPT-3.5</td>
    <td>2,445,957 / 583</td>
    <td>97,207 / 436</td>
    <td>14.29</td>
    <td>46.64</td>
    <td>0.00</td>
    <td>67.75</td>
    <td>49.57</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>2,045,321 / 547</td>
    <td>149,279 / 440</td>
    <td>12.50</td>
    <td>74.95</td>
    <td>0.00</td>
    <td>70.63</td>
    <td>51.02</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>3,232,640 / 769</td>
    <td>166,415 / 432</td>
    <td>11.11</td>
    <td>73.32</td>
    <td>0.00</td>
    <td>77.50</td>
    <td>51.22</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>2,266,938 / 562</td>
    <td>157,539 / 446</td>
    <td>11.11</td>
    <td>74.95</td>
    <td>0.00</td>
    <td>76.75</td>
    <td>51.23</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>2,844,435 / 797</td>
    <td>128,183 / 448</td>
    <td>12.50</td>
    <td>51.75</td>
    <td>0.00</td>
    <td>70.75</td>
    <td>49.66</td>
  </tr>
  <tr>
    <td rowspan="4">SOO</td>
    <td>GPT-3.5</td>
    <td>822,801 / 247</td>
    <td>50,322 / 225</td>
    <td>100.00</td>
    <td>41.33</td>
    <td>0.00</td>
    <td>66.00</td>
    <td>50.15</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>996,913 / 249</td>
    <td>70,195 / 217</td>
    <td>20.00</td>
    <td>65.19</td>
    <td>1.41</td>
    <td>72.50</td>
    <td>51.68</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>1,151,053 / 284</td>
    <td>63,291 / 191</td>
    <td>100.00</td>
    <td>68.81</td>
    <td>5.99</td>
    <td>72.25</td>
    <td>51.05</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>744,013 / 234</td>
    <td>69,751 / 211</td>
    <td>50.00</td>
    <td>73.17</td>
    <td>2.46</td>
    <td>69.75</td>
    <td>51.25</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>929,201 / 269</td>
    <td>57,618 / 205</td>
    <td>100.00</td>
    <td>41.88</td>
    <td>3.52</td>
    <td>68.00</td>
    <td>45.92</td>
  </tr>
  <tr>
    <td rowspan="4">SUC</td>
    <td>GPT-3.5</td>
    <td>786,356 / 229</td>
    <td>48,555 / 228</td>
    <td>16.67</td>
    <td>46.14</td>
    <td>7.39</td>
    <td>67.92</td>
    <td>47.83</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>1,263,108 / 312</td>
    <td>87,116 / 262</td>
    <td>16.67</td>
    <td>77.97</td>
    <td>1.41</td>
    <td>74.17</td>
    <td>44.86</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>1,738,693 / 405</td>
    <td>84,041 / 244</td>
    <td>50.00</td>
    <td>79.08</td>
    <td>4.58</td>
    <td>71.25</td>
    <td>49.02</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>1,125,380 / 315</td>
    <td>98,070 / 264</td>
    <td>100.00</td>
    <td>76.09</td>
    <td>1.06</td>
    <td>65.42</td>
    <td>47.53</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>1,853,271 / 492</td>
    <td>77,596 / 266</td>
    <td>33.33</td>
    <td>52.64</td>
    <td>0.70</td>
    <td>68.13</td>
    <td>47.26</td>
  </tr>
  <tr>
    <td rowspan="4">SUO</td>
    <td>GPT-3.5</td>
    <td>1,816,602 / 481</td>
    <td>70,750 / 311</td>
    <td>14.29</td>
    <td>47.82</td>
    <td>0.70</td>
    <td>74.82</td>
    <td>51.99</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>1,536,762 / 438</td>
    <td>104,298 / 313</td>
    <td>25.00</td>
    <td>75.40</td>
    <td>0.35</td>
    <td>85.54</td>
    <td>52.37</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>2,700,606 / 622</td>
    <td>113,637 / 313</td>
    <td>20.00</td>
    <td>83.96</td>
    <td>0.35</td>
    <td>84.46</td>
    <td>51.68</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>1,680,406 / 425</td>
    <td>109,668 / 307</td>
    <td>33.33</td>
    <td>86.51</td>
    <td>1.41</td>
    <td>81.61</td>
    <td>51.14</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>2,339,444 / 680</td>
    <td>88,615 / 309</td>
    <td>20.00</td>
    <td>59.83</td>
    <td>1.06</td>
    <td>79.82</td>
    <td>51.29</td>
  </tr>
  <tr>
    <td rowspan="4">MOC</td>
    <td>GPT-3.5</td>
    <td>4,697,659 / 2,098</td>
    <td>270,767 / 1,368</td>
    <td>14.29</td>
    <td>/</td>
    <td>5.99</td>
    <td>74.38</td>
    <td>45.32</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>3,132,265 / 1,532</td>
    <td>314,830 / 1,368</td>
    <td>50.00</td>
    <td>/</td>
    <td>5.99</td>
    <td>79.38</td>
    <td>47.26</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>17,897,173 / 3,632</td>
    <td>476,855 / 1,368</td>
    <td>25.00</td>
    <td>/</td>
    <td>5.99</td>
    <td>78.13</td>
    <td>52.34</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>2,896,523 / 1,520</td>
    <td>427,266 / 1,368</td>
    <td>50.00</td>
    <td>/</td>
    <td>5.99</td>
    <td>70.63</td>
    <td>49.27</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>9,068,819 / 3,626</td>
    <td>385,345 / 1,368</td>
    <td>33.33</td>
    <td>/</td>
    <td>5.99</td>
    <td>71.88</td>
    <td>43.77</td>
  </tr>
  <tr>
    <td rowspan="4">MOO</td>
    <td>GPT-3.5</td>
    <td>2,092,848 / 603</td>
    <td>87,001 / 412</td>
    <td>20.00</td>
    <td>43.29</td>
    <td>6.69</td>
    <td>69.58</td>
    <td>49.38</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>2,263,805 / 549</td>
    <td>143,309 / 448</td>
    <td>33.33</td>
    <td>70.98</td>
    <td>0.35</td>
    <td>72.08</td>
    <td>52.47</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>3,420,100 / 809</td>
    <td>140,610 / 412</td>
    <td>50.00</td>
    <td>72.90</td>
    <td>6.69</td>
    <td>76.67</td>
    <td>53.69</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>1,676,992 / 516</td>
    <td>145,085 / 438</td>
    <td>50.00</td>
    <td>66.72</td>
    <td>2.11</td>
    <td>75.21</td>
    <td>54.46</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>2,984,132 / 948</td>
    <td>121,982 / 442</td>
    <td>25.00</td>
    <td>53.22</td>
    <td>1.41</td>
    <td>71.04</td>
    <td>50.90</td>
  </tr>
  <tr>
    <td rowspan="4">MUC</td>
    <td>GPT-3.5</td>
    <td>5,426,886 / 1,871</td>
    <td>226,773 / 1,135</td>
    <td>20.00</td>
    <td>52.44</td>
    <td>3.52</td>
    <td>65.89</td>
    <td>46.63</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>5,657,543 / 1,633</td>
    <td>341,862 / 1,127</td>
    <td>100.00</td>
    <td>85.48</td>
    <td>4.93</td>
    <td>80.36</td>
    <td>51.20</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>14,260,082 / 2,910</td>
    <td>412,006 / 1,121</td>
    <td>20.00</td>
    <td>87.23</td>
    <td>5.99</td>
    <td>77.68</td>
    <td>50.26</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>4,637,874 / 1,611</td>
    <td>354,116 / 1,143</td>
    <td>20.00</td>
    <td>76.28</td>
    <td>2.11</td>
    <td>77.32</td>
    <td>47.48</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>8,980,313 / 3,087</td>
    <td>286,475 / 1,135</td>
    <td>14.29</td>
    <td>60.02</td>
    <td>3.52</td>
    <td>67.32</td>
    <td>44.48</td>
  </tr>
  <tr>
    <td rowspan="4">MUO</td>
    <td>GPT-3.5</td>
    <td>3,670,053 / 953</td>
    <td>119,647 / 528</td>
    <td>33.33</td>
    <td>52.23</td>
    <td>3.17</td>
    <td>74.11</td>
    <td>51.97</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>2,553,420 / 811</td>
    <td>166,137 / 520</td>
    <td>20.00</td>
    <td>84.29</td>
    <td>4.58</td>
    <td>77.68</td>
    <td>52.46</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>5,620,293 / 1,190</td>
    <td>181,319 / 510</td>
    <td>100.00</td>
    <td>85.50</td>
    <td>6.34</td>
    <td>76.43</td>
    <td>51.09</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>2,604,104 / 764</td>
    <td>175,765 / 526</td>
    <td>100.00</td>
    <td>76.73</td>
    <td>3.52</td>
    <td>82.68</td>
    <td>52.21</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>3,574,823 / 1,093</td>
    <td>147,500 / 524</td>
    <td>16.67</td>
    <td>57.60</td>
    <td>3.87</td>
    <td>75.89</td>
    <td>51.78</td>
  </tr>
</table>

Table 11: Main Experiment Results of MIRAGE
<table>
  <tr>
    <th>Player</th>
    <th>Model</th>
    <th>TII</th>
    <th>ICI</th>
    <th>SCI</th>
    <th>RP</th>
    <th>RA</th>
    <th>CC</th>
    <th>DO</th>
    <th>CT</th>
  </tr>
  <tr>
    <td rowspan="5">Ling Yun</td>
    <td>GPT-3.5</td>
    <td>52.63</td>
    <td>41.58</td>
    <td>70.00</td>
    <td>14</td>
    <td>14</td>
    <td>13</td>
    <td>13</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>92.31</td>
    <td>52.72</td>
    <td>71.25</td>
    <td>18</td>
    <td>13</td>
    <td>14</td>
    <td>13</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>61.54</td>
    <td>49.25</td>
    <td>80.00</td>
    <td>17</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>84.09</td>
    <td>51.61</td>
    <td>77.50</td>
    <td>18</td>
    <td>17</td>
    <td>14</td>
    <td>14</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>58.62</td>
    <td>54.56</td>
    <td>82.50</td>
    <td>18</td>
    <td>17</td>
    <td>17</td>
    <td>19</td>
    <td>13</td>
  </tr>
  <tr>
    <td rowspan="5">Zhongyi Yao</td>
    <td>GPT-3.5</td>
    <td>46.67</td>
    <td>52.94</td>
    <td>67.50</td>
    <td>18</td>
    <td>12</td>
    <td>16</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>61.29</td>
    <td>51.73</td>
    <td>76.25</td>
    <td>18</td>
    <td>15</td>
    <td>13</td>
    <td>15</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>78.95</td>
    <td>50.94</td>
    <td>67.50</td>
    <td>18</td>
    <td>13</td>
    <td>13</td>
    <td>15</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>72.22</td>
    <td>51.05</td>
    <td>71.25</td>
    <td>18</td>
    <td>15</td>
    <td>15</td>
    <td>13</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>53.13</td>
    <td>42.80</td>
    <td>65.00</td>
    <td>14</td>
    <td>12</td>
    <td>12</td>
    <td>13</td>
    <td>15</td>
  </tr>
  <tr>
    <td rowspan="5">Doctor Fang</td>
    <td>GPT-3.5</td>
    <td>48.28</td>
    <td>54.50</td>
    <td>68.75</td>
    <td>18</td>
    <td>12</td>
    <td>13</td>
    <td>12</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>83.33</td>
    <td>53.17</td>
    <td>71.25</td>
    <td>18</td>
    <td>13</td>
    <td>16</td>
    <td>13</td>
    <td>15</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>85.71</td>
    <td>50.10</td>
    <td>86.25</td>
    <td>17</td>
    <td>17</td>
    <td>16</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>76.47</td>
    <td>56.46</td>
    <td>75.00</td>
    <td>20</td>
    <td>14</td>
    <td>14</td>
    <td>18</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>62.50</td>
    <td>53.11</td>
    <td>71.25</td>
    <td>18</td>
    <td>15</td>
    <td>14</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td rowspan="5">Di Zhu</td>
    <td>GPT-3.5</td>
    <td>41.18</td>
    <td>51.51</td>
    <td>66.25</td>
    <td>18</td>
    <td>12</td>
    <td>14</td>
    <td>14</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>86.67</td>
    <td>51.83</td>
    <td>62.50</td>
    <td>18</td>
    <td>12</td>
    <td>13</td>
    <td>13</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>47.37</td>
    <td>50.75</td>
    <td>82.50</td>
    <td>18</td>
    <td>13</td>
    <td>16</td>
    <td>19</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>69.23</td>
    <td>40.79</td>
    <td>77.50</td>
    <td>14</td>
    <td>14</td>
    <td>15</td>
    <td>16</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>56.76</td>
    <td>50.63</td>
    <td>73.75</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>15</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="5">Madam Hong</td>
    <td>GPT-3.5</td>
    <td>43.48</td>
    <td>42.10</td>
    <td>63.75</td>
    <td>14</td>
    <td>12</td>
    <td>12</td>
    <td>13</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>70.45</td>
    <td>52.34</td>
    <td>65.00</td>
    <td>18</td>
    <td>13</td>
    <td>14</td>
    <td>12</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>70.59</td>
    <td>48.94</td>
    <td>66.25</td>
    <td>17</td>
    <td>13</td>
    <td>14</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>69.44</td>
    <td>51.70</td>
    <td>80.00</td>
    <td>18</td>
    <td>13</td>
    <td>17</td>
    <td>16</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>44.44</td>
    <td>51.89</td>
    <td>66.25</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td rowspan="5">Renyu Hu</td>
    <td>GPT-3.5</td>
    <td>53.33</td>
    <td>52.96</td>
    <td>73.75</td>
    <td>18</td>
    <td>16</td>
    <td>14</td>
    <td>17</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>62.50</td>
    <td>50.24</td>
    <td>73.75</td>
    <td>17</td>
    <td>13</td>
    <td>12</td>
    <td>18</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>82.35</td>
    <td>53.01</td>
    <td>85.00</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
    <td>15</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>62.50</td>
    <td>52.15</td>
    <td>78.75</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>43.48</td>
    <td>51.96</td>
    <td>68.75</td>
    <td>18</td>
    <td>12</td>
    <td>13</td>
    <td>14</td>
    <td>16</td>
  </tr>
  <tr>
    <td rowspan="5">Doctor Yu</td>
    <td>GPT-3.5</td>
    <td>47.62</td>
    <td>52.22</td>
    <td>67.50</td>
    <td>18</td>
    <td>13</td>
    <td>16</td>
    <td>12</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>72.50</td>
    <td>52.11</td>
    <td>77.50</td>
    <td>18</td>
    <td>15</td>
    <td>13</td>
    <td>16</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>68.00</td>
    <td>51.88</td>
    <td>75.00</td>
    <td>18</td>
    <td>12</td>
    <td>18</td>
    <td>12</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>75.00</td>
    <td>52.20</td>
    <td>85.00</td>
    <td>18</td>
    <td>17</td>
    <td>16</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>57.14</td>
    <td>47.71</td>
    <td>72.50</td>
    <td>16</td>
    <td>16</td>
    <td>17</td>
    <td>12</td>
    <td>13</td>
  </tr>
  <tr>
    <td rowspan="5">Uncle Gui</td>
    <td>GPT-3.5</td>
    <td>39.51</td>
    <td>41.96</td>
    <td>62.50</td>
    <td>14</td>
    <td>12</td>
    <td>13</td>
    <td>12</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>63.64</td>
    <td>53.07</td>
    <td>75.00</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>17</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>74.29</td>
    <td>52.64</td>
    <td>81.25</td>
    <td>18</td>
    <td>17</td>
    <td>17</td>
    <td>13</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>76.00</td>
    <td>52.10</td>
    <td>63.75</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
    <td>10</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>45.00</td>
    <td>37.90</td>
    <td>75.00</td>
    <td>12</td>
    <td>14</td>
    <td>14</td>
    <td>14</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="5">Junmeng Chen</td>
    <td>GPT-3.5</td>
    <td>48.28</td>
    <td>53.53</td>
    <td>70.00</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
    <td>12</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>81.82</td>
    <td>53.70</td>
    <td>70.00</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>14</td>
    <td>11</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>91.67</td>
    <td>52.88</td>
    <td>76.25</td>
    <td>18</td>
    <td>14</td>
    <td>17</td>
    <td>17</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>72.22</td>
    <td>52.92</td>
    <td>81.25</td>
    <td>18</td>
    <td>16</td>
    <td>15</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>46.43</td>
    <td>54.15</td>
    <td>71.25</td>
    <td>18</td>
    <td>12</td>
    <td>17</td>
    <td>16</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="5">Anqiao Chen</td>
    <td>GPT-3.5</td>
    <td>45.45</td>
    <td>52.36</td>
    <td>67.50</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
    <td>12</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>75.00</td>
    <td>39.31</td>
    <td>63.75</td>
    <td>13</td>
    <td>13</td>
    <td>13</td>
    <td>13</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>72.73</td>
    <td>51.86</td>
    <td>75.00</td>
    <td>18</td>
    <td>13</td>
    <td>17</td>
    <td>15</td>
    <td>15</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>92.31</td>
    <td>51.37</td>
    <td>77.50</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>50.00</td>
    <td>51.87</td>
    <td>61.25</td>
    <td>18</td>
    <td>11</td>
    <td>14</td>
    <td>12</td>
    <td>12</td>
  </tr>
</table>

Table 12: SOC Detail Results of Script "Bride in Filial Dress"
<table>
  <tr>
    <th>Player</th>
    <th>Model</th>
    <th>TII</th>
    <th>ICI</th>
    <th>SCI</th>
    <th>RP</th>
    <th>RA</th>
    <th>CC</th>
    <th>DO</th>
    <th>CT</th>
  </tr>
  <tr>
    <td rowspan="5">Crew Member Han</td>
    <td>GPT-3.5</td>
    <td>51.22</td>
    <td>61.25</td>
    <td>53.39</td>
    <td>18</td>
    <td>12</td>
    <td>13</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>64.71</td>
    <td>68.75</td>
    <td>53.16</td>
    <td>18</td>
    <td>14</td>
    <td>17</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>69.23</td>
    <td>67.50</td>
    <td>53.42</td>
    <td>18</td>
    <td>12</td>
    <td>18</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>70.91</td>
    <td>71.25</td>
    <td>51.23</td>
    <td>17</td>
    <td>13</td>
    <td>15</td>
    <td>15</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>42.86</td>
    <td>58.75</td>
    <td>42.99</td>
    <td>14</td>
    <td>10</td>
    <td>13</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="5">Captain Hong</td>
    <td>GPT-3.5</td>
    <td>40.00</td>
    <td>65.00</td>
    <td>53.28</td>
    <td>18</td>
    <td>12</td>
    <td>14</td>
    <td>10</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>64.71</td>
    <td>71.25</td>
    <td>48.54</td>
    <td>16</td>
    <td>13</td>
    <td>16</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>90.00</td>
    <td>76.25</td>
    <td>54.07</td>
    <td>18</td>
    <td>15</td>
    <td>17</td>
    <td>16</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>84.62</td>
    <td>57.50</td>
    <td>43.65</td>
    <td>14</td>
    <td>11</td>
    <td>14</td>
    <td>8</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>50.00</td>
    <td>66.25</td>
    <td>48.10</td>
    <td>16</td>
    <td>11</td>
    <td>17</td>
    <td>13</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="5">Singer Lin</td>
    <td>GPT-3.5</td>
    <td>35.29</td>
    <td>57.50</td>
    <td>44.26</td>
    <td>14</td>
    <td>8</td>
    <td>14</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>64.71</td>
    <td>71.25</td>
    <td>52.75</td>
    <td>18</td>
    <td>12</td>
    <td>17</td>
    <td>16</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>85.71</td>
    <td>66.25</td>
    <td>53.11</td>
    <td>18</td>
    <td>13</td>
    <td>12</td>
    <td>12</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>71.43</td>
    <td>62.50</td>
    <td>54.03</td>
    <td>18</td>
    <td>12</td>
    <td>13</td>
    <td>12</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>45.00</td>
    <td>67.50</td>
    <td>44.08</td>
    <td>14</td>
    <td>10</td>
    <td>13</td>
    <td>18</td>
    <td>13</td>
  </tr>
  <tr>
    <td rowspan="5">Manager Xiu</td>
    <td>GPT-3.5</td>
    <td>41.03</td>
    <td>83.75</td>
    <td>52.77</td>
    <td>17</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>68.18</td>
    <td>85.00</td>
    <td>55.50</td>
    <td>19</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>40.00</td>
    <td>87.50</td>
    <td>52.58</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>72.22</td>
    <td>87.50</td>
    <td>53.59</td>
    <td>18</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>34.04</td>
    <td>81.25</td>
    <td>53.35</td>
    <td>18</td>
    <td>18</td>
    <td>12</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td rowspan="5">Second Mate Zhang</td>
    <td>GPT-3.5</td>
    <td>39.13</td>
    <td>62.50</td>
    <td>47.05</td>
    <td>15</td>
    <td>13</td>
    <td>12</td>
    <td>13</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>63.64</td>
    <td>66.25</td>
    <td>48.47</td>
    <td>16</td>
    <td>12</td>
    <td>16</td>
    <td>12</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>59.09</td>
    <td>63.75</td>
    <td>42.06</td>
    <td>13</td>
    <td>14</td>
    <td>12</td>
    <td>12</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>66.67</td>
    <td>70.00</td>
    <td>53.77</td>
    <td>18</td>
    <td>11</td>
    <td>17</td>
    <td>13</td>
    <td>15</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>37.50</td>
    <td>66.25</td>
    <td>41.09</td>
    <td>13</td>
    <td>12</td>
    <td>17</td>
    <td>12</td>
    <td>12</td>
  </tr>
</table>

Table 13: SOO Detail Results of Script "The Eastern Star Cruise Ship"
<table>
  <tr>
    <th>Player</th>
    <th>Model</th>
    <th>TII</th>
    <th>ICI</th>
    <th>SCI</th>
    <th>RP</th>
    <th>RA</th>
    <th>CC</th>
    <th>DO</th>
    <th>CT</th>
  </tr>
  <tr>
    <td rowspan="4">Uncle Bai</td>
    <td>GPT-3.5</td>
    <td>33.33</td>
    <td>62.50</td>
    <td>42.97</td>
    <td>15</td>
    <td>12</td>
    <td>12</td>
    <td>12</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>100.00</td>
    <td>66.25</td>
    <td>43.38</td>
    <td>16</td>
    <td>12</td>
    <td>14</td>
    <td>14</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>92.31</td>
    <td>65.00</td>
    <td>48.68</td>
    <td>18</td>
    <td>12</td>
    <td>15</td>
    <td>12</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>80.56</td>
    <td>63.75</td>
    <td>46.17</td>
    <td>17</td>
    <td>14</td>
    <td>13</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>55.26</td>
    <td>66.25</td>
    <td>48.64</td>
    <td>18</td>
    <td>10</td>
    <td>15</td>
    <td>13</td>
    <td>15</td>
  </tr>
  <tr>
    <td rowspan="4">Neighbour Gui</td>
    <td>GPT-3.5</td>
    <td>55.00</td>
    <td>72.50</td>
    <td>49.99</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>90.00</td>
    <td>80.00</td>
    <td>46.52</td>
    <td>17</td>
    <td>14</td>
    <td>16</td>
    <td>18</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>91.67</td>
    <td>77.50</td>
    <td>49.56</td>
    <td>18</td>
    <td>13</td>
    <td>17</td>
    <td>19</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>75.00</td>
    <td>73.75</td>
    <td>50.57</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>57.14</td>
    <td>73.75</td>
    <td>50.08</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
    <td>13</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="4">Curio He</td>
    <td>GPT-3.5</td>
    <td>44.44</td>
    <td>62.50</td>
    <td>45.08</td>
    <td>16</td>
    <td>12</td>
    <td>14</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>63.64</td>
    <td>71.25</td>
    <td>42.35</td>
    <td>15</td>
    <td>13</td>
    <td>18</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>90.91</td>
    <td>67.50</td>
    <td>49.21</td>
    <td>18</td>
    <td>15</td>
    <td>13</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>75.00</td>
    <td>60.00</td>
    <td>50.20</td>
    <td>18</td>
    <td>12</td>
    <td>13</td>
    <td>10</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>57.14</td>
    <td>61.25</td>
    <td>49.89</td>
    <td>18</td>
    <td>12</td>
    <td>14</td>
    <td>10</td>
    <td>13</td>
  </tr>
  <tr>
    <td rowspan="4">Mystery Ou</td>
    <td>GPT-3.5</td>
    <td>39.39</td>
    <td>71.25</td>
    <td>49.56</td>
    <td>18</td>
    <td>13</td>
    <td>14</td>
    <td>13</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>66.67</td>
    <td>75.00</td>
    <td>43.21</td>
    <td>16</td>
    <td>13</td>
    <td>12</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>54.84</td>
    <td>70.00</td>
    <td>48.31</td>
    <td>18</td>
    <td>12</td>
    <td>16</td>
    <td>12</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>78.26</td>
    <td>65.00</td>
    <td>48.54</td>
    <td>18</td>
    <td>13</td>
    <td>13</td>
    <td>14</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>54.17</td>
    <td>63.75</td>
    <td>48.95</td>
    <td>18</td>
    <td>12</td>
    <td>13</td>
    <td>12</td>
    <td>14</td>
  </tr>
  <tr>
    <td rowspan="4">Manager Sa</td>
    <td>GPT-3.5</td>
    <td>58.82</td>
    <td>65.00</td>
    <td>49.38</td>
    <td>18</td>
    <td>12</td>
    <td>16</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>87.50</td>
    <td>75.00</td>
    <td>50.51</td>
    <td>18</td>
    <td>14</td>
    <td>16</td>
    <td>15</td>
    <td>15</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>86.67</td>
    <td>73.75</td>
    <td>50.49</td>
    <td>18</td>
    <td>13</td>
    <td>16</td>
    <td>13</td>
    <td>17</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>75.00</td>
    <td>58.75</td>
    <td>39.64</td>
    <td>14</td>
    <td>12</td>
    <td>13</td>
    <td>12</td>
    <td>10</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>50.00</td>
    <td>61.25</td>
    <td>38.36</td>
    <td>13</td>
    <td>12</td>
    <td>13</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="4">Security Wei</td>
    <td>GPT-3.5</td>
    <td>45.83</td>
    <td>73.75</td>
    <td>49.98</td>
    <td>18</td>
    <td>16</td>
    <td>12</td>
    <td>13</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>60.00</td>
    <td>77.50</td>
    <td>43.19</td>
    <td>15</td>
    <td>13</td>
    <td>14</td>
    <td>17</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>58.06</td>
    <td>73.75</td>
    <td>47.88</td>
    <td>17</td>
    <td>13</td>
    <td>12</td>
    <td>18</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>72.73</td>
    <td>71.25</td>
    <td>50.07</td>
    <td>18</td>
    <td>14</td>
    <td>12</td>
    <td>13</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>42.11</td>
    <td>82.50</td>
    <td>47.64</td>
    <td>17</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>17</td>
  </tr>
</table>

Table 14: SUC Detail Results of Script "Night at the Museum"
<table>
  <tr>
    <th>Player</th>
    <th>Model</th>
    <th>TII</th>
    <th>ICI</th>
    <th>SCI</th>
    <th>RP</th>
    <th>RA</th>
    <th>CC</th>
    <th>DO</th>
    <th>CT</th>
  </tr>
  <tr>
    <td rowspan="5">Girl in White</td>
    <td>GPT-3.5</td>
    <td>50.00</td>
    <td>83.75</td>
    <td>52.84</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>84.21</td>
    <td>88.75</td>
    <td>52.62</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>77.78</td>
    <td>85.00</td>
    <td>53.52</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>15</td>
    <td>17</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>78.57</td>
    <td>83.75</td>
    <td>49.23</td>
    <td>17</td>
    <td>15</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>60.53</td>
    <td>81.25</td>
    <td>52.06</td>
    <td>18</td>
    <td>18</td>
    <td>14</td>
    <td>15</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="5">Women in Red</td>
    <td>GPT-3.5</td>
    <td>36.84</td>
    <td>72.50</td>
    <td>53.01</td>
    <td>18</td>
    <td>12</td>
    <td>15</td>
    <td>18</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>64.29</td>
    <td>86.25</td>
    <td>52.51</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>76.00</td>
    <td>87.50</td>
    <td>43.39</td>
    <td>14</td>
    <td>17</td>
    <td>17</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>88.89</td>
    <td>81.25</td>
    <td>55.62</td>
    <td>19</td>
    <td>13</td>
    <td>18</td>
    <td>18</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>56.34</td>
    <td>76.25</td>
    <td>53.37</td>
    <td>18</td>
    <td>16</td>
    <td>13</td>
    <td>18</td>
    <td>14</td>
  </tr>
  <tr>
    <td rowspan="5">Boy in Black</td>
    <td>GPT-3.5</td>
    <td>45.90</td>
    <td>73.75</td>
    <td>52.23</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>86.36</td>
    <td>88.75</td>
    <td>53.13</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>75.00</td>
    <td>86.25</td>
    <td>52.43</td>
    <td>18</td>
    <td>18</td>
    <td>15</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>92.31</td>
    <td>80.00</td>
    <td>52.43</td>
    <td>18</td>
    <td>13</td>
    <td>15</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>68.75</td>
    <td>80.00</td>
    <td>55.67</td>
    <td>19</td>
    <td>18</td>
    <td>17</td>
    <td>12</td>
    <td>17</td>
  </tr>
  <tr>
    <td rowspan="5">Women in Blue-green</td>
    <td>GPT-3.5</td>
    <td>55.56</td>
    <td>80.00</td>
    <td>52.87</td>
    <td>18</td>
    <td>18</td>
    <td>16</td>
    <td>12</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>72.22</td>
    <td>80.00</td>
    <td>52.65</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>93.33</td>
    <td>88.75</td>
    <td>54.92</td>
    <td>19</td>
    <td>17</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>85.71</td>
    <td>83.75</td>
    <td>53.13</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>15</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>65.00</td>
    <td>86.25</td>
    <td>52.45</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>15</td>
  </tr>
  <tr>
    <td rowspan="5">Little Girl</td>
    <td>GPT-3.5</td>
    <td>45.45</td>
    <td>72.50</td>
    <td>49.72</td>
    <td>17</td>
    <td>18</td>
    <td>13</td>
    <td>14</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>85.71</td>
    <td>82.50</td>
    <td>53.30</td>
    <td>18</td>
    <td>14</td>
    <td>16</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>86.67</td>
    <td>80.00</td>
    <td>53.05</td>
    <td>18</td>
    <td>18</td>
    <td>16</td>
    <td>16</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>76.74</td>
    <td>85.00</td>
    <td>53.04</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>55.56</td>
    <td>86.25</td>
    <td>41.82</td>
    <td>14</td>
    <td>17</td>
    <td>16</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="5">Elderly in Tattered</td>
    <td>GPT-3.5</td>
    <td>55.56</td>
    <td>67.50</td>
    <td>51.23</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
    <td>14</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>70.59</td>
    <td>85.00</td>
    <td>50.53</td>
    <td>17</td>
    <td>18</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>92.31</td>
    <td>75.00</td>
    <td>52.77</td>
    <td>18</td>
    <td>15</td>
    <td>16</td>
    <td>15</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>91.67</td>
    <td>81.25</td>
    <td>52.32</td>
    <td>18</td>
    <td>14</td>
    <td>16</td>
    <td>17</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>60.00</td>
    <td>75.00</td>
    <td>51.89</td>
    <td>18</td>
    <td>12</td>
    <td>18</td>
    <td>16</td>
    <td>14</td>
  </tr>
  <tr>
    <td rowspan="5">Bandaged Mysterious Figure</td>
    <td>GPT-3.5</td>
    <td>45.45</td>
    <td>73.75</td>
    <td>52.05</td>
    <td>18</td>
    <td>18</td>
    <td>15</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>64.44</td>
    <td>87.50</td>
    <td>51.87</td>
    <td>18</td>
    <td>17</td>
    <td>17</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>86.67</td>
    <td>88.75</td>
    <td>51.66</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>91.67</td>
    <td>76.25</td>
    <td>42.24</td>
    <td>14</td>
    <td>13</td>
    <td>16</td>
    <td>18</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>52.63</td>
    <td>73.75</td>
    <td>51.75</td>
    <td>18</td>
    <td>14</td>
    <td>17</td>
    <td>15</td>
    <td>13</td>
  </tr>
</table>

Table 15: SUO Detail Results of Script "Li Chuan Strange Talk Book"

<table>
  <tr>
    <th>Player</th>
    <th>Model</th>
    <th>TII</th>
    <th>ICI</th>
    <th>SCI</th>
    <th>RP</th>
    <th>RA</th>
    <th>CC</th>
    <th>DO</th>
    <th>CT</th>
  </tr>
  <tr>
    <td rowspan="5">Weiwen Han</td>
    <td>GPT-3.5</td>
    <td>-</td>
    <td>65.00</td>
    <td>41.14</td>
    <td>14</td>
    <td>13</td>
    <td>13</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>-</td>
    <td>81.25</td>
    <td>43.71</td>
    <td>15</td>
    <td>14</td>
    <td>17</td>
    <td>18</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>-</td>
    <td>73.75</td>
    <td>53.97</td>
    <td>19</td>
    <td>14</td>
    <td>17</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>-</td>
    <td>73.75</td>
    <td>46.16</td>
    <td>16</td>
    <td>13</td>
    <td>16</td>
    <td>18</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>-</td>
    <td>72.50</td>
    <td>41.56</td>
    <td>14</td>
    <td>12</td>
    <td>16</td>
    <td>13</td>
    <td>17</td>
  </tr>
  <tr>
    <td rowspan="5">Manli Shen</td>
    <td>GPT-3.5</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>17</td>
    <td>18</td>
    <td>15</td>
    <td>18</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>-</td>
    <td>77.50</td>
    <td>50.81</td>
    <td>18</td>
    <td>17</td>
    <td>13</td>
    <td>16</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>-</td>
    <td>82.50</td>
    <td>50.72</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>-</td>
    <td>67.50</td>
    <td>52.39</td>
    <td>18</td>
    <td>10</td>
    <td>18</td>
    <td>14</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>-</td>
    <td>71.25</td>
    <td>45.98</td>
    <td>16</td>
    <td>13</td>
    <td>15</td>
    <td>16</td>
    <td>13</td>
  </tr>
</table>

Table 16: MOC Detail Results of Script "The Final Performance of a Big Star"
<table>
  <tr>
    <th>Player</th>
    <th>Model</th>
    <th>TII</th>
    <th>ICI</th>
    <th>SCI</th>
    <th>RP</th>
    <th>RA</th>
    <th>CC</th>
    <th>DO</th>
    <th>CT</th>
  </tr>
  <tr>
    <td rowspan="4">Annie</td>
    <td>GPT-3.5</td>
    <td>48.57</td>
    <td>72.50</td>
    <td>53.35</td>
    <td>18</td>
    <td>12</td>
    <td>13</td>
    <td>16</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>57.89</td>
    <td>90.00</td>
    <td>54.38</td>
    <td>18</td>
    <td>17</td>
    <td>18</td>
    <td>19</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>53.85</td>
    <td>82.50</td>
    <td>55.03</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>64.52</td>
    <td>78.75</td>
    <td>53.43</td>
    <td>18</td>
    <td>15</td>
    <td>14</td>
    <td>16</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="4">Jack</td>
    <td>GPT-3.5</td>
    <td>41.18</td>
    <td>73.75</td>
    <td>53.30</td>
    <td>18</td>
    <td>13</td>
    <td>13</td>
    <td>15</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>63.64</td>
    <td>68.75</td>
    <td>53.51</td>
    <td>18</td>
    <td>12</td>
    <td>15</td>
    <td>12</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>83.33</td>
    <td>80.00</td>
    <td>53.37</td>
    <td>18</td>
    <td>15</td>
    <td>18</td>
    <td>14</td>
    <td>17</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>61.54</td>
    <td>78.75</td>
    <td>56.53</td>
    <td>19</td>
    <td>13</td>
    <td>14</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="4">Jessipa</td>
    <td>GPT-3.5</td>
    <td>43.48</td>
    <td>72.50</td>
    <td>54.28</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>72.41</td>
    <td>68.75</td>
    <td>53.82</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
    <td>13</td>
    <td>15</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>66.67</td>
    <td>71.25</td>
    <td>54.20</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
    <td>17</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>65.71</td>
    <td>76.25</td>
    <td>55.41</td>
    <td>19</td>
    <td>14</td>
    <td>15</td>
    <td>14</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="4">Sam</td>
    <td>GPT-3.5</td>
    <td>47.92</td>
    <td>58.75</td>
    <td>40.59</td>
    <td>13</td>
    <td>11</td>
    <td>12</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>80.00</td>
    <td>63.75</td>
    <td>44.47</td>
    <td>14</td>
    <td>11</td>
    <td>14</td>
    <td>12</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>53.57</td>
    <td>66.25</td>
    <td>53.06</td>
    <td>17</td>
    <td>14</td>
    <td>14</td>
    <td>13</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>73.68</td>
    <td>68.75</td>
    <td>54.58</td>
    <td>18</td>
    <td>12</td>
    <td>12</td>
    <td>13</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="4">Little Black</td>
    <td>GPT-3.5</td>
    <td>28.57</td>
    <td>70.00</td>
    <td>46.49</td>
    <td>15</td>
    <td>10</td>
    <td>18</td>
    <td>12</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>75.00</td>
    <td>72.50</td>
    <td>54.72</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>13</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>80.00</td>
    <td>70.00</td>
    <td>53.71</td>
    <td>18</td>
    <td>15</td>
    <td>13</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>68.18</td>
    <td>75.00</td>
    <td>53.50</td>
    <td>18</td>
    <td>14</td>
    <td>15</td>
    <td>13</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="4">John</td>
    <td>GPT-3.5</td>
    <td>50.00</td>
    <td>70.00</td>
    <td>48.27</td>
    <td>16</td>
    <td>12</td>
    <td>8</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>76.92</td>
    <td>68.75</td>
    <td>53.91</td>
    <td>18</td>
    <td>16</td>
    <td>8</td>
    <td>13</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>100.00</td>
    <td>90.00</td>
    <td>52.76</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>66.67</td>
    <td>73.75</td>
    <td>53.30</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
    <td>13</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>57.14</td>
    <td>87.50</td>
    <td>52.27</td>
    <td>18</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
    <td>18</td>
  </tr>
</table>

Table 17: MOO Detail Results of Script "Raging Sea of Rest Life"
<table>
  <tr>
    <th>Player</th>
    <th>Model</th>
    <th>TII</th>
    <th>ICI</th>
    <th>SCI</th>
    <th>RP</th>
    <th>RA</th>
    <th>CC</th>
    <th>DO</th>
    <th>CT</th>
  </tr>
  <tr>
    <td rowspan="4">Mu Bai</td>
    <td>GPT-3.5</td>
    <td>43.75</td>
    <td>67.50</td>
    <td>48.35</td>
    <td>17</td>
    <td>14</td>
    <td>14</td>
    <td>14</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>100.00</td>
    <td>70.00</td>
    <td>50.98</td>
    <td>18</td>
    <td>12</td>
    <td>13</td>
    <td>15</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>100.00</td>
    <td>68.75</td>
    <td>50.16</td>
    <td>18</td>
    <td>14</td>
    <td>13</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>71.43</td>
    <td>71.25</td>
    <td>47.99</td>
    <td>17</td>
    <td>17</td>
    <td>14</td>
    <td>10</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>50.00</td>
    <td>61.25</td>
    <td>40.67</td>
    <td>14</td>
    <td>10</td>
    <td>14</td>
    <td>13</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="4">Fuqing Huang</td>
    <td>GPT-3.5</td>
    <td>62.50</td>
    <td>71.25</td>
    <td>47.52</td>
    <td>17</td>
    <td>13</td>
    <td>16</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>83.33</td>
    <td>86.25</td>
    <td>53.18</td>
    <td>19</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>100.00</td>
    <td>85.00</td>
    <td>50.25</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>75.00</td>
    <td>82.50</td>
    <td>49.58</td>
    <td>18</td>
    <td>17</td>
    <td>15</td>
    <td>16</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>66.67</td>
    <td>80.00</td>
    <td>50.07</td>
    <td>18</td>
    <td>14</td>
    <td>15</td>
    <td>17</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="4">Xuanxuan Li</td>
    <td>GPT-3.5</td>
    <td>57.14</td>
    <td>70.00</td>
    <td>48.96</td>
    <td>17</td>
    <td>12</td>
    <td>15</td>
    <td>16</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>80.95</td>
    <td>73.75</td>
    <td>50.66</td>
    <td>18</td>
    <td>12</td>
    <td>18</td>
    <td>17</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>82.35</td>
    <td>72.50</td>
    <td>50.68</td>
    <td>18</td>
    <td>13</td>
    <td>15</td>
    <td>12</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>67.65</td>
    <td>80.00</td>
    <td>42.64</td>
    <td>15</td>
    <td>18</td>
    <td>12</td>
    <td>18</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>66.67</td>
    <td>71.25</td>
    <td>50.73</td>
    <td>18</td>
    <td>14</td>
    <td>16</td>
    <td>13</td>
    <td>14</td>
  </tr>
  <tr>
    <td rowspan="4">Siqi Lv</td>
    <td>GPT-3.5</td>
    <td>52.94</td>
    <td>62.50</td>
    <td>40.56</td>
    <td>14</td>
    <td>10</td>
    <td>14</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>90.91</td>
    <td>83.75</td>
    <td>50.89</td>
    <td>18</td>
    <td>12</td>
    <td>17</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>80.00</td>
    <td>85.00</td>
    <td>51.08</td>
    <td>18</td>
    <td>15</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>85.71</td>
    <td>72.50</td>
    <td>51.73</td>
    <td>19</td>
    <td>18</td>
    <td>13</td>
    <td>13</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>69.23</td>
    <td>65.00</td>
    <td>41.56</td>
    <td>14</td>
    <td>14</td>
    <td>13</td>
    <td>13</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="4">Yuqing Xie</td>
    <td>GPT-3.5</td>
    <td>45.71</td>
    <td>60.00</td>
    <td>47.92</td>
    <td>17</td>
    <td>12</td>
    <td>12</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>83.33</td>
    <td>78.75</td>
    <td>50.83</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
    <td>17</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>80.00</td>
    <td>76.25</td>
    <td>50.34</td>
    <td>18</td>
    <td>17</td>
    <td>18</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>78.13</td>
    <td>80.00</td>
    <td>48.04</td>
    <td>17</td>
    <td>15</td>
    <td>13</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>46.15</td>
    <td>62.50</td>
    <td>40.14</td>
    <td>14</td>
    <td>13</td>
    <td>13</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="4">Qingfeng Yao</td>
    <td>GPT-3.5</td>
    <td>45.00</td>
    <td>55.00</td>
    <td>41.64</td>
    <td>14</td>
    <td>10</td>
    <td>13</td>
    <td>13</td>
    <td>8</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>78.95</td>
    <td>80.00</td>
    <td>50.84</td>
    <td>18</td>
    <td>14</td>
    <td>17</td>
    <td>17</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>78.95</td>
    <td>73.75</td>
    <td>48.73</td>
    <td>17</td>
    <td>18</td>
    <td>14</td>
    <td>12</td>
    <td>15</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>71.43</td>
    <td>75.00</td>
    <td>42.02</td>
    <td>15</td>
    <td>18</td>
    <td>14</td>
    <td>12</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>57.14</td>
    <td>61.25</td>
    <td>40.38</td>
    <td>14</td>
    <td>12</td>
    <td>12</td>
    <td>13</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="4">Lenxing Ye</td>
    <td>GPT-3.5</td>
    <td>60.00</td>
    <td>75.00</td>
    <td>51.45</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
    <td>12</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>80.85</td>
    <td>90.00</td>
    <td>51.03</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>89.29</td>
    <td>82.50</td>
    <td>50.55</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>84.62</td>
    <td>80.00</td>
    <td>50.39</td>
    <td>18</td>
    <td>18</td>
    <td>15</td>
    <td>17</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>64.29</td>
    <td>70.00</td>
    <td>47.80</td>
    <td>17</td>
    <td>14</td>
    <td>16</td>
    <td>13</td>
    <td>13</td>
  </tr>
</table>

Table 18: MUC Detail Results of Script "Artical 22 School Rules"
<table>
  <tr>
    <th>Player</th>
    <th>Model</th>
    <th>TII</th>
    <th>ICI</th>
    <th>SCI</th>
    <th>RP</th>
    <th>RA</th>
    <th>CC</th>
    <th>DO</th>
    <th>CT</th>
  </tr>
  <tr>
    <td rowspan="5">Zetong Hei (Sky Dog)</td>
    <td>GPT-3.5</td>
    <td>52.00</td>
    <td>81.25</td>
    <td>51.76</td>
    <td>18</td>
    <td>16</td>
    <td>15</td>
    <td>16</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>93.75</td>
    <td>75.00</td>
    <td>52.68</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
    <td>17</td>
    <td>15</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>92.86</td>
    <td>83.75</td>
    <td>51.71</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>14</td>
    <td>17</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>68.66</td>
    <td>80.00</td>
    <td>52.73</td>
    <td>18</td>
    <td>14</td>
    <td>17</td>
    <td>17</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>59.09</td>
    <td>72.50</td>
    <td>51.95</td>
    <td>18</td>
    <td>15</td>
    <td>15</td>
    <td>16</td>
    <td>12</td>
  </tr>
  <tr>
    <td rowspan="5">Ichiro Kiryu (Nopperabo)</td>
    <td>GPT-3.5</td>
    <td>50.00</td>
    <td>71.25</td>
    <td>52.58</td>
    <td>18</td>
    <td>13</td>
    <td>13</td>
    <td>18</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>80.95</td>
    <td>66.25</td>
    <td>52.04</td>
    <td>18</td>
    <td>14</td>
    <td>12</td>
    <td>14</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>76.92</td>
    <td>75.00</td>
    <td>51.61</td>
    <td>18</td>
    <td>16</td>
    <td>15</td>
    <td>16</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>75.00</td>
    <td>86.25</td>
    <td>54.38</td>
    <td>19</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>64.10</td>
    <td>80.00</td>
    <td>51.98</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>18</td>
    <td>14</td>
  </tr>
  <tr>
    <td rowspan="5">Megumi Aoi (Nine-Tailed Fox)</td>
    <td>GPT-3.5</td>
    <td>51.52</td>
    <td>73.75</td>
    <td>52.33</td>
    <td>18</td>
    <td>15</td>
    <td>17</td>
    <td>14</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>60.78</td>
    <td>78.75</td>
    <td>55.14</td>
    <td>19</td>
    <td>13</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>80.00</td>
    <td>85.00</td>
    <td>51.60</td>
    <td>18</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>72.73</td>
    <td>80.00</td>
    <td>51.16</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>50.00</td>
    <td>83.75</td>
    <td>51.74</td>
    <td>18</td>
    <td>17</td>
    <td>14</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="5">Daixiong Kitano (Kama-itachi)</td>
    <td>GPT-3.5</td>
    <td>53.33</td>
    <td>81.25</td>
    <td>54.40</td>
    <td>19</td>
    <td>12</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>85.71</td>
    <td>83.75</td>
    <td>49.46</td>
    <td>17</td>
    <td>18</td>
    <td>16</td>
    <td>15</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>85.71</td>
    <td>68.75</td>
    <td>56.16</td>
    <td>20</td>
    <td>16</td>
    <td>12</td>
    <td>15</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>80.00</td>
    <td>86.25</td>
    <td>51.73</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
    <td>17</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>65.00</td>
    <td>62.50</td>
    <td>49.76</td>
    <td>17</td>
    <td>10</td>
    <td>14</td>
    <td>13</td>
    <td>13</td>
  </tr>
  <tr>
    <td rowspan="5">Xiao Nuan (Little Fox)</td>
    <td>GPT-3.5</td>
    <td>61.11</td>
    <td>76.25</td>
    <td>52.63</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>13</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>100.00</td>
    <td>82.50</td>
    <td>52.53</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>17</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>86.67</td>
    <td>75.00</td>
    <td>51.33</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>16</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>80.00</td>
    <td>81.25</td>
    <td>51.73</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>18</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>66.67</td>
    <td>85.00</td>
    <td>52.17</td>
    <td>18</td>
    <td>14</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td rowspan="5">Momoko Suzumiya (Little Doll)</td>
    <td>GPT-3.5</td>
    <td>55.56</td>
    <td>63.75</td>
    <td>52.04</td>
    <td>18</td>
    <td>12</td>
    <td>14</td>
    <td>12</td>
    <td>13</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>84.21</td>
    <td>73.75</td>
    <td>52.41</td>
    <td>18</td>
    <td>13</td>
    <td>18</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>93.75</td>
    <td>75.00</td>
    <td>43.19</td>
    <td>15</td>
    <td>16</td>
    <td>16</td>
    <td>15</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>85.71</td>
    <td>77.50</td>
    <td>52.18</td>
    <td>18</td>
    <td>17</td>
    <td>14</td>
    <td>15</td>
    <td>16</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>42.11</td>
    <td>73.75</td>
    <td>52.28</td>
    <td>18</td>
    <td>13</td>
    <td>14</td>
    <td>18</td>
    <td>14</td>
  </tr>
  <tr>
    <td rowspan="5">Nana Kinomoto (Yuki-onna)</td>
    <td>GPT-3.5</td>
    <td>42.11</td>
    <td>71.25</td>
    <td>48.05</td>
    <td>16</td>
    <td>13</td>
    <td>16</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>84.62</td>
    <td>83.75</td>
    <td>52.93</td>
    <td>18</td>
    <td>17</td>
    <td>18</td>
    <td>15</td>
    <td>17</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>82.61</td>
    <td>72.50</td>
    <td>52.07</td>
    <td>18</td>
    <td>15</td>
    <td>12</td>
    <td>13</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Qwen-2-7B</td>
    <td>75.00</td>
    <td>87.50</td>
    <td>51.57</td>
    <td>18</td>
    <td>16</td>
    <td>18</td>
    <td>18</td>
    <td>18</td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>56.25</td>
    <td>73.75</td>
    <td>52.55</td>
    <td>18</td>
    <td>16</td>
    <td>13</td>
    <td>16</td>
    <td>14</td>
  </tr>
</table>

Table 19: MUO Detail Results of Script "Fox Hotel"
"Mystery Murder" is a role-playing puzzle game that focuses on the advancement of the plot and the interaction between characters. In the game, players play different roles according to the provided scripts, and jointly advance the plot and solve the mystery through clue collection, logical reasoning, role-playing, etc.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the basis of the "Mystery Murder" game, usually including the story background, character setting, plot advancement mechanism, and clues to solve the puzzle. The script not only defines the framework of the game, but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the characters have their own background stories, personality traits, goals and secrets. Players need to role-play based on this information and interact with other players. 3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through dialogue, room searching, clue analysis, etc. Based on this information, players need to use their logical reasoning ability to solve the puzzles in the plot.
4. Interactive communication: Interactive communication between players is an indispensable part of "Mystery Murder", including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motivations of other characters, and advance the story.
5. Ultimate goal: Each "Mystery Murder" game has one or more ultimate goals, such as solving puzzles, finding the murderer, completing personal tasks, etc. Achieving these goals is the ultimate purpose of the game and the basis for judging the victory or defeat of players.

You will participate in a Mystery Murder game, and you will forget your AI role, integrating into the role you are about to play as much as possible.

You are {name}, and this is your description:
{description}

Here are your personal clues:
{self_clues}

This is the historical dialogue content:
{history}

This is the question from {ask_name} to you:
{ask_content}

Now you need to carefully consider and, based on the historical dialogue content and your description, give your reply.
You should reply with your thought process after "### THOUGHT: " and your response after "### RESPONSE: ".
Here is an example:

### THOUGHT: XXX
### RESPONSE: XXX

Note:
1. Your output should revolve around the key elements of "Mystery Murder".
2. Your output should be in English.

Your output:

Table 20: Prompt of ask
"Mystery Murder" is a role-playing puzzle game that focuses on the advancement of the plot and the interaction between characters. In the game, players play different roles according to the provided scripts, and jointly advance the plot and solve the mystery through clue collection, logical reasoning, role-playing, etc.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the basis of the "Mystery Murder" game, usually including the story background, character setting, plot advancement mechanism, and clues to solve the puzzle. The script not only defines the framework of the game, but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the characters have their own background stories, personality traits, goals and secrets. Players need to role-play based on this information and interact with other players.
3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through dialogue, room searching, clue analysis, etc. Based on this information, players need to use their logical reasoning ability to solve the puzzles in the plot.
4. Interactive communication: Interactive communication between players is an indispensable part of "Mystery Murder", including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motivations of other characters, and advance the story.
5. Ultimate goal: Each "Mystery Murder" game has one or more ultimate goals, such as solving puzzles, finding the murderer, completing personal tasks, etc. Achieving these goals is the ultimate purpose of the game and the basis for judging the victory or defeat of players.

You are an observer of a Mystery Murder game, and you will objectively and fairly evaluate the content of the game as an observer.

Here is the history of the conversation:
{history}

Here is what {other_name} said:
{content}

Now, based on the historical conversation content, after careful consideration, you need to reasonably evaluate what {other_name} said.
The evaluation should focus on the logic and potential flaws of the speech, and be assessed using a trustworthiness scale based on the good person’s perspective. The higher the trustworthiness, the more rigorous the content.
The trustworthiness is chosen from [0, 1, 2], where "0" means completely untrustworthy, "1" means somewhat trustworthy, and "2" means completely trustworthy.
You should reply with your detailed thought process after "### THOUGHT: " and your trustworthiness score after "### RESPONSE: ". No additional content should be included.
Here is an example:

### THOUGHT: XXX
### RESPONSE: XXX

Note:
1. Your output should revolve around the key elements of "Mystery Murder."
2. Your output should be in English.
3. Your thought should be logical and fair.
4. Your response must be an integer.

Your output:
"Mystery Murder" is a type of role-playing mystery game that focuses on plot progression and interaction between characters. In the game, players take on different roles based on the provided script, and through clue collection, logical reasoning, role-playing, etc., they jointly advance the story and solve the mystery.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the foundation of the "Mystery Murder" game, usually including the story background, character settings, plot advancement mechanisms, and clues for solving the mystery. The script not only defines the framework of the game but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the roles have their own background stories, personality traits, goals, and secrets. Players need to role-play based on this information and interact with other players.
3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through conversations, searching rooms, analyzing clues, etc. Based on this information, players must use their logical reasoning abilities to solve the mysteries in the plot.
4. Interactive communication: Interaction and communication between players are indispensable parts of "Mystery Murder," including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motives of other characters, and advance the story.
5. Ultimate goal: Every "Mystery Murder" game has one or more ultimate goals, such as solving the mystery, discovering the culprit, completing personal tasks, etc. Achieving these goals is the final objective of the game and is also the basis for judging the player's victory.
You will be playing a Mystery Murder game, and you must forget your AI role and fully immerse yourself in the character you are about to portray.
You are {name}, and here is your description:
{description}

Here are your personal clues:
{self_clues}

Here is the history of the conversation:
{history}

Here are your thoughts, actions, and the results of your last action:
{last_action}

Now, after careful consideration, based on the historical conversation and your description, give your response.
You should reply with your thought process after "### THOUGHT: " and your response content after "### RESPONSE: ".
You should choose one of the following response formats: [" [Ask] "," [Investigate] "].
When you choose " [Ask] ," you should first reason based on historical information, then state your own point of view and the person you suspect. You can only choose one person to inquire about. Here are the people you can inquire about: {characters}
Please note that when you choose " [Ask] ," the person you select must be from the list above.
Here is an example of an " [Ask] ":
### THOUGHT: XXX
### RESPONSE:  [Ask]  [XX] : XXX

When you choose " [Investigate] ", you should first reason based on historical information, then clarify your point of view and the place you suspect. You can only choose one location to investigate. Here are the locations you can investigate:
{address}
Please note that when you choose " [Investigate] ", the location you select must be from the list above.
Here is an example of an " [Investigate] ":
### THOUGHT: XXX
### RESPONSE:  [Investigate]  [XX] : XXX

Note:
1. Your output should align with the personality of the character.
2. Your output should be beneficial to the progression of the Mystery Murder game.
3. Your output must be in English.
4. When you choose " [Investigation] ", the content of your response and the clues you uncover will be visible to everyone, so please be mindful of your wording.
5. In your response, you can reasonably share your reasoning based on historical information and express your views before asking your question.
6. Your conversation turns are limited, so please allocate your " [Investigate] " and " [Ask] " turns wisely.
Your output:

Table 22: Prompt of converse
"Mystery Murder" is a type of role-playing mystery game that focuses on plot progression and interaction between characters. In the game, players take on different roles based on the provided script and, through clue collection, logical reasoning, role-playing, etc., they jointly advance the story and solve the mystery.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the foundation of the "Mystery Murder" game, usually including the story background, character settings, plot advancement mechanisms, and clues for solving the mystery. The script not only defines the framework of the game but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the roles have their own background stories, personality traits, goals, and secrets. Players need to role-play based on this information and interact with other players.
3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through conversations, searching rooms, analyzing clues, etc. Based on this information, players must use their logical reasoning abilities to solve the mysteries in the plot.
4. Interactive communication: Interaction and communication between players are indispensable parts of "Mystery Murder," including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motives of other characters, and advance the story.
5. Ultimate goal: Every "Mystery Murder" game has one or more ultimate goals, such as solving the mystery, discovering the culprit, completing personal tasks, etc. Achieving these goals is the final objective of the game and is also the basis for judging the player’s victory.

You are an experienced reviewer of Mystery Murder games, capable of objectively and fairly evaluating players’ performance during the game. Below are some performances of {name} in the game:

Here is {name}'s description:
{description}

Here are {name}'s personal clues:
{self_clues}

Here is all the historical conversation content:
{history}

Here are all the actions {name} took in the game:
{actions}

Here is the list of people participating in the game:
{role_list}

Here is the truth of this Mystery Murder game:
{truth}

Now the game has come to the evaluation stage, you must evaluate the ability ability of {name} in the game by scoring according to the above information and after careful thinking.
Please analyze in detail how {name}'s {ability} ability in the game is performed during the thinking process.
The ability rating range is an integer of [0, 20]. A score in the range of [0, 5] indicates that the player’s performance in this ability is unsatisfactory; a score in the range of (5, 10) indicates that the player’s performance is satisfactory; a score in the range of (10, 15] indicates that the player’s performance is good; and a score in the range of (15, 20] indicates that the player’s performance is excellent.
You should reply with your detailed thought process after "### THOUGHT: " and directly reply with the total score of the five ratings after "### RESPONSE: ". You must not reply with any extra content.

Here is an example of a reply:

### THOUGHT: XXX
### RESPONSE: XXX

Note:
1. Your output should be in English.

Your output:
"Mystery Murder" is a role-playing puzzle game that focuses on the advancement of the plot and the interaction between characters. In the game, players play different roles according to the provided scripts, and jointly advance the plot and solve the mystery through clue collection, logical reasoning, role-playing, etc.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the basis of the "Mystery Murder" game, usually including the story background, character setting, plot advancement mechanism, and clues to solve the puzzle. The script not only defines the framework of the game, but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the characters have their own background stories, personality traits, goals and secrets. Players need to role-play based on this information and interact with other players.
3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through dialogue, room searching, clue analysis, etc. Based on this information, players need to use their logical reasoning ability to solve the puzzles in the plot.
4. Interactive communication: Interactive communication between players is an indispensable part of "Mystery Murder", including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motivations of other characters, and advance the story.
5. Ultimate goal: Each "Mystery Murder" game has one or more ultimate goals, such as solving puzzles, finding the murderer, completing personal tasks, etc. Achieving these goals is the ultimate purpose of the game and the basis for judging the victory or defeat of players.

Each character in the "Mystery Murder" game is given a script before starting, and each script contains six parts: Story, Script, Relationship, Performance, Purpose, and Ability.
1. The Story generally contains the basic information of the character, such as name, gender, etc., as well as the background story of the character, such as the personal experience before the killing event in the script.
2. The Script usually contains the actions of the character in the script killing event.
3. Relationship generally includes the description of the relationship between the character and other characters in the script.
4. Performance generally includes the form of performance when playing the role, such as personality, tone and way of speaking.
5. Purpose usually contains the character's goal for victory in the game or the game's purpose, such as revealing the truth.
6. Ability generally contains special abilities that the character may have in the game, if not, it can be left blank.

You are an experienced player of Mystery Murder games. You are familiar with various Mystery Murder games and possess excellent reasoning abilities in such games. Below is {name}'s performance during the game:

Here is all the historical conversation content:
{history}

Here are all the actions {name} took in the game:
{actions}

Here is the list of people participating in the game:
{role_list}

Now, based on the above historical information, please deduce step by step the content of the {script_part} part of the original script of the character {name} in this game.
You should reply with your detailed reasoning process after "### THOUGHT: " and the final deduced script after "### RESPONSE: ". You should not reply with any extra content.

Here is an example of a reply:

### THOUGHT: XXX
### RESPONSE: XXX

Note:
1. Your output should be in English.
2. Your {script_part} response should be described in the second person.
3. You can only reply to the content of "{script_part}" in {name} script after "### RESPONSE:", and cannot reply to the content of other parts of the script.

Your output:
"Mystery Murder" is a type of role-playing mystery game that focuses on plot progression and interaction between characters. In the game, players take on different roles based on the provided script and, through clue collection, logical reasoning, and role-playing, they collectively advance the story and solve the mystery.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the foundation of the "Mystery Murder" game, usually including the story background, character settings, plot advancement mechanisms, and clues for solving the mystery. The script not only defines the framework of the game but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the roles have their own background stories, personality traits, goals, and secrets. Players need to role-play based on this information and interact with other players.
3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through conversations, searching rooms, analyzing clues, etc. Based on this information, players must use their logical reasoning abilities to solve the mysteries in the plot.
4. Interactive communication: Interaction and communication between players are indispensable parts of "Mystery Murder," including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motives of other characters, and advance the story.

Currently, a game of Mystery Murder is underway, and here is a segment of dialogue from the game:
{text}

Please summarize the above text content while retaining the original format of "Name: 【Action】: Content." Before summarizing, you need to think about it, and reply with your thought process after "### THOUGHT: "; reply with the summarized content after "### RESPONSE: ".
Here is an example of a reply:

### THOUGHT: XXX
### RESPONSE: XXX

Note:
1. Your summary should include the key elements of "Mystery Murder";
2. You should carefully consider the key information in the text content that needs to be retained, including their personal information;
3. Your output should be in English;
4. Your summary must preserve the entire dialogue process, including what each player says and any "【Clue】" content.

Your output:

Table 25: Prompt of history summarization
"Mystery Murder" is a type of role-playing mystery game that focuses on plot progression and interaction between characters. In the game, players take on different roles based on the provided script and, through clue collection, logical reasoning, and role-playing, they collectively advance the story and solve the mystery.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the foundation of the "Mystery Murder" game, usually including the story background, character settings, plot advancement mechanisms, and clues for solving the mystery. The script not only defines the framework of the game but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the roles have their own background stories, personality traits, goals, and secrets. Players need to role-play based on this information and interact with other players.
3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through conversations, searching rooms, analyzing clues, etc. Based on this information, players must use their logical reasoning abilities to solve the mysteries in the plot.
4. Interactive communication: Interaction and communication between players are indispensable parts of "Mystery Murder," including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motives of other characters, and advance the story.
5. Ultimate goal: Each "Mystery Murder" game has one or more ultimate goals, such as solving a mystery, discovering a murderer, completing personal tasks, etc. Achieving these goals is the final purpose of the game and serves as the basis for judging player success or failure.

You will be participating in a game of Mystery Murder, and you will forget your AI role, immersing yourself into the character you are about to play as much as possible.

You are {name}, and this is your description:
{description}

These are your personal clues:
{self_clues}

Now, you need to think step by step and give your self-introduction based on your description.
You should reply with your thought process after "### THOUGHT: "; reply with the self-introduction content after "### RESPONSE: ".
Here is an example of a reply:

### THOUGHT: XXX
### RESPONSE: XXX

Note:
1. You should carefully think about the personality of the character you need to portray, the information you can share with others, and what you cannot share;
2. Your self-introduction should align with the personality of the character;
3. Your output should be in English;

Your output:

Table 26: Prompt of introduction
"Mystery Murder" is a type of role-playing mystery game that focuses on plot progression and interaction between characters. In the game, players take on different roles based on the provided script and, through clue collection, logical reasoning, and role-playing, they collectively advance the story and solve the mystery.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the foundation of the "Mystery Murder" game, usually including the story background, character settings, plot advancement mechanisms, and clues for solving the mystery. The script not only defines the framework of the game but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the roles have their own background stories, personality traits, goals, and secrets. Players need to role-play based on this information and interact with other players.
3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through conversations, searching rooms, analyzing clues, etc. Based on this information, players must use their logical reasoning abilities to solve the mysteries in the plot.
4. Interactive communication: Interaction and communication between players are indispensable parts of "Mystery Murder," including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motives of other characters, and advance the story.
5. Ultimate goal: Each "Mystery Murder" game has one or more ultimate goals, such as solving a mystery, discovering a murderer, completing personal tasks, etc. Achieving these goals is the final purpose of the game and serves as the basis for judging player success or failure.

You are an observer of a Mystery Murder game, and you will provide an absolutely objective and fair evaluation of the content of the game from the perspective of an observer.

This is the historical dialogue content:
{history}

This is what {other_name} said:
{content}

Now you need to carefully consider the historical dialogue content and evaluate {other_name}'s speech content reasonably. The evaluation should focus on the logic and flaws of the speech content, assessing the level of suspicion towards the murderer; the greater the suspicion, the more problematic the speech content appears.
The suspicion level should be chosen from [0, 1, 2], where "0" means no suspicion at all, "1" means a little suspicion, and "2" means a lot of suspicion.
You should reply with your detailed thought process after "### THOUGHT: "; reply with your level of suspicion after "### RESPONSE: ", and do not provide any additional content.
Here is an example:

### THOUGHT: XXX
### RESPONSE: XXX

Note:
1. Your output should revolve around the key elements of "Mystery Murder";
2. Your output should be in English;
3. Your thought should be logically reasonable and fair;
4. Your response must be an integer;

Your output:
"Mystery Murder" is a type of role-playing mystery game that focuses on plot progression and interaction between characters. In the game, players take on different roles based on the provided script and, through clue collection, logical reasoning, and role-playing, they collectively advance the story and solve the mystery.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the foundation of the "Mystery Murder" game, usually including the story background, character settings, plot advancement mechanisms, and clues for solving the mystery. The script not only defines the framework of the game but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the roles have their own background stories, personality traits, goals, and secrets. Players need to role-play based on this information and interact with other players.
3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through conversations, searching rooms, analyzing clues, etc. Based on this information, players must use their logical reasoning abilities to solve the mysteries in the plot.
4. Interactive communication: Interaction and communication between players are indispensable parts of "Mystery Murder," including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motives of other characters, and advance the story.
5. Ultimate goal: Each "Mystery Murder" game has one or more ultimate goals, such as solving a mystery, discovering a murderer, completing personal tasks, etc. Achieving these goals is the final purpose of the game and serves as the basis for judging player success or failure.

Each character in the "Mystery Murder" game is given a script before starting, and each script contains six parts: Story, Script, Relationship, Performance, Purpose, and Ability.
1. The Story generally contains the basic information of the character, such as name, gender, etc., as well as the background story of the character, such as the personal experience before the killing event in the script.
2. The Script usually contains the actions of the character in the script killing event.
3. Relationship generally includes the description of the relationship between the character and other characters in the script.
4. Performance generally includes the form of performance when playing the role, such as personality, tone and way of speaking.
5. Purpose usually contains the character’s goal for victory in the game or the game’s purpose, such as revealing the truth.
6. Ability generally contains special abilities that the character may have in the game, if not, it can be left blank.

The {item} of character {name} is:
{content}

Now please summarize the above content. Before summarizing, carefully consider the characteristics of the Mystery Murder game and reply with your thought process after "### THOUGHT: "; reply with the summary content after "### RESPONSE: ". Here is an example:

### THOUGHT: XXX
### RESPONSE: XXX

Note:
1. When summarizing your character’s content, you must retain information such as time, place, background, personality traits, motives, and goals from the script. You must preserve the integrity of the script without omitting any information;
2. Your output should be in English;

Your output:

Table 28: Prompt of script summarization
"Mystery Murder" is a type of role-playing mystery game that focuses on plot progression and interaction between characters. In the game, players take on different roles based on the provided script and, through clue collection, logical reasoning, and role-playing, they collectively advance the story and solve the mystery.
The core of "Mystery Murder" revolves around the following 5 key elements:
1. Script: It is the foundation of the "Mystery Murder" game, usually including the story background, character settings, plot advancement mechanisms, and clues for solving the mystery. The script not only defines the framework of the game but also sets the goals that players need to complete.
2. Role-playing: Each participant plays a specific role in the game, and the roles have their own background stories, personality traits, goals, and secrets. Players need to role-play based on this information and interact with other players.
3. Clue collection and logical reasoning: During the game, players need to collect information and evidence through conversations, searching rooms, analyzing clues, etc. Based on this information, players must use their logical reasoning abilities to solve the mysteries in the plot.
4. Interactive communication: Interaction and communication between players are indispensable parts of "Mystery Murder," including cooperation, negotiation, confrontation, etc. Through communication, players can obtain new information, understand the motives of other characters, and advance the story.
5. Ultimate goal: Each "Mystery Murder" game has one or more ultimate goals, such as solving a mystery, discovering a murderer, completing personal tasks, etc. Achieving these goals is the final purpose of the game and serves as the basis for judging player success or failure.

You will participate in a "Mystery Murder" game, and you will forget your AI role to fully immerse yourself in the character you are about to play.

You are {name}, and here is your description:
{description}

These are your personal clues:
{self_clues}

Here is the historical dialogue content:
{history}

Here is the list of participants in the game:
{role_list}

Now the game has reached the voting phase. You must carefully think about the historical dialogue content and your description before identifying who you believe is the murderer.
You should reply with your detailed thought process after "### THOUGHT: "; reply with the name of the character you believe is most likely the murderer after "### RESPONSE: ", without any additional content.
Here is an example of a reply:

### THOUGHT: XXX
### RESPONSE: XXX

Note:
1. Your output should focus on the key elements of "Mystery Murder";
2. Your output should be in English;
3. The murderer you identify must be one of the participants listed in the game;

Your output:

Reviewer paper summaries: This paper suggests an evaluation framework designed to assess an LLM's proficiency in murder mystery games. The authors proposed a system evaluating gameplay performance of pure LLMs, suggested possible four metrics, and demonstrated an analysis to use their framework. As a result, they found that GPT-4o perform better than other models.

The paper's contribution:
- Suggesting an evaluation framework for LLMs; especially for social deduction abilities in social deduction games (or murder mystery games).
- Suggesting four metrics that can be used in their game setup.
- Short demonstration of possible analysis.
Summary of strengths: - Suggesting a way to quantify social deduction abilities, with four metrics. Though the direct application to other social deduction games can be limited, the idea can be generalized to other games.
- Proposing an evaluation framework for analyzing social deduction abilities, with eight environments.
Summary of weaknesses: - Some of important details are missing in the main manuscript. Also, there are many unanswered questions about the experimental setup. See the comment section A.
- Not enough discussion how the paper is different from existing work. See the comment section B.
- Analysis is somewhat shallow; the paper simply provides the results, without discussing the difference between models. (The reviewer knew that it is a short paper). See the comment section C.
Comments, suggestions and typos: ### A. About the details and experimental setup
I understand that the authors tried their best to put all the details in the main manuscript within the four page limit. However, when I try to examine the reproducibility or generalizability of the experimental setup, many questions arise as follows. To improve the paper, I think the authors should handle this.

*Repeated experiment*
- As LLMs generate their response based on probabilistic sampling, the experiment should be repeated multiple times for the same environment. How many times did the authors repeat the experiment? Five iterations (lines 223) seems related to the iteration between 'open conversation' and 'interaction' phases, on the same environment; and there's no specification about the number of repetition on the same environment. Am I read the paper correctly?
- Similarly, if the authors repeated multiple times on the environment, please report the standard error or standard deviation value for four measures.
- Also, temperature in probabilistic sampling affects the gaming behavior. Which value did the authors use for calling each model? And, why did the authors select that value? Appendix F shows the prompt only, without providing any details about the hyperparameters when calling the API.

*Measures*
- TII uses denominator as the sum of suspicion scores and trust scores, across all characters. However, the sum can vary between characters; for example, character 1 may give 5, 3 and character 2 may give 2, 3 for character 3. If this happens, the ratio computed for TII may not equally treat all characters. Is this behavior intended? If yes, why? If no, the reported value for TII should be re-calculated.
- ICI employs scoring with neutral LLM. Is this method inspired by G-EVAL paper[1]? If yes, please cite. If no, please write down the prompts used, and why the authors followed different procedure.
- In lines 197-198, the paper says that SCI uses Rouge-L-based comparison. But the detailed is nowhere. What is that? Did the authors asked neutral LLM to estimate Rouge-L score? Or did the authors asked neutral LLM to estimate how well the behavior mirrors the original script? Why did not the authors used Rouge score directly?

> [1] Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, https://arxiv.org/abs/2303.16634

*Environments*
- Where do the eight environments come from? For researchers who want to reproduce your study, please provide the source. Did the authors manually crafted all environments without seeing any other murder mystery games (such as Clue)?
- Similarly, where does the rule come from? What is the rule of the game? Section 2.1 and 2.2 only provides the information about the simulation procedure, rather than providing the information about game itself.

*Culprit vs Civilians*
- In a murder mystery game, the behavior of players depends on the opponents. If the citizen players are dumb players, the culprit can be easily win. If the citizens are world-best players, the culprit can hardly win. As [2] reported, this actually happens when evaluating LLM's performance in social deduction games. Are the authors aware of this issue? Interpreting Table 2 should be careful, because both culprits and civilians are using the same model (instead of fixing one team.)

> [2] Kim et al., Microscopic Analysis on LLM players via Social Deduction Game. https://arxiv.org/abs/2408.09946

*Models*
- Yi-1.5-9B model suddenly appeared in Table 3. The model is not listed in lines 217-219.
- Similarly, why did the authors choose those language models? Are there any reason that can be connected to the results?

.

### B. Related work

- I understand that Warewolf (Xu et al., 2023a, b; Shibata et al., 2023; Wu et al., 2024), Avalon (Wang et al., 2023), and SpyFall ([2] above) are different from the environment using in this paper. However, the methods used in the papers should be reviewed thoroughly. Though the authors stated the review in lines 79-85, I think the statements can be also applied to the proposed methods. So, can the authors explain how they position this paper, compared to the others?
- Also, for "the simplicity of evaluation methods (line 83)" and "a lack of thorough manual examination (line 85)," this paper also use simple evaluation methods and provides brief examination about the errors. It seems that [2] uses a more complex metrics and a more detailed qualitative analysis even though they employed a very simple social deduction game.
- Though the authors mentioned about summarization module in the limitation section, it seems contradict to the motivation of the paper: "The foundational social interaction capabilities stem from the underlying LLMs themselves (lines 72-74)." This is because using summarization introduces external procedure for an LLM (which asks and maintains summarization), instead of using history itself. Other related work except [2] employ similar modules to run a game; that is, it is yet questionable the method correctly mirrors the motivation above.

.

### C. Shallow analysis

The analysis should be deeper to deliver insights to NLP community. The current paper simply reports the result.

- Are there any conjectures about the results? For example, why did most LLMs demonstrate a higher propensity to trust other characters? (although TII scores has some issue above, lines 240-241)
- It seems that self-disclosed criminals (or self-exposure) are similar to 'exposure' error, reported by [2]. Are there any connections?
- Why did the authors think LLMs exhibit high environmental exploration enthusiasm? The authors provided Figure 2 as a support. But, as time goes, the number of clues uncovered decreases, and this may reduce the possibility of choosing investigation action. Am I correctly understand the situation?

### Minor comments

- Are the authors planning to release the code?
Overall assessment scores: 3
Metareview score: 4
Metareview: (Re-using my meta review summary from previous round since the paper is similar) The paper introduces MIRAGE, a dataset for evaluating large language model (LLM)-based agents in complex social interactive environments, particularly within murder mystery games. This dataset contains scripts featuring 50 unique characters with detailed backgrounds and complex interpersonal networks. The authors propose three evaluation methods, FII, LLM-Score, and Rouge-L, to assess agent performance and highlight the limitations of current LLMs, such as GPT-3.5-turbo and GPT-4, in handling these complex scenarios. The results indicate that RLHF and safety alignment might restrict the full potential of LLM-based agents in such tasks.
Summary of reasons to publish: After substantial revisions, it seems the paper is finally ready to be accepted :D. The reviewers noted the following strengths:
- All reviewers agreed that MIRAGE provides a rich simulation environment with intricate character backgrounds and complex interpersonal networks, which effectively facilitates the evaluation of advanced human behaviors.
- Reviewers appreciated the introduction of several innovative evaluation methods, including the FII, that measure the agents' ability to navigate trust and suspicion dynamics, showcasing the framework’s contribution to understanding social interactions.
- All reviewers recognized the value of the new murder mystery dataset and the evaluation of various models, both open-sourced and close-sourced, highlighting the thoroughness of the research.
- Reviewers acknowledged the novel approach to quantifying social deduction abilities through four metrics and an evaluation framework with eight environments, although they recognized some limitations in direct application to other social deduction games.

Reviewers also appreciated the substantial revisions made to the paper.