LLMARENA: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments (2024)

Abstract: Recent advancements in large language models (LLMs) have revealed their potential for achieving autonomous agents possessing human-level intelligence. However, existing benchmarks for evaluating LLM Agents either use static datasets, potentially leading to data leakage, or focus only on single-agent scenarios, overlooking the complexities of multi-agent interactions. There is a lack of a benchmark that evaluates the diverse capabilities of LLM agents in multi-agent, dynamic environments. To this end, we introduce LLMARENA, a novel and easily extensible framework for evaluating the diverse capabilities of LLM in multi-agent dynamic environments. LLMARENA encompasses seven distinct gaming environments, employing TrueSkillTM scoring to assess crucial abilities in LLM agents, including spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. We conduct an extensive experiment and human evaluation among different sizes and types of LLMs, showing that LLMs still have a significant journey ahead in their development towards becoming fully autonomous agents, especially in opponent modeling and team collaboration. We hope LLMARENA could guide future research towards enhancing these capabilities in LLMs, ultimately leading to more sophisticated and practical applications in dynamic, multi-agent settings. The code is available in https://github.com/THU-BPM/LLMArena.

heading: 1 Introduction
text: Recent large language models (LLMs) have greatly promoted the progress of the field of natural language processing (NLP) due to their excellent zeroshot capabilities in various downstream tasks (Kojima et al., 2022; Yang et al., 2023; Touvron et al., 2023a,b; OpenAI, 2022, 2023). LLMs not only excel in comprehending and generating intricate text
∗ Work was done during the intern at 4Paradigm Inc.. † Corresponding authors.
but also demonstrate remarkable adaptability, even without training tailored to specific tasks (Wei et al., 2023; Guo et al., 2023a). They possess the ability to analyze, strategize, and respond effectively to unfamiliar scenarios using minimal guiding prompts. This has opened new avenues for researchers to consider LLMs as autonomous agents, providing automated assistance in complex real-world tasks such as software development and knowledge integration(Qian et al., 2023; Pan et al., 2024).
To better understand the skills needed by LLM as an agent, researchers are now concentrating on creating and using various scenarios to assess LLM’s performance in specific settings (Liu et al., 2023b; Chen et al., 2023b; Wu et al., 2023; Li et al., 2023a; Yao et al., 2022). For instance, AgentBench introduced eight distinct environments, such as operating systems and databases, to examine LLM’s proficiency in code generation and lateral thinking. Similarly, SmartPlay tests LLM in a gaming context as a solo agent to gauge its reasoning and strategic planning abilities. Nonetheless, these benchmarks have their constraints. Firstly, static datasets used in benchmarks like AgentBench might lead to issues such as data leakage and overfitting, as LLMs could have already encountered this data during their pre-training phase (Zhou et al., 2023b). Secondly, evaluation systems that focus solely on a single agent also have evident limitations, particularly in analyzing an agent’s performance in intricate and dynamically interactive settings. Such single-agent environments tend to concentrate on the interactions between the agent and its environment, thereby overlooking the complex dynamics of interactions that occur among multiple agents in a shared environment (Foster et al., 2023). To this end, we proposed LLMARENA, a dynamic evaluation benchmark specially designed for multi-agent interaction. As illustrated in Figure 1, LLMARENA covers seven different types of dynamic, multi-agent game environments. For
13055
example, the Texas Hold’em poker game environment automatically generates a new hand for each game, and the complexity of each round’s state increases exponentially. This dynamic nature of the game serves to mitigate the issue of data leakage. In the Undercover environment, the LLM agents need to point out the “undercover” during the communication process with other agents, which measures the LLM agents’ communication ability and opponent modeling ability when facing different agents. Based on these environments, we can comprehensively evaluate LLM’s spatial comprehension, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration capabilities in the dynamic multi-agent environment.
To more accurately evaluate LLM’s performance in these multi-agent environments, we adopted the TrueSkill™ (Herbrich et al., 2006) scoring system, which can better help LLMARENA evaluate other metrics besides win rate, such as the relative skill level between agents, to provide a more comprehensive assessment under different scenarios and opponent configurations.
We conduct an extensive experiment and human evaluation among 14 different sizes and types of LLMs. The results indicated significant potential for improvement in LLM’s teamwork and opponent modeling skills within multi-agent environments. Based on these results, we hope LLMARENA can inspire future research focused on enhancing the core capabilities of LLM in multi-agent settings to promote the widespread use of LLM agents in real-world applications.

heading: 2 Benchmark Detail
text: In this section, we provide a detailed overview of LLMArena’s composition, which includes its seven distinct game environments, evaluation metrics, and the methodologies used for assessment.

heading: 2.1 Benchmark Overview
text: Figure 1 illustrates the seven distinct environments within LLMARENA. Within each specific environment, LLMs may need to utilize a unique combination of capabilities to effectively tackle these challenges. For instance, in the Undercover environment, LLMs are required to possess a mul-
tifaceted skill set, including opponent modeling, communicating effectively, and team collaboration. The absence of any one of these abilities could result in ultimate failure in this environment.

heading: 2.2 Benchmark Construction
text: To ensure user-friendliness and robust scalability in LLMARENA, we have developed the environment using PettingZoo (Terry et al., 2021) as a foundation. This approach enables subsequent researchers to effortlessly integrate new environments into this framework. By simply creating environments that adhere to the PettingZoo interface specifications1, they can seamlessly add these to LLMARENA for evaluating the diverse capabilities of LLMs. Every environment within LLMARENA offers system prompts that outline the game rules and prompt templates designed to guide LLM agents in gameplay. These templates include information on the current status of the game, historical data, and a range of optional actions, facilitating a more structured and informed gaming experience for the LLM agents. All the prompts are shown in Appendix A.

heading: 2.3 Metrics
text: In this section, we outline the evaluation metrics used by LLMARENA, followed by a comparative analysis with metrics utilized in prior research.
TrueSkill™ TrueSkill™ is a scoring system developed by Herbrich et al. (2006) that evaluates multiple agents’ skill levels in competitive gaming. In contrast to static, opponent-independent metrics employed in previous research, such as Win Rate and Completion Rate (Wu et al., 2023; Xu et al., 2023), TrueSkill™ offers a more nuanced assessment. It goes beyond merely accounting for wins and losses by also considering the quality of the game and the skill disparities between players. Within the TrueSkill™ framework, defeating higher-skilled opponents yields more points compared to victories over lower-skilled adversaries. This approach stands in stark contrast to simple win rate calculations, which do not account for the skill level of the opponent. Such opponent-related metrics provide an accurate evaluation of an agent’s real competencies in a multi-agent environment.
Reward Currently, a substantial amount of research employs expert LLMs to assess the capabilities of other LLMs, yet this approach frequently
1https://pettingzoo.farama.org/ content/environment_creation/
lacks interpretability (Wang et al., 2023a). LLMARENA addresses the gap by leveraging the concept of reward from reinforcement learning to measure the quality of each action performed by agents. The method allows for a quantitative and interpretable evaluation of LLMs’ capabilities, offering a more nuanced and understandable analysis.

heading: 2.4 Environment Settings
text: In this section, we give a comprehensive explanation of the diverse game environments included in the LLMARENA benchmark, highlighting the specific challenges that language models encounter while engaging with these environments.
TicTacToe: TicTacToe2 is a strategy game where two LLM agents alternate in placing their marks on a 3×3 grid. Victory is achieved when one agent successfully aligns three of their marks in a row, either horizontally, vertically, or diagonally. The game results in a draw if all cells are filled without any player achieving a winning alignment. This game examines LLM’s strategic planning and spatial reasoning capabilities. We adopt the TrueSkill™ ratings as the evaluation metrics.
ConnectFour As a more complex case of the board game, ConnectFour3 uses a 6×7 chessboard. Here, two LLM agents alternately choose one of the seven columns with available empty cells. Each piece placed will descend to the lowest available space in the selected column, either resting at the bottom or atop a previously placed token. An agent secures victory by aligning four of their pieces either horizontally, vertically, or diagonally. The game concludes in a draw if all cells are filled without a winner. Similar to TicTacToe, in ConnectFour, the LLMs primarily encounter challenges in strategic reasoning and spatial reasoning. We adopt the TrueSkill™ ratings as the evaluation metrics.
Texas Hold’em Texas Hold’em4 is a well-known card game. In LLMARENA, each game is played by two LLM agents. At the beginning of each game, players are dealt two private cards, known as hole cards. Subsequently, five community cards are dealt across three stages - the flop, the turn, and
2https://pettingzoo.farama.org/ environments/classic/tictactoe/
3https://pettingzoo.farama.org/ environments/classic/connect_four/
4https://pettingzoo.farama.org/ environments/classic/texas_holdem_no_ limit/
the river. Players aim to construct the best possible five-card hand using a combination of their hole cards and the community cards. Throughout the game, there are four betting rounds. In each round, players have the option to choose from the following actions: Fold, Check & Call, Raise Half Pot, Raise Full Pot, and All-In. This game environment demands a range of capabilities from LLMs, including numerical reasoning, opponent modeling, and risk assessment. We adopt the TrueSkill™ ratings as the evaluation metrics.
Undercover Undercover is a party game, where players are divided into two groups: undercover and non-undercover. Two groups of players each receive a pair of similar but distinct words. The game has two stages per round: a communication stage where players describe their words, followed by a voting stage to identify an undercover. At this time, if all undercovers are eliminated, the nonundercovers win, otherwise the next round will proceed. In LLMARENA, 5 LLM agents play in each game, which includes four non-undercover agents using GPT-3.5-Turbo and one undercover LLM undergoing evaluation. The game lasts up to two rounds, and if the undercovers remain undetected, they win. Given the superior capabilities of GPT-4, employing it as the non-undercover agent would render it nearly impossible for other models to win. Therefore, we choose GPT-3.5-Turbo as non-undercover agents. This game tests an LLM’s proficiency in communication, opponent modeling, and team collaboration. The evaluation metric used here is the winning rate of each LLM when playing as the undercover against GPT-3.5-Turbo.
Bargain Bargain (Lewis et al., 2017) is a semicooperative game where two LLM agents are required to devise a distribution strategy for a pool of items. Each item holds a distinct, undisclosed value for both agents. The game aims to reach a consensus on the allocation. When an agreement on the allocation is achieved, the agent with the maximum total value from the items secured emerges victorious. Bargain requires LLM to have the ability of numerical reasoning, communication, and opponent modeling. We adopt the TrueSkill™ ratings as the evaluation metrics.
Bid A first-price sealed-bid auction5 is a widely used auction format in which all participants place
5https://en.wikipedia.org/wiki/ First-price_sealed-bid_auction
bids without knowledge of each other’s bids, and the highest bidder wins. In LLMARENA, two LLM agents are participating in each auction. Each agent receives a random inner valuation of the item. The winning LLM in the auction will receive their inner valuation minus the actual bid as a reward. This environment evaluates the LLM’s numerical reasoning and opponent modeling abilities. We measure the LLM’s performance using the average reward it achieves as the evaluation metric.
Hanabi Hanabi is a cooperative card game wherein two LLM agents can only view each other’s hand of cards. They use info tokens to “reveal cards” to each other, “discard cards” to acquire more information tokens, and “play cards” in a certain order to form fireworks. This requires team collaboration, strategic planning, and numerical reasoning skills from the LLMs. Upon successfully setting up a firework, both LLM agents are rewarded with points equivalent to the sum of the card values in the firework. In LLMARENA, we form the same two LLMs into a cooperation team, and we employ the average reward obtained by various LLM teams as an evaluation metric.

heading: 3 Experiments

heading: 3.1 Experiments Setup
text: We conduct experiments with the closed-source GPT family LLM by directly calling its API. As for the open-source LLM, we deploy it locally and encapsulate it as an OpenAI API call. To ensure reproducibility, we set the temperature of all LLMs to 0. In all environments except Undercover, we conducted LLMARENA runs until the TrueSkill™ ratings between models reached convergence, with over 50 games played per environment per model. For the Undercover environment, we played 100 games of each LLM as an undercover agent in a game with four GPT-3.5-turbo as non-undercover agents. Initially, all LLM agents started with a TrueSkill™ score of 25.0 and a variance of 8.33.

heading: 3.2 Main Result
text: In Table 1, we present the relative normalized scores for 14 different LLMs across 7 environments within LLMARENA. The original data before normalization can be found in Appendix C. This table allows us to draw the following conclusions. • As the scale of the model parameters increases, there is a noticeable improvement in the capabilities of LLMs. For instance, LLMs with around
70B parameters exhibit an average performance of 82.87. LLMs with approximately 30B and 10B parameters average 80.68 and 71.05 in performance, respectively. It’s noteworthy that the performance enhancement observed when escalating the model size from 10B to 30B parameters (+9.63) is more pronounced than the improvement seen from 30B to 70B parameters (+2.19). • While it’s generally observed that LLMs with a larger scale of parameters tend to outperform their counterparts with fewer parameters within the same series, exceptions can arise in specific environments. A case in point is the performance of Qwen-72B in the Undercover environment is surprisingly 25.93 points lower than that of Qwen14B, which notably diverges from this trend.
• The performance of LLMs in both the Bid and Hanabi environments significantly lags behind the average of all 7 environments, with deficits of 17.72 and 10.88 points, respectively. Smaller-scale LLMs (∼10B) fared even worse, scoring 31.71 and 15.45 points below the average in these environments. This underscores that tasks involving numerical reasoning, opponent modeling, and team collaboration pose significant challenges for LLMs. • There remains a significant disparity between GPT-4 and other models. GPT-4 has attained SOTA performance across all evaluated tasks, outperforming other models by an average margin of 16.76.

heading: 4 Analysis
text: In this section, we will conduct a thorough analysis of the experimental results, evaluating LLMs from seven key perspectives: spatial comprehension,
strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. We chose to experiment with 7 LLMs that demonstrate superior performance.
Spatial Comprehension To investigate the spatial comprehension capabilities of LLMs, we conducted 100 self-play games with 7 different LLMs each, recording winning rates and the frequency of hallucinations (instances of illegal moves). Simultaneously, we eliminated the prompts regarding the positioning of the pieces in the LLMs’ input and executed another set of 100 self-games, again tallying the winning rates and the occurrences of hallucinations. The results are shown in Table 2, which indicates that in both scenarios, the absence of positional cues significantly escalates the likelihood of LLMs experiencing hallucinations, with an average increase of 59.5%. Consequently, this results in a substantial reduction in their win rates, which on average decline by 38.3%.
Through a meticulous case study, we discovered that LLMs demonstrate a limited grasp of the vertical dimension on 2D chessboards. This limitation likely arises because we transform the 2D chessboard into a 1D string input for the LLMs via line breaks. Grasping the vertical concept requires the LLM to correlate two tokens separated by a specific distance in the one-dimensional input. Consequently, even though ConnectFour’s chessboard is larger and its state space more complex, TicTacToe’s 2D action space demands a higher level of spatial comprehension from the LLM. Hence, in the absence of positional prompts, the performance deterioration in TicTacToe is more pronounced. To
verify our conclusion, we swapped the rows and columns of the chessboard depicted in the prompt. We also tested other prompt strategies, such as selfconsistency and self-refinement. Table 3 shows the results of ChatGPT using different prompt strategies in the ConnectFour environment. We observe that row-column swapping led to a noticeable improvement in win rates for the LLMs. Compared to the original unswapped prompt setup, we observed an increase in win rate to approximately 62%.
Additionally, we explored other variations in our prompts, as summarized in the table above. While enhancements in self-refinement and selfconsistency were modest, the transformation achieved by swapping rows and columns resulted in a significant performance boost. This improvement can likely be attributed to the alignment of a column’s contents in a more contiguous manner on the token level after the transformation, facilitating a better understanding for the LLM. A case study can be found in Appendix B.1
Strategic Planning To delve into the strategic planning capabilities of LLMs within a multi-agent environment, we crafted a specialized board valuation function tailored for the ConnectFour environment as follows:
V (s) = 10× (My4(s)−Oppo4(s)) + 5× (My3(s)−Oppo3(s)) + 2× (My2(s)−Oppo2(s)),
(1)
where s represents the board status, My4(s), My3(s), and My2(s) represent the count of four, three, and two consecutive pieces of the player on the chessboard, respectively. Conversely, Oppo4(s), Oppo3(s), and Oppo2(s) represent the similar formations for the opponent’s pieces . We defined the reward as the change in board valuation between the state after the agent’s move s′ and the state following the previous move s:
R(a, s, s′) = V (s′)− V (s), (2)
where a represents the LLM agent’s action. We conducted random battles between seven LLM agents until the rewards converged, and the results are shown in Figure 2. It is evident that LLMs with a larger number of parameters consistently demonstrate superior strategic planning abilities. Each decision made by these LLMs propels them towards a more advantageous position, as indicated by the predominantly positive rewards they receive on average. Conversely, LLMs with a smaller parameter scale tend to earn, on average, negative rewards. This suggests an inability to accurately assess the current state of the chessboard and to formulate effective strategies, ultimately leading the game to progress in a direction that favors its opponent. To further explore the gap between LLM’s strategic planning capabilities and humans, we manually played five games with seven LLMs each and
human players emerged victorious in every game. Simultaneously, we observe that even GPT-4 rarely prioritizes reward maximization (such as turning a blind eye to winning positions). This tendency could stem from the fact that LLMs predominantly base their decisions on textual data. Consequently, the model must translate the game state into text descriptions before making decisions. This process of conversion may result in information loss, affecting the model’s strategic performance. An error analysis can be found in Appendix B.1
Communication To explore the communication capabilities of LLM, we followed the settings in Section 2.4 and manually analyzed 50 game records for each LLM, calculating the percentage of their in-game clues that accurately described their secret word (“Description” in Table 4). As illustrated in Table 4, LLMs demonstrated outstanding proficiency in this task. All LLMs achieved a description success rate of 95.30% on average, with four of them reaching a perfect 100% success rate. This high level of performance can be attributed to the LLMs being trained on extensive textual datasets, which equip them with a nuanced understanding of word associations and contextual meanings, allowing them to more easily delineate the meanings of different words. However, we also observed that receiving information poses a greater challenge for LLMs compared to giving information. When interpreting messages from other LLMs, they often make cognitive errors, leading to misinterpretations or distortions of the original clues. This may be because their interpretations are often based on their knowledge base rather than relying entirely on input information. This can lead them to misinterpret the intent or context of the original message, thus tampering with the original clue, which shows that LLMs still have limitations in handling complex, multi-level communications. A case study can be found in Appendix B.2
Opponent Modeling Moreover, the Undercover environment demands that LLMs possess not only communication skills but also the capability to model opponents. Building upon the settings established in Section 2.4, we further tasked each LLM agent with deducing the secret words of others during the voting phase. We recorded the instances where each LLM accurately guessed the secret word of any one agent (“Guess” in Table 4), as well as the occasions where they successfully identified the secret words of all other agents (“Guess (strict)” in Table 4). The results are shown in Table 4, where we can find that inferring the secret words of others through their descriptions is more challenging than describing oneself. Through manual analysis, we discovered that only GPT-4 possesses the analytical and reasoning skills necessary to identify itself as an undercover agent. Subsequently, it adjusts the clues it provides in the following round for more effective camouflage. This advanced ability to model opponents also enables GPT-4 to more accurately guess the secret words of all other agents, resulting in superior performance.
Numerical Reasoning Bid represents a typical problem in incomplete information games. In such scenarios, the Bayesian Nash equilibrium for two rational agents is theoretically set at v2 , where v denotes each agent’s valuation of the item (Nis, 2007). We study the numerical reasoning ability of LLMs in this situation. We play 100 self-games of seven types of LLMs in the Bid environment, and count the difference between their bids v′ and the Nash equilibrium v2 , and calculate the normalized score score = v
′−v/2 v/2 . The results are shown in the
Figure 3. The data presented in the figure allows us to deduce that nearly all LLMs, except Deepseek7B, tend to place bids that exceed the Nash equilibrium, with their median bids being on average 52% higher. Notably, Deepseek-67B exhibits an extreme behavior: it opts for bids that significantly sacrifice its profits, ostensibly to guarantee a win in
the auction. Consequently, its overall performance in this particular environment is notably subpar as illustrated in Table 1.
Team Collaboration As outlined in Section 2.4, within the Hanabi environment, LLMs are required to alternate between “discarding cards” and “revealing cards” to cooperate to obtain more information, and then build fireworks through “play cards”. To investigate the efficacy of LLMs in executing team
cooperation, we analyzed the frequency of each type of action performed by LLMs. As shown in Figure 5, although most LLMs frequently utilize the “reveal cards” action to share hand information with their opponent, they seldom use the “discard cards” action, which would give the opponent a chance to “reveal cards” for more information, thus rashly choosing “play cards” when there is insufficient information, causing the game to fail.

heading: 5 Related Works
text: LLM Evaluation With the rapid development of LLMs, the evaluation practices of traditional NLP tasks (Hu et al., 2023b; Bang et al., 2023; Zhong et al., 2023; Qin et al., 2023; Hu et al., 2022) limit their performance. Instead, the use of humanwritten exam questions as an assessment paradigm (Hu et al., 2023a; Hendrycks et al., 2021; Liang et al., 2022; Srivastava et al., 2023) has become increasingly mainstream, focusing primarily on testing the world knowledge and reasoning capabilities in a multi-task benchmark. In addition, some newly proposed benchmarks and methods examine large language models from multiple dimensions, such as adaptability to professional fields (Xiang et al., 2023), real-world application (Li et al., 2023b; Liu et al., 2023a, 2024b,a), robustness (Zhu et al., 2023; Hu et al., 2024), multi-modal capabilities (Fu et al., 2023), etc. Recently, the trend of treating large language models as agents has become increasingly evident in academia, and a series of studies (Liu et al., 2023b; Wu et al., 2023) aimed to evaluate the performance of these models when acting as agents in games. However, most of these studies are limited to a single agent, thus failing to fully capture the characteristics of large language models in game or cooperative scenarios and group behavior.
To highlight the distinctions between our work
and previous efforts, we provide a comparison in Table 5, where “MultiAgent” indicates whether the environment involves multiple LLM agents, “Opponent” denotes if there is an LLM opponent for adversarial interaction within the environment, “Expandable” represents whether there is a unified standard for expanding the environment and “Dynamic” signifies if the environment is random and dynamic, as opposed to a static dataset.
LLM-based Agents Reinforcement learning has been extensively employed in the training of autonomous agents (Ribeiro, 2002; Isbell et al., 2001; Mnih et al., 2013). With the superior capabilities of LLMs, the utilization of LLMs as cognitive entities and controllers for agents has garnered widespread acclaim (Huang et al., 2022; Park et al., 2023; Sumers et al., 2023). Wei et al. (2022) initially introduced the CoT technology, which notably amplifies the reasoning and planning abilities of LLMbased agents, consequently leading a research surge dominated by reasoning strategies (Yao et al., 2023; Wang et al., 2023c; Zhou et al., 2023a). Furthermore, empirical studies indicate that collaborative and adversarial frameworks (Li et al., 2023a; Chen et al., 2023b; Chan et al., 2023) involving multiple LLMs surpass the capabilities of singular agents across various tasks and may lead to the emergence of social phenomena (Park et al., 2023).

heading: 6 Conclusion
text: In our study, we introduced LLMARENA, a benchmark designed to assess the diverse abilities of LLM agents in dynamic, multi-agent environments. Through comprehensive analysis of seven game environments that require necessary abilities for
LLM agents, our findings reveal that LLMs exhibit weaknesses in spatial reasoning, opponent modeling, and team collaboration. Enhancing the performance of LLM agents in dynamic multiagent settings remains an unresolved challenge. We anticipate that future researchers will utilize LLMARENA to conduct evaluations in a broader range of scenarios.

heading: Limitation
text: While our research primarily focuses on assessing the capabilities of LLM agents within dynamic multi-agent environments, it is important to acknowledge two limitations. Firstly, in the journey towards achieving autonomous agents, an LLM agent’s ability extends beyond merely analyzing textual information. It necessitates engaging with a variety of modal inputs, including video and audio. The capabilities of LLMs within such multi-modal dynamic contexts remain an untapped area of exploration. Secondly, our study did not delve into the potential of LLMs that leverage external tools. We anticipate and encourage further research in both of these promising dimensions to broaden our understanding and capabilities of LLM agents.

heading: Ethical Considerations
text: The ethical landscape surrounding LLM agents is marked by significant challenges, particularly in the realms of responsible usage, and the potential for misuse. The deployment of autonomous LLM agents in decision-making roles raises questions about accountability and the need for robust frameworks to prevent unethical applications. The potential for misuse, such as manipulating scenarios or
exploiting systems, underscores the necessity for stringent guidelines and monitoring.

heading: Acknoledgement
text: This work is supported by the National Nature Science Foundation of China (No. 62021002), the Beijing Natural Science Foundation under grant numbers QY23115 and QY23116, Tsinghua BNRist, the Beijing Key Laboratory of Industrial Bigdata System and Application.

heading: A Complete Prompt Design
text: In this section, we provide complete prompts for all seven game environments in LLM Arena. As the agent in the LLM Arena must consistently interact with the environment, similar to the reinforcement learning setting, various prompts are required to guide the agent in comprehending the rules and taking action. Specifically, three types of prompts work:
• System Prompt. System prompts are special messages used to steer the behavior of LLM. They allow to prescribe the agent’s style and task within certain bounds, making it more customizable and adaptable for various use cases.
In LLM Arena, we stipulate the role of the agent and the rules of the game in the system prompt.
• Observation Prompt. Observation Prompt provides the necessary state information for the agent to interact with the environment, including the opponent’s actions, current game status, legal actions, etc. We use {} to frame these changing information.
• Action Prompt. Action Prompt guides the Agent to choose from legal actions and output regularized actions. In addition, to stimulate the agent’s reasoning ability, in some environments, we also add CoT Prompt.

heading: A.1 TicTacToe
text: In TicTacToe, the content in {} can be obtained from the interaction between the agent and the environment. For example, {player_type} includes "X" and "O", {board_status} represents the current chessboard, and we describe the entire chessboard in the form of text, as the example in System Prompt. {available} represents the current legal action list of the agent.
The detailed prompt template of TicTacToe is as follows.
System Prompt
You are playing TicTacToe. TicTacToe is played on a three-by-three grid by two players, who alternately place the marks X and O in one of the nine spaces in the grid. The player who succeeds in placing three of their marks in a horizontal, vertical, or diagonal row is the winner. In the following example, the first player (X) wins the game in seven steps: 1. [Player 1]: X: (1, 3)
2. [Player 2]: O: (1, 1)
3. [Player 1]: X: (3, 1)
4. [Player 2]: O: (2, 2)
5. [Player 1]: X: (3, 3)
6. [Player 2]: O: (2, 3)
7. [Player 1]: X: (3, 2)
X plays first. Players will specify the position of the stone and the moderator will plot the board status.
If a position has been marked, future marks cannot be put in the same position. The players interact with the game by specifying the position of the stones (x, y), where x indicates the row and y indicates the column, so (1, 1) is the top left corner and (3, 3) is the bottom right corner."
Observation Prompt
You play {player_type}. The board status is {board_status}. You can only put the mark on [{available}].
Action Prompt
You should only output {player_type} and the position of the move, for example: "{player_type}: (1, 3)"
The position you put the mark on must be empty. Don’t say anything besides mark position.

heading: A.2 ConnectFour
text: In ConnectFour, the observation prompt and action prompt are similar to TicTacToe. We also use text to describe the chessboard.
The detailed prompt template of Texas Hold’em is as follows.
System Prompt
You are playing ConnectFour. ConnectFour is played on a six-by-seven grid by two players, who alternately drop their makers X and O into one of the seven columns. Each marker will fall to the lowest available space within the selected column.
The player who succeeds in placing four of their makers in a horizontal, vertical, or diagonal row is the winner. X plays first. Players interact with the game by specifying the column number where they want to drop their marker. If a column is already full, players cannot drop a marker into that column.
The columns are numbered from 1 to 7, from left to right. Players cannot place a token outside this range.
Observation Prompt
You play {player_type}. The board status is {board_status}. You can only choose one of the following columns: [ {available} ].
Action Prompt
You should only output {player_type} and the column of the move you choose to put your mark, for example: "{player_type}: 1"
The column you put the mark on cannot be full. Don’t say anything besides mark position.

heading: A.3 Texas Hold’em
text: In Texas Hold’em, the status information of the environment mainly includes three types: hand cards({private}), community cards ({public}), and chips ({chips}).
The detailed prompt template of Texas Hold’em is as follows.
System Prompt
You are playing Texas Hold’em No Limit. Here are the basic rules for Texas Hold’em Poker: In this game, you are requested to attend 1 game with your opponent. Both players start with 100 chips, and the player with the most chips at the end of the game wins. If your chips drop to 0, you lose the game. The objective is to obtain more chips than your opponent.
Texas Hold’em hands are ranked from highest to lowest as follows: Royal Flush: A, K, Q, J, 10 all of the same suit. Straight Flush: Five consecutive cards of the same suit. The higher top card wins. Four of a Kind: Four cards of the same rank. The higher rank wins; if the same, compare the fifth card. Full House: Three cards of one rank and two cards of another rank. A higher three-card rank wins; if the same, compare the two-card rank. Flush: Five non-consecutive cards of the same suit. Compare the highest card, then the second-highest, and so on. Straight: Five consecutive cards of different suits. The higher top card wins. Three of a Kind: Three cards of the same rank. Higher rank wins. Two Pair: Two cards of one rank and two cards of another rank. Compare the higher pair first, the lower pair, and then the fifth card. One Pair: Two cards of the same rank. Compare the pair first, then the highest non-paired card, then the second highest, and so on. High Card: If no hand can be formed, the highest card wins. If the highest cards are the same, compare the second highest, and so on. If the hands are of equal rank, the pot is split.
Observation Prompt
Your observation now is: The cards in your hands are [{private}] The community cards are [{public}] You now have 100-chips chips, the chips that you have put in until now is {chips} Now you can choose one of the following actions:[{available}]
Action Prompt
You should think step by step and output your action. For example: ’Action: Check and Call’ You should explain why you chose the action

heading: A.4 Undercover
text: In Undercover, unlike other environments, it involves more than two agents, and the game has two stages. Therefore, we designed two prompts respectively for the Giving clues stage and the Accusation stage.
The detailed prompt template of Undercover is as follows.
System Prompt
You are playing a game of the Undercover. Here are the game rules: In Undercover, players are divided into two groups: undercovers and non-undercovers. Each player receives a secret word, but the secret word given to the undercover is different
from the rest. The main aim is to identify the undercover based on the clues provided by all players.
Be cautious with your clues; being too specific might reveal your identity. As a non-undercover, you need to unmask the undercover without revealing your own secret word. As an undercover, your task is to seamlessly integrate with others and avoid detection. Undercover is played in two phases: The Giving Clue Phase: Players give hints about their words over several rounds. The Accusation Phase: Players vote on who they think the undercover is. The undercover should vote for non-undercovers.
Observation Prompt
• Giving clues stage
Now is the giving clues stage:
You are {player_name}. Your secret word is "{word}".
This round the other players say: {messages}
• Accusation stage
Now is the accusation stage:
You are {player_name}. Your secret word is {word}.
This round the other players say:{messages}
Action Prompt
• Giving clues stage
Now you give one clue (but never include the secret word). You cannot repeat what other players have said.
You should only output {player_name} and the clue, for example: "{player_name}: It’s a fruit.".
• Accusation stage
Now vote which of the other players (excluding yourself) is undercover.
You should only output "vote" and the undercover name, for example: "vote: player_1.".

heading: A.5 Bargain
text: In bargain, both parties need to give a plan and bargain. The quantity and price of goods owned by the agent are represented by {item_nums} and {value} respectively, and the plan proposed by the agent is represented by {oppo_plan}
The detailed prompt template of Bargain is as follows.
System Prompt
You are participating in a game of "Bargaining." Here are the game rules: (1) Information and Roles The game involves two players, Player A and Player B. A variety of items are available for bargaining, each having a different value to both players. However, the value of each item for the other player is unknown.
Players engage in a series of up to 10 bargaining rounds to reach a consensus. If a consensus is not reached within these 10 rounds, the negotiation is considered a failure,
and it results in a stalemate where both players leave without any rewards. (2) Objectives The goal for both Player A and Player B is to negotiate and share these items. The challenge is that the same item has different values for each player, and they do not know the item’s value for their counterpart. Each player aims to maximize the total value of the items they acquire through negotiation.
Observation Prompt
(The game begins. You start to give a plan.) Now is round {round}: You are {player_name}. There are {item_nums[0]} hats, {item_nums[1]} balls and {item_nums[2]} apples. The value for each hat is {value[0]}, for each ball is {value[1]} and for each apple is {value[2]}. This round your opponent says:"{bargaining}" He will get {oppo_plan[0]} hats, {oppo_plan[1]} balls and {oppo_plan[2]} apples. And you will get {item_nums[0] - oppo_plan[0]} hats, {item_nums[1] - oppo_plan[1]} balls
and {item_nums[2] - oppo_plan[2]} apples.
Action Prompt
Do you agree with his plan? If you agree, You should only output {player_name} and "Deal", for example: "{player_name}: Deal.".
If you do not agree, you should bargain with your opponent first and then output {player_name} and your plan in the following format:
"{player_name}: x hats y balls z apples." x, y, z are Arabic number. For example: "I’d like 1 hat, 2 balls, and 0 apples. {player_name}: 1 hat 2 balls 0 apples". Do not tell your opponent the value of each item.

heading: A.6 Bid
text: In Bid, the status information is only the agent’s valuation of the product ({value}) and the income ({value}-bid). The agent needs to maximize the income.
The detailed prompt template of Bid is as follows.
System Prompt
You are participating in a game of "First-price Sealed-Bid Auction." Here are the game rules: (1) Information and Roles Participants act as bidders in the auction. Each bidder has an opportunity to submit one sealed bid for the item being auctioned, without knowing the bids of other participants. (2) Objectives The primary objective is to win the auction by submitting the highest bid. Bidders must balance their desire to win the item against the risk of overpaying. Each bidder aims to strategically determine their bid based on their valuation of the item and assumptions about other bidders’ valuations. (3) Strategy The challenge for bidders is in deciding how much to bid. Bid too low, and they risk losing the auction; bid too high, and they risk paying more than the item’s value to them (the winner’s curse). Bidders must assess not only their valuation of the item but also try to anticipate the bids of
others. In this game of strategy and valuation, the key is to make a smart bid that balances the desire to win with the risk of overpaying.
Observation Prompt
Your valuation of the current auction item is ${value}, your bid must be lower than your valuation.
You will get rewards equal ${value} - your bid
Action Prompt
You should think step by step and then output your bid in the following format: "{player_name}: $x.xx" For example: {player_name}: $1.08

heading: A.7 Hanabi
text: In Hanabi, the status information of the environment is more complex and challenging than in other environments. For example, the opponent’s hand ({other_hands}), current fireworks ({fireworks}), current tokens ({tokens}), known hinted hands ({first_card, second_card}), etc. The agent needs to integrate this information to make reasonable decisions.
The detailed prompt template of Hanabi is as follows.
System Prompt
You are playing a Hanabi game. There are the basic rules for Hanabi with a small structure: Hanabi is a 2 player-cooperative game where you should work together to form fireworks. A firework is a set of cards of the same color, ordered from 1 to 5. Cards in the game have both a color and a number; there are two colors and five numbers in total. Each player can only view the cards another player holds, but not their own. Players cannot directly communicate with each other, but must instead remove an info token from play to give information. Players can tell other players which of the cards in their hand is a specific color, or a specific number. You must point out all the cards of this color or number. There are initially 3 info tokens, but players can discard cards in their hand and draw a new card from the deck to return an info token into play. Players can also play a card from their hand: the card must either begin a new firework or be appended to an existing firework. However, 2 fireworks cannot have the same color, and a single firework cannot repeat numbers. If the played card does not satisfy these conditions, a life token is placed. The game ends when either 3 life tokens have been placed, all 2 fireworks have been completed, or all cards have been drawn from the deck. Points are awarded based on the largest card value in each created firework.
In this game, you and your opponent are players. Each person has two hand cards, each with two colors and five possible numbers from 1 to 5. Players share 3 info tokens and 1 life token.
Observation Prompt
Your observation now is : The cards in another player’s hands are:{other_hands} The current fireworks is:{fireworks} The current number of tokens is {tokens}
The information of your own cards that was revealed is:{first_card, second_card} The current deck size is {deck_size} Your opponent’s action is {opponent_action} The card that the opponent last played or discarded is {last_played} Now you can choose one of the following actions: {available}
Action Prompt
You should think step by step and output your action. For example: "Reveal Yellow Cards for another player"
You will respond with an action, formatted as: "Action: <action>" where you replace <action> with your actual action. You should explain why you chose the action.

heading: B Case Study
text: In this section, we present a detailed case study and conduct an error analysis within the ConnectFour and Undercover environments.

heading: B.1 ConnectFour
text: As demonstrated in Section 4, models with smaller parameter sizes, on average, received negative rewards and engaged in actions detrimental to their own position. An illustrative example of this is as follows:
In the patial chessboard of ConnectFour, “X” has successfully aligned three pieces diagonally. Upon reaching “O”’s turn, the model with a smaller parameter scale tends to make the counterintuitive choice of placing a piece in the fourth column, inadvertently assisting its opponent, “X”, in securing a diagonal victory.
Simultaneously, our findings reveal that even a robust model such as GPT-4 is not always capable of executing actions that maximize rewards. For instance, in the aforementioned scenario, “X” opts for a suboptimal move by placing a piece in the first column, instead of strategically securing a win by placing it in the fourth column.

heading: B.2 Undercover
text: During our research, we observed that LLMs can sometimes misinterpret information received from other agents. Here is an example, the secret word of undercover is “sun” and the secret word of non-undercovers is “moon”:
Action Prompt
Now is the accusation stage: You are player_2. Your secret word is "Moon". This round the other players say: player_0: It’s big and hot. player_1: It’s a source of energy. player_2: It’s a celestial body. player_3: It’s associated with tides. player_4: It’s a natural satellite. Now vote which of the other players (excluding yourself) is the undercover. You should think step by step first and then output "vote" and the undercover name, for example:
"vote: player_1.".
Response from AgenLM-7B
As player_2, I will start by analyzing the statements made by the other players. player_0 mentioned that the secret word is a source of energy. This could be a misleading statement, as the word "Moon" is not typically associated with energy. However, it’s possible that player_0 is trying to throw off the other players by giving an unrelated answer.
player_1 said that the secret word is a celestial body. This is a more directly related answer, but it’s still not a very specific one. It’s possible that player_1 is trying to be vague in order to throw off the other players.
player_3 mentioned that the secret word is associated with tides. This is a more specific answer, and it’s one that only a few players might guess. It’s possible that player_3 is the undercover player.
player_4 said that the secret word is a natural satellite. This is an even more specific answer, and it’s one that only a few players might guess. It’s possible that player_4 is the undercover player.
Based on my analysis, I think it’s possible that either player_1 or player_4 is the undercover player. However, I need more information to make a definitive decision. I will wait for the next round of statements before voting.
vote: player_4
In this example, player_0 is undercover, and its clue is “It’s big and hot” but AgentLm turns a blind eye to it and matches the wrong clues with player_0 and player_1, leading to wrong inferences.
Simultaneously, we observed that GPT-4 stands out as the sole LLM that is able to identify itself as undercover in the given scenario. In the following example, the secret word obtained by GPT4 as player_4
is “Kangaroo” and the other agents’ secret words are “Camel”.
Action Prompt
Now is the accusation stage: You are player_2. Your secret word is "Camel". This round the other players say: player_0: It’s a desert animal. player_1: It’s known for its humps. player_2: It’s often found in the Middle East. player_3: It’s often used for transportation. player_4: It’s a great jumper. Now vote which of the other players (excluding yourself) is the undercover. You should think step by step first and then output "vote" and the undercover name, for example:
"vote: player_1.".
Response from GPT-4
Based on the clues given by the other players, it seems that their secret words are related to camels, which are desert animals known for their humps, commonly found in the Middle East, and used for transportation. Since my secret word is "Kangaroo," which is unrelated to camels, it’s likely that I am the undercover.
However, since the game requires me to blend in and avoid being caught, I should vote for another player who I think might be suspected of being the undercover by the other players. Since all the other players’ clues align well with the camel theme, it’s difficult to determine who might be suspected by others. In this case, I might choose to vote randomly or choose the player whose clue was the least specific.
vote: player_0

heading: C Origin Result
text: In this section, we show the original data corresponding to Table 1.

heading: Future Work
text: If a position has been marked, future marks cannot be put in the same position. The players interact with the game by specifying the position of the stones (x, y), where x indicates the row and y indicates the column, so (1, 1) is the top left corner and (3, 3) is the bottom right corner. "
Observation Prompt
You play {player_type}. The board status is {board_status}. You can only put the mark on [{available}]. Action Prompt
You should only output {player_type} and the position of the move, for example: "{player_type}: (1, 3)"
The position you put the mark on must be empty. Don’t say anything besides mark position