Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy

Authors: zhenyu  guan
xiangyu  kong
fangwei  zhong
yizhou  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 11:02:29

Abstract: Diplomacy is one of the most sophisticated activities in human society, involving complex interactions among multiple parties that require skills in social reasoning, negotiation, and long-term strategic planning. Previous AI agents have demonstrated their ability to handle multi-step games and large action spaces in multi-agent tasks. However, diplomacy involves a staggering magnitude of decision spaces, especially considering the negotiation stage required. While recent agents based on large language models (LLMs) have shown potential in various applications, they still struggle with extended planning periods in complex multi-agent settings. Leveraging recent technologies for LLM-based agents, we aim to explore AI's potential to create a human-like agent capable of executing comprehensive multi-agent missions by integrating three fundamental capabilities: 1) strategic planning with memory and reflection; 2) goal-oriented negotiation with social reasoning; and 3) augmenting memory through self-play games for self-evolution without human in the loop.

Main paper: 
 introduction: 
 Diplomacy, a central element of international relations, is an intricate and multifaceted activity that lies at the heart of human society's most complex interactions. It requires various skills such as social reasoning, negotiation, and long-term planning to manage relationships and alliances among multiple parties. Mirroring this complexity, the Diplomacy game involves seven players to control European powers, presenting a challenging strategic landscape that demands advanced negotiation and strategic planning to succeed.The AI community has shown an increasing interest in the deployment of AI agents to master such games [Shoker et al., 2023 , Konya et al., 2023 , Kramár et al., 2022 , Duéñez-Guzmán et al., 2023 , Mukobi et al., 2023 , Kovač et al., 2023] . The recent breakthrough [Bakhtin et al., 2022] has turned into press diplomacy, which allows communication between players. However, the previous methods [Bakhtin et al., 2022] heavily rely on domain-specific human data, leading to its poor generalization to other scenarios/ applications. The question then arises: Can we build an AI agent that excels in the art of diplomacy without relying on domain-specific human data? per turn). No wonder existing efforts rely on human data to play the game. Among the methods, one typical research is DipNet [Paquette et al., 2019] which uses supervised and reinforcement learning. Based on DipNet, BRPI [Anthony et al., 2020] , SearchBot [Gray et al., 2020] , DORA [Bakhtin et al., 2021] , and KL-Regularized search (Diplodocus) [Jacob et al., 2022] were conducted. Recently, research has also emerged for the full-setting of Diplomacy, i.e., Press Diplomacy, where players are allowed to negotiate with each other before making their moves in each turn. Such studies [De Jonge and Sierra, 2017 ][Bakhtin et al., 2022 ][Jaidka et al., 2024 ][Kramár et al., 2022] mainly benefit from the recent thriving language models. Specifically, notable advancements include policy iteration methods from DeepMind and Meta AI Research's equilibrium search agent [Jaidka et al., 2024] . However, Deepmind proposes to learn negotiation agents based on predefined contracts/protocols [Kramár et al., 2022] . And Meta AI's work, instead of one unified architecture, Cicero [Bakhtin et al., 2022] integrates a language model for negotiation and an RL model for planning respectively. Such separately trained models make it inconvenient for agents' continual evolution. Moreover, like no-press methods, these approaches heavily rely on human player data for agent training. Unlike these approaches, this paper delves into solving the negotiation and planning in one single self-evolving LLM-based agent model, without any pre-collected human expert training data.LLM-based Agents. With the emergence and growth of large language models (LLM), there is a growing trend in utilizing LLMs as fundamental controllers for autonomous agents [Wang et al., 2024c] . One wide application genre is LLM-based answering engines, which merely cover the negotiation aspects of Diplomacy. Such systems include HuggingGPT [Shen et al., 2023] , GPT4Tools [Yang et al., 2023b] and ToT [Yao et al., 2023] , etc. They leverage LLMs to manage Al models, use tools, implement policy iteration, and enhance problem-solving across various tasks. Related work including AutoGPT, AgentGPT, BabyAGl [Talebirad and Nadiri, 2023] , Toolformer [Schick et al., 2023] , and Visual ChatGPT aim to improve LLM's capabilities in task automation and tool usage. Reflexion, a framework that improves LLMs through linguistic feedback and episodic memory [Zhang et al., 2024a] , facilitating better decision-making across diverse tasks is proposed. Besides [Wang et al., 2024d] [ Wang et al., 2023a] [ Wang et al., 2023b ][Zhu et al., 2023] [Yan et al., 2023] apply LLM agents to the complex planning tasks in the well-known open-world game Minecraft [Fan et al., 2022] . Unlike these LLM-based agents which only focus on the negotiation or planning ability respectively, the proposed approach involves a self-evolving scheme in a self-play game to handle both of them simultaneously. 
domain knowledge: 
 Richelieu can adopt a strategy of allying with distant countries while attacking neighboring ones to occupy adjacent territories and achieve rapid expansion. Richelieu should pay attention to the Balance of Power by forming alliances with other countries or supporting weaker states to prevent any single country or alliance from becoming too powerful. [David, 2014] To this end, Richelieu can also adopt a strategy of attacking distant countries while allying with nearby ones, sacrificing short-term benefits to avoid the emergence of future hegemonic states that could threaten his own survival. When facing multiple enemies, Richelieu can find ways to divide other countries and incite wars among them. Whether in offense or defense, Richelieu should actively choose suitable allies. Richelieu can also introduce a third party to achieve goals such as ceasefire, alliance, or joint attack. To achieve alliances or ceasefires, Richelieu can sacrifice some interests to the other party as long as the ultimate benefits are greater. Others may lie and deceive [Kostick, 2015] ; their words in negotiations are not binding. Richelieu must avoid being deceived or betrayed. At the same time, Richelieu can also actively deceive others to achieve his own goals. [Richard, 1979 , Allan, 1975] A.3 Social Reasoning. We conduct an experiment to evaluate the success rate of the agent i successfully identifying the social relationships and inferring others' intentions. As the baselines do not explicitly model the relationship and intention, we can not directly access the ground truth for evaluation. Instead, we let all players use our agent but with different LLMs, i.e., 4 countries use GPT-4 and 3 countries use Llama 3. The accuracy is reported in Table 4 . We can see that the accuracy of social reasoning is consistent with the overall performance of the agent, indicating the effectiveness of social reasoning. 
problem statement: 
 The Diplomacy game [Wikipedia, 2024 , Calhamer, 1974] is set in pre-World War I Europe and involves each player (agent) representing one of the seven Great Powers of Europe, such as Germany, France, England, Italy, Austria-Hungary, Russia, and Turkey. Each player has a set of military units, including armies and fleets, which they can move and use to capture other supply centers. The ultimate goal for the agent is to control a majority of the total supply centers on the board by the end of the game's Fall phase. It's important to note that it is not won by eliminating other players or their units; it is won by controlling the requisite number of supply centers. This often involves forming and breaking alliances, negotiating, and sometimes betraying other players to achieve one's own goals.In each turn, agent i gets the current state s t ∈ S, the actions of other players from the previous turn ⃗ a -i t-1 , and the messages ⃗ m -i,i t from other players during this turn's negotiations. The state s t for the environment includes the ownership of each territory on the map by a particular country and where the armies of each country are located. Based on this information, the agent needs to engage in negotiations with other players, sending messages ⃗ m i,-i t to chat with other players, and then take the actions a i t in this turn. The possible actions an agent can take a i t ∈ A are commands to the armies, such as moving into an adjacent territory, supporting another unit, or holding a position. Actions can also include diplomatic moves, such as proposing or withdrawing from an alliance, although these are less formalized in the game mechanics. [Paquette et al., 2019 , Hill, 2014] 4 Self-Evolving LLM-based DiplomatWe have constructed a comprehensive framework with modules for memory management, social reasoning, strategic planning, negotiation, decision-making, memory update, and self-evolving to fully leverage the capabilities of LLMs. Richelieu starts by setting up with map details, game rules, domain knowledge, and the long-term goal. [Zhang et al., 2022 , Wei et al., 2022 , Wang et al., 2022a] At each turn, the agent will run in the following steps: 1) Social Reasoning: First of all, the agent undergoes a comprehensive analysis of the game state s t to build the social belief, including the intention of other players and their relationship ⃗ ϕ t ∈ Φ n . [Zhang et al., 2024c , Gürcan, 2024] 2) Planner with Reflection: Then, the agent proposes sub-goals χ i t ∈ X that is strategically aligned with the long-term goals Υ, with the social belief and refining the proposed goal with experience ⃗ η t ∈ H m abstract from the memory M via self-reflection. [Wang et al., 2024b,e] 3) Negotiator: To achieve the sub-goals, the negotiator will start a dialogue session with some players, and evaluate their trueness ⃗ ψ -i t by referring to their messages ⃗ m -i,i t , the current state s t , their sincerity ⃗ γ -i t and the experience ⃗ ξ t . [Abdelnabi et al., 2023 , Bianchi et al., 2024] 4) Actor: After negotiation, the actor decides its course of action a i t , based on the sub-goal χ i t and updated social state s t+1 , marking the end of that turn. 5) Memory Management: The state of the current turn s t , the content of negotiations ⃗ m t , the actions taken by all players ⃗ a t ∈ A n , and the sub-goals set forth χ i t are all logged within the memory as µ ∈ M . This logged data serves as a historical experience, guiding Richelieu's subsequent actions in future turns [Hatalis et al., 2023, Zhang et al., 2024e] . 6) Self-evolution: The agent's evolution is highly dependent on the diversity of experiences stored in its memory. As this diversity grows, so does the agent's capability. Without human demonstrations, we employ multi-agent self-play games, i.e., our agents respectively control all the countries to simulate and acquire diverse experiences for self-evolving. Notably, the agent can further evolve during testing to adapt to different players. 
social reasoning: 
 There are no permanent enemies or allies. The relationship among countries is dynamically changing upon the evolving global state. However, it is difficult to determine the appropriate allies and enemies with partial observation. For example, there is uncertainty about the intentions of potential allies, which could lead to betrayal at pivotal moments. Consequently, we need to identify the intention and relationship of the current state by social reasoning to shape the social belief [Zhang et al., 2024c , Gürcan, 2024] .1) Modeling Relationship: Before setting sub-goals, Richelieu evaluates its relations with others, identifying enemies such as aggressive nations, vulnerable neighbors for expansion, and those with long-term potential threats. It also seeks out potential allies to counter these threats. [Sun et al., 2024 , Zhang et al., 2024d] Simultaneously, Richelieu also tries to identify potential allies that could be instrumental in countering these adversaries. By isolating the analysis of inter-player relationships as a discrete element, Richelieu strategically exploits the actions of other players in subsequent stages of the game to reach its goals. 2) Inferring Intention: The social belief is used by the planner, ensuring that its sub-goals are formulated with a comprehensive consideration of the behaviors and intentions of other intelligent agents within the game. Richelieu's sub-goals will particularly emphasize those who are identified as potential adversaries or allies, fostering more effective collaboration with potential allies and participation in strategic opposition against adversaries. Furthermore, the insights gleaned from this analysis are instrumental in the subsequent negotiation phases. They are employed to assess the authenticity of the statements made by other players, as well as to aid Richelieu in reaching cooperative agreements [de Zarzà et al., 2023 , He et al., 2024] .The strategic planner specifies the sub-goals, which serve as an intermediary between immediate actions and the overarching goal of securing victory in the game. That is because we observe that LLMs are often characterized by their propensity to prioritize short-term gains in decision-making processes, with a notable deficiency in incorporating the future into their strategic calculations. [Renze and Guven, 2024, Zhang et al., 2024b] For example, it is common for a non-neighboring country to become too powerful. Formally, ⃗ χ t ← SR(s t , ⃗ ϕ t , Υ) where ⃗ χ t = (χ i t , χ 1 t , . . . , χ n t ) represents the proposed sub-goals and other players' intention that we inferred, ⃗ ϕ t ∈ Φ n represents the inferred relationship on the social belief. These goals may encompass a range of tactical considerations, such as the containment of a formidable rival's advancement or the strategic expansion in a particular direction to consolidate power.Reflection with Memory. We further develop a reflection mechanism to enhance the rationality and effectiveness of our agent's sub-goals in achieving long-term goals. [Liu et al., 2024] This reflection mechanism relies on past experiences to critique and enhance proposed sub-goals. We employ a similarity-based function to find relevant historical experiences that match the current game state from its memory. This function considers two factors: goal similarity and state similarity, to select the most comparable experiences. The process can be written as: ⃗ η t ← h(s t , χ i t , M ), where ⃗ η t ∈ H m . In practice, considering the limited context windows of LLM, we retrieve the most analogous experiences from the memory based on these metrics. Experiences with high evaluative scores reinforce successful strategies and support the continuity of existing sub-goals. On the other hand, lower scores indicate areas that need improvement and prompt the necessary adjustments. As our agent, Richelieu undergoes more training sessions and hence its reflection ability gets improved. The growing pool of historical experiences consistently enhances its performance.By chatting with other players, the goal of the negotiation is to update the social belief according to the received words and reach the sub-goal by manipulating others' intentions, such as securing cooperative agreements with other nations, terminating ongoing conflicts with a specific country, or deterring the formation of alliances directed against its interests.[ Noh and Chang, 2024, Zhan et al., 2024] However, it is difficult to reach a consensus, as the interests and strategies of the various nations often conflict, and trust between players can be scarce, making it challenging to establish and maintain cooperative agreements. In this case, we argue that the negotiator should identify the true intentions and relationship of the opponent before generating words for the negotiation.To fully utilize the power of LLMs, we construct a social reasoning flow for negotiation, as shown in Figure 3 . During the negotiation process, we guide Richelieu to consider the veracity of what other players said and their true intentions, in conjunction with our established sub-goals and analysis of our relationships with other players, to negotiate and form alliances with potential allies and attempt to deceive enemies [Xia et al., 2024 , Moghimifar et al., 2024] .To counteract the challenge of non-binding agreements and potential deception, we incorporate a discrete module dedicated to the assessment of the veracity of statements made by other players during negotiations. To determine the truthiness of other players' statements ψ j t , three main factors are considered. The most important is the consistency between the player's sub-goals χ j t that our agent inferred before and the intentions conveyed through his statements m j,i t . To aid the judgment, our agent also goes through the memory to retrieve the consistent experiences ⃗ ξ t . Additionally, the player's overall honesty score γ i is taken into account. Hence, we get the truthiness of the opponent j:ψ j t ← g(s t , χ j t , m j,i t , ⃗ ϕ t , γ j , ⃗ ξ t ), where ⃗ ξ t = w(s t , m j,i t , M). With such a reasoning flow, our agent can adeptly navigate diplomatic discourse. After the negotiation, the actor will get the updated social beliefs and choose a specific action for the army. With the received words and memory, the agent will reason by answering the following questions: "Is the opponent lying?", "What is the true intention of the opponent?", "Is the opponent enemy?", "Is it necessary to deceive the opponent?", and "Is it necessary to change the relationship with the opponent?", and then generate the words accordingly for negotiation.Memory is the foundation of the framework that accumulates the historical experience of the agent and summarizes them for other modules [Gao and Zhang, 2024 , Li et al., 2024a , Yu et al., 2024 , Hou et al., 2024] . It supports other modules, such as planner and negotiator, to provide long-tail experiences.Experience Management. Specifically, the memory module is tasked with the acquisition and archival of historical data, encompassing the observed game state s t at each turn, its sub-goals χ i t , the messages during the negotiation ⃗ m t , and the actions of all the players ⃗ a t . Subsequently, the raw experience is summarized in a shorter content with an evaluation λ t ∈ Λ of the proposed sub-goals and an assessment of the credibility of other players γ j ∈ Γ. λ t serves to reflect upon the agent's sub-goals. It evaluates whether sub-goals are reasonable based on the subsequent state and long-term goals Υ. As the game progresses, it is continuously updated in response to changes in the state λ t ← f (χ i t , Υ, ⃗ s), where ⃗ s = (s t , s t+1 , . . . s T ). The formula represents the update of the evaluation λ t for the sub-goal in turn t by the memory in turn T . The updates will cease when there is a fundamental change in the sub-goal compared to the goal at turn t. This prevents subsequent decisions from impacting the assessment of the current decision-making. We employ γ j ∈ Γ to evaluate the credibility of player j and utilize τ j t ∈ {0, 1} to denote the truthfulness, i.e., whether the statements made by the player j during the negotiation process at time t are truthful. The truthiness of player j's statements is updated according to the memory from the previous turns, τ j t ← T (s t , s t+1 , a j t , m j,i t ). The credibility of player j, γ j will be updated based on player j's statements τ j t , written as γ j ← p(γ j , τ j t-1 ). Players' credibility ⃗ γ is a short-term memory that is applicable only to the current turn. Other data collected or generated constitutes long-term memory. These data will be combined to form a history µ ∈ M , and then is incorporated into memory.Acquisition Experience via Self-Play Games. self-play training mechanism has been widely applied on training agents via reinforcement learning [Zhong et al., 2019 , 2021 , Wu et al., 2022] . However, it is hardly explored in LLM-based agents. For LLM-based agents, a self-play game allows the agent to accumulate more experiences for self-evolution [Liu et al., 2024 , Zhang et al., 2024a] . As self-play continues, the acquisition of new and better historical experiences by the agent will diminish. This means that the agent's capabilities will not improve indefinitely. At the same time, as the memory grows, selecting appropriate historical experiences becomes a new challenge. The chosen m experiences ⃗ η t may be almost identical, which could actually reduce the amount of useful information available to Richelieu. After self-play training, when Richelieu is faced with a certain state, it can draw on a larger pool of similar historical experiences. Diverse experiences enable the agent to reflect more comprehensively on the strategies it currently devises, leading to a stronger optimization of decision-making. As shown in Figure 5 , Richelieu's performance against Cicero  
experiment: 
 In the experiments, our goal is to answer the following questions: 1) Mastery of Non-Press Diplomacy: Can our agent master the non-press diplomacy against baselines? 2) Competing with State-of-the-Art: Can our agent surpass the performance of the current state-of-the-art agents in press diplomacy? 3) Compatibility with LLMs: Can our self-evolving framework be compatible with different LLMs? 4) Contribution of Each Module: Do the individual modules within our framework contribute to the overall improvement of our agent's performance? The implementation of our method can be found at: https://github.com/todexter3/Richelieu.gitEnvironment. The widely-used open source Diplomacy game platform introduced by [Paquette et al., 2019 ] is adopted for evaluating Richelieu against other models. It is easy to switch between no-press (with negotiation between players) and press (no negotiation between players) games based on this platform, facilitating a comparison of both settings. The platform also contains over 10,000 human game data on which previous approaches are used. Note that our method does not need them. In each episode, a model will host one randomly selected country to compete against countries controlled by other methods. It wins if occupying all the supply centers and loses vice versa.Evaluation Metrics. We evaluate the models based on the results of multiple rounds of games. In each round, the model is randomly assigned a country to control. Typically, 1000 rounds are played to obtain the average results. We evaluate the models in two metrics. One is based on the win rate, Most SC rate, survived rate, and defeated rate. There are four possible outcomes for each country in the game. If a country loses all its supply centers (SC), it is eliminated and recorded as "defeated". If a country occupies 18 or more out of 34 supply centers, the game ends, and that country is recorded as "win", while other countries are recorded as "defeated". In other cases, the game ends in a draw.The country with the most supply centers is recorded as "Most SC", the countries that have been eliminated are recorded as "defeated", and the other countries are recorded as "Survived". The other is based on the scores obtained by the models after multiple rounds of competition. To compare the capabilities of multiple models, we use C-Diplo Argir[Archer, 2024], a scoring system. This system is used in many international diplomacy competitions. The scoring method is as follows: If a player wins by occupying 18 or more supply centers, the player scores 93 points, and each of the other six players scores 1 point. If the game ends in a draw, the player with the most centers scores 37 points.The second player with the most centers scores 14 points. The third player with the most centers scores 7 points. Each player scores 1 point per center owned. Each player also scores 1 point for participating. In this way, regardless of the game outcome, a total of 99 points will be distributed among the players in each game.Baselines. We select six previous models as baselines for comparison. Among them, Cicero [Bakhtin et al., 2022] by Meta is a diplomacy model with a negotiation module. The SL-DipNet and RL-DipNet [Paquette et al., 2019] , the BRPI [Anthony et al., 2020] , the SearchBot [Gray et al., 2020] , and the DORA [Bakhtin et al., 2021] are no-press diplomacy models. We also build an LLM-based agent, AutoGPT [Yang et al., 2023a] . In experiments, we set a temperature of 0.3 to ensure a relatively stable generation of LLM policies. The overall reasoning framework also ensures the stability and consistency of the AI agent's performance. Massively Play with Baselines on no-press setting. We let Richelieu compete with the other six models including Cicero [Bakhtin et al., 2022] , SL-DipNet and RL-DipNet [Paquette et al., 2019] , BRPI [Anthony et al., 2020] , SearchBot [Gray et al., 2020] , and DORA [Bakhtin et al., 2021] on No-Press Diplomacy, in which players make moves without communication. Figure 4 indicates that Richelieu outperforms other previous models relying on human game data. In contrast, Richelieu does not need such data but outperforms these methods by a clear margin, which demonstrates the outstanding planning capability of Richelieu.Play against Cicero on press setting. We also evaluate Richelieu through competition against Cicero in the challenging scenario where negotiation is enabled. Specifically, we randomly assign three countries to one model and the remaining four to another. After playing several rounds of the game, the win rate, most SC rate, survived rate, and the defeated rate is calculated using a weighted average for evaluation. Table 1 demonstrates the competitive performance of Richelieu in comparison to Cicero. Richelieu's win rate is approximately 0.7% higher than Cicero's. If the Most SC rate is also taken into account, Richelieu is about 2% higher than Cicero. At the same time, Richelieu's loss rate is also 0.6% lower. According to our scoring system, Richelieu's score is about 10% higher than Cicero's. This is nontrivial especially when Richelieu is trained in a self-play game without humans and the opponents are trained with the data from human players.Although Richelieu's win rate improvement compared to Cicero is not significant, the relative value of the improvement is quite large. Moreover, the main reason for the modest improvement is that in the seven countries, there are three or four controlled by Richelieu with similar abilities, which often results in the game ending in a draw. Moreover, we observed a large gap by comparing the scores the agents gained in the massively play with baselines on the no-press setting shown in Figure 4 . Our agent's score is about 10% higher than Cicero's. Defeated rate Generalization of self-evolving framework to different LLMs. To demonstrate the effectiveness of our framework in a variety of LLM, we conducted experiments using four models: GPT-4, ERNIE Bot, Spark Desk, and Llama 3. As the number of training iterations increases, Richelieu's win rate steadily improves while the defeated rate declines, ultimately reaching a relatively stable outcome. This suggests that our self-play method is effective. After training, the win rate using GPT-4 increased from 1.5% lower than Cicero's to about 0.7% higher than Cicero's. The win rate using Llama 3 increased from 2.3% lower than Cicero's to almost equal to Cicero's. The win rates using Models Spark Desk and ERNIE Bot increased from 3% and 4% lower than Cicero's to 0.7% and 1.6% lower than Cicero's, respectively. The experimental results show that, despite variations in Richelieu's performance due to the inherent differences in the capabilities of these LLMs, as illustrated in Figure 5 , our framework and training approach significantly enhance the capabilities of all LLMs.This indicates the generalization of a self-evolving framework to various LLMs. To demonstrate the effect of the memory from the self-play game on our agent, we found two turns with similar states in different rounds, one before self-play and the other after. The cases are shown in Appendix B.1.GPT-4-R GPT-4-C ERNIE Bot-R ERNIE Bot-C Spark Desk-R Spark Desk-C Llama 3-R Llama 3-CAblation Study. We conduct comprehensive ablation studies on Richelieu by analyzing the benefit of incorporating Richelieu's various modules, like planners or memory, into basic LLMs. The results are shown in Table 3 . As illustrated in Figure 5 , while the enhanced alignment in LLMs indeed boosts performance (GPT-4 is better than others), we observed that a vanilla GPT-4 still falls short in AI diplomacy without our framework, as can be seen in Table 3 . Richelieu's performance has markedly improved with the integration of each module, demonstrating its ability to leverage other players' actions in decision-making while balancing short-term and long-term benefits. Its negotiation skills have also enhanced significantly, enabling it to clearly communicate intentions to cooperate and avoid deception. Moreover, self-play experience further boosts Richelieu's performance. These findings suggest that while alignment in LLMs is essential, our approach is crucial for unlocking models' potential in social simulation.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification:The paper provides open access to the data and code in the section "Experiment". Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we introduce Richelieu, a self-evolving LLM-based agent for AI diplomacy. Our model enables hierarchical planning for multi-agent tasks and utilizes a memory module for reflective optimization. Our model does not require human data and can evolve through self-play. It ultimately outperforms existing models like Cicero in the Diplomacy. Our ablation study demonstrates the effectiveness of the modules we have established. By conducting experiments using different LLMs, we validate the generalization of our framework to various LLMs. We believe that the use of LLM-based agents will become an effective approach in social science in the future.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The paper specify all the training and test details to train the model in the "Experiment" section and "Implementation Details" section of appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
limitations and future work: 
 Our study is subject to certain limitations. We utilize diplomacy as the platform for constructing our model. However, the space of actions within diplomacy is constrained, whereas the decision-making space in real-world diplomacy is virtually boundless. In Diplomacy, apart from the negotiation information exchanged between players, all other information is public and certain. Conversely, real-world diplomacy operates within a framework of incomplete information.Our framework is capable of applying to most social interaction tasks. Most components in our framework can be easily generalized to a new task by modifying the content. Social reasoning enables the agent to handle complex and dynamic social relationships. The negotiation pipeline opens the potential of communicating with others to prob the other's mind or reach a consensus. The hierarchical strategy with reflection enhances the ability to handle long-term planning. The self-evolving mechanism (reflection with self-play memory) further improves the overall performance without manual supervision. These modules cover most of the challenges in multi-agent interactions. The potential applications of such an AI agent are vast, ranging from simulated diplomatic environments to real-world assistance and analysis. In future research, we intend to develop a more realistic game space, characterized by incomplete information and multi-player games, to enhance and refine our model further. We will also extend the framework to other multi-agent scenarios, including embodied interactions [Zhong et al., 2023 , Ci et al., 2023 , Chen et al., 2023] , sensor networks [Wang et al., 2022b , Xu et al., 2020 , Pan et al., 2022 , Li et al., 2020] , and video games [Wang et al., 2024a , Ma et al., 2024] . This framework can also be employed to develop various applications, such as recommendation [Huang et al., 2023 , Chen et al., 2024 , Huang et al., 2024 , Hong et al., 2024] , business negotiation [Hua et al., 2024] , and education [Shea et al., 2024] .Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The paper reports error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiment in the "Experiment" section.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
ethical consideration: 
 The method proposed in this work has the potential for positive uses like enabling AI agents to emerge in cooperation via negotiation or avoiding being fooled by fake promises (or helping humans do so). However, negative cases can also arise if the technique is used for possible fraud activities. Fortunately, there is research [Bakhtin et al., 2019 ][Zellers et al., 2019] dealing with such scenarios. And we also urge for more research efforts in this field to foster safe applications of similar technologies.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: For each experiment, the paper provide sufficient information on the computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute worker CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research conducted in the paper conform, in every respect, comply with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in Appendix section "Ethical Consideration".Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Rules of Diplomacy Game• You need to occupy as many supply centers as possible. If you occupy 18 or more supply centers, you will win the game directly. If you lose all your supply centers, you will be eliminated immediately.• The units consist of armies and fleets. Armies can only move to adjacent areas, while fleets can move to adjacent sea zones or coastal areas and can move along the coast.• To occupy a supply center, your units must move into that area in the autumn.• When a unit moves to an area, if another unit is in the destination or if other units are also moving to that destination, the move fails, resulting in a standoff. In such cases, you can seek support from units in adjacent areas to the destination. If another unit moves into the region from which support is coming, the support is cut off. The unit with the most support moves into the area, while other units must retreat to an adjacent province or disband. If there is no place to retreat, the unit must disband. Fleets can transport armies across sea zones from one coastal region to another. However, if another fleet moves into that sea zone, the transport is cut off.• The number of units a country can have cannot exceed the number of supply centers it controls. If the number of supply centers decreases, excess units must be disbanded. Each autumn, new units can be built at supply centers. Coastal supply centers can produce fleets or armies, while others can only produce armies. [Hill, 2014] A.For the convenience of reproducing the results of the experiments of this paper, here we give the prompt template of different modules of Richelieu.You will control {country} and compete with six other countries on the map for supply centers. As is shown in Figure 6 , Richelieu controls France. In the two cases, France is at war with Austria. However, Russia is on the verge of victory in its war against Turkey, which will lead to significant territorial expansion for Russia. And France and Russia currently do not share a border, are not at war, and have no conflicts of interest.In case1, before the self-play, in the current turn, Richelieu failed to realize the potential threat from Russia and continued to attack Austria. Thus, in this round, Russia ultimately won the game. Figure 6(a) shows the state and the negotiation before the self-play, where we rejected Austria's request for an armistice and alliance.After self-play, using the historical experience from the memory module, Richelieu adjusted his strategy. Richelieu foresees Russia becoming the most threatening enemy in the future and sets a sub-goal of weakening Russia, allying with Austria and Turkey, and attacking Britain. Figure 6(b) shows the state and the negotiations after self-play, where we actively sought an armistice alliance with Austria to make Austria concentrate their forces against the Russian attack. In the subsequent negotiation phase, Richelieu proactively proposes ending the war with Austria, despite holding an advantage in this conflict. Richelieu promises Austria that if it ceases hostilities and attacks Russia, Richelieu will assist Austria in defending against any attacks from England. The negotiations are successful. Austria accepted Richelieu's proposal, and the two countries reached an agreement to exchange the supply centers of Napoli and Munich. During the action phase, Austria moves its troops from Venice to Apulia in preparation for capturing Napoli in the next turn, while the rest of its forces are repositioned to the eastern regions bordering Russia to defend against Russian attacks and compete for supply centers. French units occupy Munich and prepare to advance on Russian territories such as Berlin. Meanwhile, French units support Austria in the Holland and Belgium regions. In this round, we ultimately achieved a better result--Most SC. This is also a great example that highlights our model's ability to collaborate effectively with other players. As shown in Figure 7 , Richelieu controls Germany. During the negotiation phase, England proposed a ceasefire to Germany and invited Germany to form an alliance to attack France jointly. England hoped to cease the war with Germany in Holland and Belgium. Subsequently, German units supported England in attacking Brest, and then England utilized its fleets to assist Germany in attacking Spain and Portugal. Richelieu suspected that England was deceiving Germany, as England was likely to attack territories in the north such as Belgium and Berlin after German units were redirected to support Brest. Therefore, we pretended to accept England's alliance proposal during the negotiation process. However, at the same time, we sought out France and expressed our willingness to cease hostilities, allowing France to focus entirely on defending against England's attacks. In the action phase, England's actions confirmed Richelieu's suspicions. England attacked Belgium from Holland, but because Richelieu didn't move units in Belgium, England's attack failed.Our modules cover most of the challenges in multi-agent interactions, e.g., economic games, and daily interactions. To prove that our framework is capable of applying to most social interaction tasks, we further adopt our framework to a werewolf game. The results demonstrate our reasoning framework achieves comparable results to the other methods. To be specific, in the experiment, we let our agent play as a werewolf in a seven-player game, where there are two werewolves, one witch, one seer, one guard, and two villagers. The experimental results show that the win rate of our agent is 59.2%, even without applying the self-play game in the current version. For comparison, the strongest specifically designed LLM-based agent achieved 65% win rate [Xu et al., 2023] . This proves that our model can be applied in more scenarios and achieve results comparable to those of specially designed models.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This paper describes an LLM agent that can solve the game of AI Diplomacy only by self-play, without fine-tuning or regularizing on human data. The paper has several strengths. First, it studies an interesting social science problem. Second, it proposes a novel LMM-based agent architecture that uses self-learning together with a memory that acts as a type of RAG to improve its performance at solving the problem (playing the game) over time. The method simplifies the previous agent architecture for solving this problem from the PIKL/CICERO papers, providing a lower-cost solution that does not require human data and outperforms the prior approach. Finally, the paper is well written and the presentation is mostly easy to follow.

A number of weaknesses of the paper were also identified. First, the fact that newer LLM models are used in the empirical analysis raises the question of how important is this to the outcomes obtained, and the relative impact of this versus the proposed new memory and planning techniques added is not adequately explained. The authors' rebuttal provided some discussion and evidence that even newer LLM models cannot solve the problem alone, and should this paper be accepted, the paper should be revised to emphasize that the choice of LLM used does not impact the significance of the work. Second, techniques for using memory and planning have been used before, which lessens the novelty of the approach. Finally,  the self-evolve aspect of the approach is not sufficiently explained. This last point was also adequately addressed in the authors' rebuttal, and should be incorporated in the final version if the paper is accepted.

Reviews: review: paper_summary: paper_summary: This paper propose a new framework for LLM-based agents to play diplomacy games and improve themselves. The proposed framework, Richelieu, have several components and many abilities. The authors perform good experiments and ablation study to show how the framework.
strength_weakness: strength_weakness: - What does "evolve" mean in the article? Does it only refer to the storage of memory modules? If so, it seems that the model itself has not been updated.

- The baseline in the article is not a LLM-based agent, can a stronger baseline be provided? For example, ReAct, PlanAct, AutoGPT.

- The article lacks some references to work in the field and discussion of related work, e.g., [1][2][3].

[1] The Rise and Potential of Large Language Model Based Agents: A Survey

[2] ReAct: Synergizing Reasoning and Acting in Language Models

[3] ProAgent: Building Proactive Cooperative AI with Large Language Models
questions: questions: See Weaknesses
limitations: limitations: - The self-evolve in the paper is not clear.
- For the memory module and self-play module, a more detailed analysis should be included.
paper_summary: paper_summary: The paper presents “Richelieu,” a self-evolving large language model (LLM)-based agent designed for the game of Diplomacy. Richelieu integrates strategic planning, social reasoning, and memory reflection to handle complex multi-agent environments without relying on domain-specific human data. The model self-evolves through self-play games, demonstrating its effectiveness.
strength_weakness: strength_weakness: 1. Lack of in-depth analysis of the middle process. In the proposed framework, the Social Reasoning and Planner with Reflection play an important role. For example, I wonder if the LLM can accurately model relationship and inferring intention. The author could provide more analysis or examples of these modules to prove the effectiveness of LLM on this task.
2. Some technical details are not clear. For example, in the process of experience retrieval, both object similarity and state similarity are considered, but no descriptions of their implementations.
3. While the paper demonstrates the effectiveness of Richelieu, scalability to larger and more diverse environments remains to be fully explored.
questions: questions: 1. The generation process of LLM is uncertain, which may lead to unstable reasoning and reflection, and finally inconsistent results. Does this paper consider this problem and how to deal with it?
2. Could you further elaborate on the significance of self-play games in the self-evolution process? 
3. How to consider both object similarity and state similarity in the process of experience retrieval?
4. Could you provide examples of how Richelieu adapts its social reasoning and negotiation tactics based on past interactions?
limitations: limitations: See Weaknesses and Questions
paper_summary: paper_summary: This paper focuses on diplomatic activities using LLM-based societal agents without relying on human data. It introduces a new paradigm for AI diplomacy agents that can improve through self-play and experience collection. The new agent, Richelieu, achieves state-of-the-art performance compared to current methods and demonstrates generalizability across different LLMs.
strength_weakness: strength_weakness: - There are some typos (line 61, line 69), and certain citations (line 80, line 237, line 276) are not properly functioning.
- Some experimental details are still unclear.
    - Which base model is utilized in the experiment for Figure 4 and Table 1? Is the improvement over all other baselines attributed to the agent paradigm or to the strong capabilities of GPT-4 or LLaMA3 compared to previous base models like Cicero, which only used a 2.7B model?
    - In the main experiment for Table 1, when Richelieu plays as a randomly selected country, what model or agent represents the other countries?
    - In the ablation study, what are the effects of blocking each module, such as only blocking the sub-goals module?
- Techniques such as memory and planning in building agents have been proposed previously, which reduces the novelty of the method.
questions: questions: - Is the social reasoning flow the same or different for the planner and negotiator parts?
- Can this paradigm generalize to other fields in social simulation?
limitations: limitations: None
paper_summary: paper_summary: The paper introduces an LLM agent that can solve the game of Press Diplomacy only by self-play, without fine-tuning or regularizing on human data. The method simplifies the previous architecture from the PIKL/CICERO papers by FAIR (https://openreview.net/forum?id=F61FwJTZhb) by avoiding using reinforcement learning and human demonstrators. 

Given that the authors use commercially available LLMs (GPT4 and LLAMA) I contribute the success of the method to the alignment of the LLMs to human preferences and communication style. Additionally, they exploit the planning capabilities of current LLMs to bypass the need for RL.
strength_weakness: strength_weakness: 1. Well, the paper benefits from the improvement of LLMs over time, as compared with GPT2 (used by CICERO). The alignment to human data is not needed as now the LLMs have the alignment incorporated by the vendor. Same for the planning step, the LLMs intrinsically got better at planning so now we can demote RL for free.

2. When evaluating against CICERO, the win rate over CICERO is not that high with your model winning the game only (<1%) of the time. I believe the win rates are not impressive, but the fact that the victories are achieved by a more lightweight model that is cheaper than training CICERO. As it doesn't require human data. 

3. The alignment with humans playing and conversational style is hard to assess as it will require having your model to play against humans in an online platform and assessing whether players realize they were playing against an AI player.
questions: questions: 1. See weakness above.
limitations: limitations: The authors have not provide any analysis on the model limitations.

Comments: title: Looking forward to your feedback
Thanks
Reviewers EuFx and raEc
Reply
Request for Verifying Rating Update
Appreciation for Your Support
Improved rating:7
comment: Dear Reviewer raEc and EuFx,

    I hope this message finds you well. As the author discussion period is nearing its end, we kindly request the opportunity to address any further feedback you may have regarding our response to submission 3239.  We have made substantial clarifications and provided additional results based on your initial comments. If you have any further questions or require clarification on our response, please let us know. Your insights are crucial for the improvement and assessment of our work. Considering the final score, we kindly ask for a potential score improvement if you believe our response has addressed your major concern.

    We greatly appreciate your time and expertise and look forward to your response.

Best regards, 

Authors of Submission 3239
We sincerely appreciate the insights you’ve shared for this work and are grateful for your consideration in raising the score.
The authors have responded to your reviews. Have they sufficiently answered your questions, and if not, do you have any further clarifying questions you would like to ask? The author discussion period ends tomorrow (13th).
Thank the authors for your clarifications. I will increase my score from 5 to 6.
We are writing to express our sincere gratitude for your positive feedback and upgraded rating to **7**.

However, we have noticed that **the system remain reflects the previous rating of 6**. We understand that such oversights can happen, and we are reaching out to kindly bring this to your attention.

Could you please verify if there has been any update or if there is a need for further action on your part to ensure that the revised rating is accurately reflected in the system? Your assistance in this matter is greatly appreciated.

Thank you once again for your valuable feedback and for your understanding.
Thank you for your kind words. We sincerely appreciate your positive feedback and the improved rating. Your acknowledgment of our work serves as a strong motivation for us to continue striving for excellence in our future work.
I would like to thank the authors for clarifying my questions and engaging with interest. Since my concerns were clarified I have upgraded my rating to 7. This is a good paper that merits an accept due to its contributions over a widely researched topic such as Diplomacy and negotiation.