A Reinforcement Learning Environment for Cooperative Multi-Agent Games: Enhancing Negotiation Skills

In artificial intelligence, an agent is an autonomous program that interacts with its environment over time and decides its actions according to a given objective function. In a multi-agent system (MAS), multiple agents interact with each other. A MAS is decentralized, meaning that no agent controls other agents. Certain scenarios are inherently decentralized because the entities that inhabit it may need to act independently, with different or conflicting goals and information. Such situations can be naturally modeled as a MAS. There are situations where an agent must cooperate with another agent in the same environment to achieve its objective. However, there are cases where alternating between cooperative and adversarial behavior is the only way to achieve the final goal. Diplomacy is a multi-player board game, where the central aim of a player is to conquer Europe. To do so, a player must negotiate alliances and coordinate strategies with other players. The catch is that these agreements may either be public or private, and they are non-binding. No player has full information regarding its opponents’ intentions, and any player may break its established agreements. Creating a computer player (bot) capable of taking these factors into account is a difficult task because of the huge amount of possible plays and outcomes, and because it is challenging to measure the true strength of a player’s position. The solution we propose is a custom Diplomacy environment that enables reinforcement learning agents to submit agreements to other players. Agents can use this environment to learn how to negotiate using reinforcement learning methods. We created three different scenarios based on the same environment, which were designed to focus on different aspects of negotiation in Diplomacy. We tested each case with state-of-the-art reinforcement learning methods. We analyzed the success of our approach by comparing agents trained in our environment with other bots built for the Diplomacy game. Because Diplomacy is a game heavily reliable on negotiation tactics, a bot that has a strong negotiation capability is more likely to win the game. However, there are few implemented Diplomacy bots that take advantage of being able to make agreements. From the existing bots, we compare our solution to bots that simplify the game’s rules by assuming that the deals made are binding, eliminating the problem of trust. We found that while our environment is functional, and allows an agent to learn to act in Diplomacy scenarios with a reduced scope of action, the learning process for more complex situations is extremely slow, even with an adjusted negotiation scope. Software and hardware limitations translate into a prohibitively long time for the training process in our environment, which allied to the extensive observation and action spaces results in an agent that is not proficient in playing Diplomacy. However, with the environment we developed, we hope to enable further research in this game and its particular challenges. Continuing to tackle problems that have a social aspect using reinforcement learning techniques, and improving current solutions, could lead to systems capable of making better decisions in scenarios where negotiating and compromising is necessary. These systems could be used to