# Message Action Adapter Framework in Multi-Agent Reinforcement Learning
## 1 Introduction  
Artificial intelligence (AI) has shown superior performance to humans, affecting several research fields. RL agents surpass human-level control in games such as Go and Dota [1, 2]. The social influence of MARL will be increased [3]. Recent progress in deep neural networks shows the potential for artificial multi-agents [4, 5]. In a shared environment, multiple agents are jointly trained to maximize a common objective by communicating their observations or states. With cooperation, agents can solve complex problems that single or non-cooperative agents can not solve [6]. In addition, several benefits exist with cooperation, including reduced individual workloads of individual agents [7]. Such benefits are leveraged in various applications, including power control, IoT (Internet of Things), and robotics [8]. Under the communication of several agents, verification of the effects of communication is crucial. Despite the benefits of cooperation, the training dynamics of MARL show severe instability [9, 10]. Among several reasons, such as the hardness of an opponent modeling and extensive search spaces of action, one crucial factor is the communication channels in MARL, which are optimized jointly with a policy to maximize rewards. In the learning dynamics of message construction, agents must learn to process observations to construct messages for other agents while interpreting received messages. Most MARL approaches utilize deep neural networks for the communication channels. However, optimizing message representation has been shown to increase the instability [11, 12]. The instability is not only an issue for MARL but is already known for a single agent. For example, a nonstationary target problem exists in RL compared to the stationary targetin the computer vision classification task [13]. Accordingly, MARL, an extension of a single-agent RL, also includes the problem of the nonstationary target. Even worse, MARL utilizes noisy message representations for policy inputs. As such, it suffers from instability training and requires extensive training time. Although MARL could provide higher returns than individual training of agents, the contribution of MARL communication channels is not easily measurable, and this hinders the direct application of MARL, where verification of agent controls is crucial. For example, users can not deploy MARL agents in the real scenario of UGV (Unmanned Ground Vehicle) controls [14] without verifying the trained policies. To resolve the uncertainty on the contribution of communication channels in MARL, we factorize the actions conditionally on two variables: single-agent observation and messages from other agents. As such, we propose a message adapter framework (MAAF) where agents learn their actions without communication and then adapt the residual action conditional on the message while either updating or freezing the original action. This framework completely isolates the impact of messages from the observation-only control of an agent. The factorization of actions conditional on different inputs is highly important in control tasks where safe use of RL [15]. Consider a robot manipulation, where the individual robot moves following safe procedures. The disentanglement is barely achieved when a policy combines observations and messages as input. MAAF can be interpreted as a transfer learning framework from individual control to communicative action problems. We define two types of actions, observation-action and message-action, and study the difference between the two actions. Figure 1 shows two action types derived from observations and messages in a predator-prey environment. The proposed framework explicitly infers these two types of actions. This work studies the separation of messages in MARL to encourage factorized modeling of MARL action space. We propose message-based action adaptation (MAAF) to verify the contribution of message factors. We evaluate MAAF on four multi-agent environments and analyze the contribution of message factors.