# Message Action Adapter Framework in Multi-Agent Reinforcement Learning
## 7 Results
### Comparison to Baselines  
Figure 8 shows the performance of MAAF-RNN and MAAF-MHA compared to other MARL baselines in four environments. MAAF variants perform better than other baselines for environments, spread, and tag, indicating that MAAF has competitive performance to baseline algorithms. For the comparison of message pooling methods, RNN-based pooling shows slightly better adaptation than the attention-based one. One reason is that the RNN-based pool is better than the attention-based one when the agents are fixed and small. Nonetheless, we acknowledge that the message pooling methods could be environment-specific, and training attention might benefit the message passing, as shown in TarMAC [11]. For example, MAAF-MHA performs better than RNN-based in both SMAC environments.  
Figure 8: Episodic return while training five MARL agents for the best hyperparameter settings and three random seeds. We observe the competitive performance of MAPPO for all environments. In the tag environment, where communication between agents is crucial to tracking moving agents, the proposed MAAF variants perform better than all the baselines. However, MAAF shows weak results in SMAC environments where the number of actions is too large to be adapted. Therefore, MAAF shows a large gap to MAPPO in SMAC8M, which has eight agents.  
Figure 7: Episode returns for message types. The solid lines are the cases when updating base and adaptation policies, and the dashed lines are cases when the base policies are not updated. Each color represents different message types in \(\mathcal{S}_{\text{Comb}}\). When freezing, the performance drops after the 1M time step, indicating that updating both networks is robust for message types.  
MAAF trains base policies until 1M and combines actions from message communication. At this time step, the output of a noise state policy is added to the base policy. Therefore, we observe a performance drop at 1M time step. In the case of SMAC 3M and 8M, we observed a significant drop in performance. This is because SMAC has more feasible candidate actions (more than 100, while MPE has less than ten actions). Therefore, although a base policy offsets the probabilistic distribution of actions, adaptation with random policy at 1M has a large influence. That is, the logits of adapted policy \(\hat{\phi}\) for action \(a\) is the form of \(\log\hat{\phi}(a)=\log\pi(a)+\log\phi(a)\) with noisy \(\phi\sim\mathcal{N}(0,\sigma)\), and the base policy \(\pi\) is largely distracted as the size of action space increases. Similarly, the base policy of environments is more confident and is less affected by the randomly initialized adaptation policy. Therefore, initializing the adaptation policy \(\phi\) with a small \(\sigma\) may help to adapt the adaptation policy. We leave additional optimization strategies for MAAF as future work.  
We additionally evaluate MARL methods trained at the last checkpoint for 100 episodes. Table 1 shows the average evaluation returns. In addition, we provide MARL training settings on whether communication between agents exists and whether the value function uses shared observation. In tag and SMAC.3M, MAAF achieves the best evaluation results among the four models, even outperforming the value sharing (MAPPO). Compared to full communication TarMAC, MAAF-RNN achieves higher performance in all environments. Therefore, the proposed methods work when communication provides better performance when communication is essential.