Title: Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Value function factorization, Multi-Agent

Abstract: Value function factorization is widely used in cooperative multi-agent reinforcement learning. Existing approaches often impose monotonicity constraints between the joint action value and individual action values to enable decentralized execution.
However, such constraints limit the expressiveness of value factorization, restricting the range of joint action values that can be represented and hindering the learning of optimal policies. To address this, we propose Potentially Optimal Joint Actions Weighting (POW), a method that ensures optimal policy recovery where existing approximate weighting strategies may fail. POW iteratively identifies potentially optimal joint actions and assigns them higher training weights. Our approach can be seamlessly incorporated into a variety of value function factorization algorithms, and we provide a theoretical proof that this iterative weighted training guarantees recovery of the optimal policy. Experiments on matrix games, difficulty-enhanced predator-prey tasks, SMAC, SMACv2, and a highway-env intersection scenario demonstrate that our method consistently improves performance and surpasses state-of-the-art value-based multi-agent reinforcement learning methods.

TL;DR: Recovers the optimal joint policy by iteratively recognizing potentially optimal joint actions and assigning higher weights to them.