Title: MASDiff: Large Scale Multi-Agent System Emergence Control via Evolutionary Diffusion
Primary area: generative models
Keywords: Emergence Control; Multi-agent reinforcement learning; conditional diffusion model; Evolutionary strategy;

Abstract: Emergence control aims to learn the microscopic decision policies of individual agents within a system. Based on these individual policies, macroscopic properties emerges. This paper aims to learn the individual policies of 1000+ agents based on multi-agent reinforcement learning (MARL). Although CTDE and DTDE frameworks are capable of training decentralized individual policies for multiple agents, when the number of agents scales to 1000+, both frameworks can hardly work. For the CTDE framework, although credit assignment methods can be used to distribute macroscopic goals into individual agents' microscopic rewards, the combinatorial explosion of large-scale agents makes the learning process nearly impossible. As for the DTDE framework, it lacks the capability to facilitate bottom-up learning from microscopic policies to macroscopic phenomenon. To handle these challenges, this paper proposes a diffusion model driven DTDE MARL framework MASDiff which shift the focus from exploring the optimal policies to exploring the optimal sampling data. MASDiff leverages diffusion models to learn from data sampled by multi-agents. Then it employs evolutionary mechanism to generate optimized sample data that can steer the offline training of policies to promote controllable emergence. In the experiments, the emergent macroscopic data simulated by the learned policies matches well with the ground truth data. Furthermore, we demonstrate that the ability to achieve controllable emergence would enable counterfactual reasoning in such systems. We propose several 'What if' questions to indicate the change of scenarios and obtain relatively accurate counterfactual reasoning results.

TL;DR: This paper aims to learn the individual policies of 1000+ agents based on multi-agent reinforcement learning (MARL).