Title: Unveiling Complex Collective Behaviors from Simple Rewards
Primary area: interpretability and explainable AI
Keywords: Explainable Reinforcement Learning, Swarm Intelligence, Multi-Agent Reinforcement Learning

Abstract: Recent studies have shown that multiple agents trained through reinforcement learning can surprisingly exhibit swarm behaviors from simple rewards, without any rewards specifically encouraging aggregation. Explaining how complex collective behavior emerges from these simple rewards is an intriguing research problem, but the underlying process remains a black box up to now. This paper aims to reveal the hidden rules in this process. Specifically, we discovered that the reason agents are able to develop complex behaviors from simple rewards is that they implicitly learn the geometric fields of the environment and utilize these structures as desired targets for coordinated movement. Surprisingly, agents converge toward the boundary of the predators’ Voronoi diagram. The agents are able to spontaneously learn Voronoi diagrams without any guided rewards. To gain the above insights, we propose a two-stage EEC (\LinkIII) explanatory framework. This includes a novel analytical tool called the Agent Response Map (ARM), which reveals agents' decision-making patterns across space and identifies regions of aggregation and avoidance. We also extend the proposed method to a more realistic and challenging robot-swarm control task, to validate its generality and practical utility. The insights and tools presented in this paper may provide a new perspective on the connection between AI-driven multi-agent systems and real-world biological systems.

TL;DR: This paper introduces an explaination framework to explain the emergence of collective behaviors from multi-agent reinforcement learning.