Pessimism Meets Invariance: Provably Efficient Offline Mean-Field Multi-Agent RL

p N ) , where de↵ is the effective dimension of the function class for parameterizing the value function , but independent on the number of agents . Numerical experiments are provided . 1 Introduction . Significant progress has been made towards multi-agent reinforcement learning ( MARL ) for many prominent sequential decision making problems , such as social welfare optimization ( Leibo et al. , 2017 ) , fleet control of autonomous vehicles ( Shalev-Shwartz et al. , 2016 ) and playing multiplayer online battle arena ( MOBA ) games ( Berner et al. , 2019 ) . As the joint state and action space scales exponentially with the number of agents , however , MARL becomes computationally expensive . One remedy is the mean-field regime when an extremely large number of homogenous agents are involved , e.g. , social welfare optimization . The effect of each agent on the overall multi-agent system can become infinitesimal , and therefore all agents can be considered interchangeable/indistinguishable ( Yang et al. , 2018 ; Carmona et al. , 2019 ; Li et al. , 2021 ) . Accordingly , the interaction among agents can be captured by some mean-field quantity such as the empirical distribution of states , and therefore each agent only needs to find the best response to the so-called “ mean-field state ” , which avoids the curse of many agents . Most existing results on mean-field MARL ( MF-MARL ) are for the online setting ( Yang et al. , 2018 ; Zhang et al. , 2019 ) , where the agents can interact with the environment during training . However , such interaction during training can be prohibitive for some important applications ( Leibo et al. , 2017 ; Mandel et al. , 2014 ; Jaques et al. , 2019 ; Levine et al. , 2020 ) . Taking social welfare optimization as an example , repeatedly conducting social experiments on human being can be unaffordable or even unethical in the societal systems . Therefore , we can only consider the offline settings , i.e. , we learn the optimal policy based on some pre-collected experience data ( Levine et al. , 2020 ) . Unfortunately , ⇤Extension to online setting is provided in a longer technical report version , which is available upon request . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . existing offline reinforcement learning ( RL ) algorithms and theories all focus on the single agent settings , and no algorithms and theories have been developed for MARL under the offline settings , regardless of the mean-field regime or not . To bridge such a critical gap , we propose the first pessimistic algorithm – named SAFARI ( peSsimistic meAn-Field vAlue iteRatIon ) for mean-field MARL , which can provably achieve sample efficiency under the offline setting . Our proposed algorithm contains two important components : ( 1 ) To incorporate the permutation invariance of the homogenous agents , we adopt a RKHS ( Reproducing Kernel Hilbert Space ) mean-embedding approach for approximating value functions , which avoids the exponential blowup of the agents ’ state and action spaces ; ( 2 ) We develop an uncertainty quantifier , and integrate it into the value iteration procedure as the penalty function . Such a penalty function can effectively screen the “ spuriously correlated trajectories ” , i.e. , which possibly happen to appear in the experience data , but are actually unrelated to the optimal policy , but by chance induce large cumulative rewards and hence may potentially mislead the learned policy . Theoretically , we establish a data-dependent upper bound on the suboptimality of SAFARI for MFMARL without the stringent assumptions on the sufficient coverage of the experience data ( e.g. , finite concentrability coefficients ( Chen and Jiang , 2019 ) or uniformly lower bounded densities of visitation measure ( Yin et al. , 2020 ) ) . More specifically , we only assume that the experience data of N training trajectories contains enough information about the optimal policy . Then we prove that for an episodic MF-MARL problem with a horizon H , SAFARI attains a sub-optimality gap of O ( H2de↵/ p N ) , where de↵ is the effective dimension of the function class ( RKHS ) for parameterizing the value function and independent on the number of agents . In addition to the offline settings , our SAFARI algorithm can also be extended to MF-MARL under the online setting ( OMPPO algorithm ) , which is of independent interest . Details are provided in a longer technical report version , which is available upon request . The rest of this paper is organized as follows : Section 2 reviews related work on mean-field multiagent reinforcement learning and offline reinforcement learning for the single agent settings ; Section 3 introduces our problem setup of the mean-field MARL regime ; Section 4 introduces our proposed SAFARI algorithm ; Section 5 establishes the theoretical guarantees for SAFARI ; Section 6 presents numerical experiments on the multi-agent particle cooperative navigation scenario ; Section 7 draws a brief conclusion . 2 Related Work . • Mean-Field MARL . Existing literature has proposed various mean-field approximation approaches to model the population behavior of the agents for MARL with a large number , even infinitely many homogenous agents . Yang et al . ( 2017 ) investigate a mean-field game with deterministic linear state transitions , and reformulate it as a mean-field MDP , where the mean-field state lies in finitedimensional probability simplex . Yang et al . ( 2018 ) propose a mean-field approximation approach over actions , which approximates the interaction between any given agent and the population by the interaction between the agent ’ s action and the averaged actions of its neighboring agents . Such an averaging approach over the local actions , however , is only applicable when a sparse graph over agents is given , which requires extensive prior knowledge . Carmona et al . ( 2019 ) investigate a mean-field MDP from the perspective of mean-field control . As the mean-field state lies in a probability simplex and continuous in nature , they propose to discretize the joint state-action space such that conventional RL algorithms can be applied . Wang et al . ( 2020 ) investigate a mean-field MDP motivated by permutation invariance . They require a central controller managing the actions of all the agents , and therefore is restricted to handling the curse of many agents from the exponential blowup of joint state space . More recently , Li et al . ( 2021 ) investigate a similar mean-field MDP , which allows agents to make their own local actions without resorting to a centralized controller . All these methods focus on the online settings . In comparison , our proposed SAFARI algorithm and theory focus on the offline settings . • RL for Mean-Field Game . Our work is also related to the literature that studies RL methods for mean-field games ( Huang et al. , 2003 ; Lasry and Lions , 2006a , b ; Huang et al. , 2007 ) . Such a game can be viewed as the infinite-agent limit of general-sum Markov game with homogeneous agents , and the aggregated effect of the other agents is also summarized as a mean-field state . In contrast to mean-field MARL , the solution concept of mean-field game is the Nash equilibrium , which corresponds to a pair of a local policy ⇡⇤ of the representative agent and a mean-field state d⇤ satisfying the following two properties : ( i ) when the mean-field state is set to d⇤ , ⇡⇤ is the optimal policy of the representative agent ; and ( ii ) when all agents adopt ⇡⇤ , the resulting mean-field state is d ⇤ . Recently , there are many recent works developing RL methods for solving mean-field games . See , e.g. , Guo et al . ( 2019 , 2020b , a ) ; Fu et al . ( 2019 ) ; Anahtarcı et al . ( 2019 ) ; Anahtarci et al . ( 2020 ) ; Anahtarcı et al . ( 2020 ) ; Perrin et al . ( 2020 ) ; Elie et al . ( 2020 ) ; uz Zaman et al . ( 2020 ) ; Cui and Koeppl ( 2021 ) and the references therein . Most of these methods adopts a double-loop structure , where the inner loop finds the optimal local policy given the current mean field state and the outer loop updates the mean-field states . Moreover , these works often assume the data distribution is well-explored with either a generative model ( Azar et al. , 2012 ) or bounded concentrability coefficients ( Munos , 2007 ) . Our mean-field MARL problem is similar to the inner-loop problem of finding the optimal local policy in mean-field games . In contrast to these existing works , our algorithm and theory can be applied to datasets that are possibly not well-explored . Moreover , as mean-field MARL and mean-field games are different models , our work is not directly comparable to these works . • Offline Single-Agent RL . Our work is also closely related to the literature on offline single-agent RL , which often focuses on either policy evaluation or policy optimization . In particular , in policy evaluation , the goal is to estimate the value function of a target policy , whereas in policy optimization , we aims to learn the optimal policy , which can be achieved via estimating the optimal value function . For both these tasks , in the offline setting , due to the lack of continuing exploration ( Szepesvári , 2010 ) , the distribution shift ( Levine et al. , 2020 ) is a fundamental challenge . That is , the trajectories in the dataset and those induced by the target policy or the optimal policy might have diverse distributions . Such a challenge is further exacerbated when function approximators are adopted to represent the desired value functions . To overcome such a challenge , most of the existing theoretical works imposes certain well-exploration assumptions on the dataset . Some of commonly made assumptions include uniformly lower bounded visitation measure of the behavior policy , uniformly upper bounded importance sampling ratio , and bounded concentrability coefficients . See , e.g. , Antos et al . ( 2007 , 2008 ) ; Munos and Szepesvári ( 2008 ) ; Farahmand et al . ( 2010 , 2016 ) ; Scherrer et al . ( 2015 ) ; Jiang and Li ( 2016 ) ; Thomas and Brunskill ( 2016 ) ; Farajtabar et al . ( 2018 ) ; Liu et al . ( 2018 ) ; Xie et al . ( 2019 ) ; Nachum et al . ( 2019a , b ) ; Tang et al . ( 2019 ) ; Zhang et al . ( 2020b ) ; Chen and Jiang ( 2019 ) ; Kallus and Uehara ( 2019 , 2020 ) ; Jiang and Huang ( 2020 ) ; Uehara et al . ( 2020 ) ; Duan et al . ( 2020 ) ; Yin and Wang ( 2020 ) ; Yin et al . ( 2020 ) ; Nachum and Dai ( 2020 ) ; Yang et al . ( 2020a ) ; Fu et al . ( 2020b ) ; Fan et al . ( 2020 ) ; Xie and Jiang ( 2020a , b ) ; Liao et al . ( 2020 ) ; Zhang et al . ( 2020a ) ; Ren et al . ( 2021 ) and the references therein . However , in practice , such assumptions on the dataset often fail to hold ( Fujimoto et al. , 2019 ; Agarwal et al. , 2020 ; Fu et al. , 2020a ; Gulcehre et al. , 2020 ) . In light of this , there is a line of recent works that proposes various pessimism-based offline single-agent RL algorithms with empirical evidence or theoretical guarantees ( Yu et al. , 2020 ; Kidambi et al. , 2020 ; Kumar et al. , 2020 ; Liu et al. , 2020b ; Buckman et al. , 2020 ; Jin et al. , 2020b ; Xiao et al. , 2021 ) . In particular , Liu et al . ( 2020b ) propose a regularized variant of fitted Q-iteration ( Antos et al. , 2007 , 2008 ; Munos and Szepesvári , 2008 ) , which is shown to attain the optimal policy within a restricted policy class without assuming the dataset is well-explored . Moreover , with an arbitrary dataset , Buckman et al . ( 2020 ) ; Jin et al . ( 2020b ) ; Xiao et al . ( 2021 ) identify the critical role of pessimism in achieving offline sample efficiency . Among these works , our work is particularly related to Jin et al . ( 2020b ) , which develops a pessimistic variant of the value iteration algorithm with finite-dimensional linear function approximation . In comparison , our SAFARI algorithm extends such an algorithm to mean-field MARL and we propose to employ RKHS mean embedding for handling the difference between finite-agent empirical mean-field state and its infinite-agent counterpart . Moreover , our algorithm and analysis involve infinite-dimensional RKHS , which strictly generalizes those in Jin et al . ( 2020b ) . Notation : Given a space X , we denote M ( X ) as the collection of probability distributions supported on X . Let u , v , w 2 H be elements in a Hilbert space , we denote hu , vi as the inner product , and u⌦v as the outer product satisfying ( u⌦ v ) w = u hv , wi . For a scalar a , we denote { a } + = max { 0 , a } . We use O ( · ) to hide absolute constants and log factors .

Summary: The paper (mainly) provides a pessimism based offline RL algorithm for MF-MARL. It extends existing literature by 1. Generalizing pessimism based offline RL algorithms from single agent to MF-MARL 2. Considering MF-MARL from an offline regime It also extends existing literature on online MF-MARL by converting the pessimism penalty to a optimistic bonus and showed the sample efficiency of the algorithm under this regime.