Pessimism Meets Invariance: Provably Efficient Offline Mean-Field Multi-Agent RL

p N ) , where de↵ is the effective dimension of the function class for parameterizing the value function , but independent on the number of agents . Numerical experiments are provided . 1 Introduction . Significant progress has been made towards multi-agent reinforcement learning ( MARL ) for many prominent sequential decision making problems , such as social welfare optimization ( Leibo et al. , 2017 ) , fleet control of autonomous vehicles ( Shalev-Shwartz et al. , 2016 ) and playing multiplayer online battle arena ( MOBA ) games ( Berner et al. , 2019 ) . As the joint state and action space scales exponentially with the number of agents , however , MARL becomes computationally expensive . One remedy is the mean-field regime when an extremely large number of homogenous agents are involved , e.g. , social welfare optimization . The effect of each agent on the overall multi-agent system can become infinitesimal , and therefore all agents can be considered interchangeable/indistinguishable ( Yang et al. , 2018 ; Carmona et al. , 2019 ; Li et al. , 2021 ) . Accordingly , the interaction among agents can be captured by some mean-field quantity such as the empirical distribution of states , and therefore each agent only needs to find the best response to the so-called “ mean-field state ” , which avoids the curse of many agents . Most existing results on mean-field MARL ( MF-MARL ) are for the online setting ( Yang et al. , 2018 ; Zhang et al. , 2019 ) , where the agents can interact with the environment during training . However , such interaction during training can be prohibitive for some important applications ( Leibo et al. , 2017 ; Mandel et al. , 2014 ; Jaques et al. , 2019 ; Levine et al. , 2020 ) . Taking social welfare optimization as an example , repeatedly conducting social experiments on human being can be unaffordable or even unethical in the societal systems . Therefore , we can only consider the offline settings , i.e. , we learn the optimal policy based on some pre-collected experience data ( Levine et al. , 2020 ) . Unfortunately , ⇤Extension to online setting is provided in a longer technical report version , which is available upon request . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . existing offline reinforcement learning ( RL ) algorithms and theories all focus on the single agent settings , and no algorithms and theories have been developed for MARL under the offline settings , regardless of the mean-field regime or not . To bridge such a critical gap , we propose the first pessimistic algorithm – named SAFARI ( peSsimistic meAn-Field vAlue iteRatIon ) for mean-field MARL , which can provably achieve sample efficiency under the offline setting . Our proposed algorithm contains two important components : ( 1 ) To incorporate the permutation invariance of the homogenous agents , we adopt a RKHS ( Reproducing Kernel Hilbert Space ) mean-embedding approach for approximating value functions , which avoids the exponential blowup of the agents ’ state and action spaces ; ( 2 ) We develop an uncertainty quantifier , and integrate it into the value iteration procedure as the penalty function . Such a penalty function can effectively screen the “ spuriously correlated trajectories ” , i.e. , which possibly happen to appear in the experience data , but are actually unrelated to the optimal policy , but by chance induce large cumulative rewards and hence may potentially mislead the learned policy . Theoretically , we establish a data-dependent upper bound on the suboptimality of SAFARI for MFMARL without the stringent assumptions on the sufficient coverage of the experience data ( e.g. , finite concentrability coefficients ( Chen and Jiang , 2019 ) or uniformly lower bounded densities of visitation measure ( Yin et al. , 2020 ) ) . More specifically , we only assume that the experience data of N training trajectories contains enough information about the optimal policy . Then we prove that for an episodic MF-MARL problem with a horizon H , SAFARI attains a sub-optimality gap of O ( H2de↵/ p N ) , where de↵ is the effective dimension of the function class ( RKHS ) for parameterizing the value function and independent on the number of agents . In addition to the offline settings , our SAFARI algorithm can also be extended to MF-MARL under the online setting ( OMPPO algorithm ) , which is of independent interest . Details are provided in a longer technical report version , which is available upon request . The rest of this paper is organized as follows : Section 2 reviews related work on mean-field multiagent reinforcement learning and offline reinforcement learning for the single agent settings ; Section 3 introduces our problem setup of the mean-field MARL regime ; Section 4 introduces our proposed SAFARI algorithm ; Section 5 establishes the theoretical guarantees for SAFARI ; Section 6 presents numerical experiments on the multi-agent particle cooperative navigation scenario ; Section 7 draws a brief conclusion . 2 Related Work . • Mean-Field MARL . Existing literature has proposed various mean-field approximation approaches to model the population behavior of the agents for MARL with a large number , even infinitely many homogenous agents . Yang et al . ( 2017 ) investigate a mean-field game with deterministic linear state transitions , and reformulate it as a mean-field MDP , where the mean-field state lies in finitedimensional probability simplex . Yang et al . ( 2018 ) propose a mean-field approximation approach over actions , which approximates the interaction between any given agent and the population by the interaction between the agent ’ s action and the averaged actions of its neighboring agents . Such an averaging approach over the local actions , however , is only applicable when a sparse graph over agents is given , which requires extensive prior knowledge . Carmona et al . ( 2019 ) investigate a mean-field MDP from the perspective of mean-field control . As the mean-field state lies in a probability simplex and continuous in nature , they propose to discretize the joint state-action space such that conventional RL algorithms can be applied . Wang et al . ( 2020 ) investigate a mean-field MDP motivated by permutation invariance . They require a central controller managing the actions of all the agents , and therefore is restricted to handling the curse of many agents from the exponential blowup of joint state space . More recently , Li et al . ( 2021 ) investigate a similar mean-field MDP , which allows agents to make their own local actions without resorting to a centralized controller . All these methods focus on the online settings . In comparison , our proposed SAFARI algorithm and theory focus on the offline settings . • RL for Mean-Field Game . Our work is also related to the literature that studies RL methods for mean-field games ( Huang et al. , 2003 ; Lasry and Lions , 2006a , b ; Huang et al. , 2007 ) . Such a game can be viewed as the infinite-agent limit of general-sum Markov game with homogeneous agents , and the aggregated effect of the other agents is also summarized as a mean-field state . In contrast to mean-field MARL , the solution concept of mean-field game is the Nash equilibrium , which corresponds to a pair of a local policy ⇡⇤ of the representative agent and a mean-field state d⇤ satisfying the following two properties : ( i ) when the mean-field state is set to d⇤ , ⇡⇤ is the optimal policy of the representative agent ; and ( ii ) when all agents adopt ⇡⇤ , the resulting mean-field state is d ⇤ . Recently , there are many recent works developing RL methods for solving mean-field games . See , e.g. , Guo et al . ( 2019 , 2020b , a ) ; Fu et al . ( 2019 ) ; Anahtarcı et al . ( 2019 ) ; Anahtarci et al . ( 2020 ) ; Anahtarcı et al . ( 2020 ) ; Perrin et al . ( 2020 ) ; Elie et al . ( 2020 ) ; uz Zaman et al . ( 2020 ) ; Cui and Koeppl ( 2021 ) and the references therein . Most of these methods adopts a double-loop structure , where the inner loop finds the optimal local policy given the current mean field state and the outer loop updates the mean-field states . Moreover , these works often assume the data distribution is well-explored with either a generative model ( Azar et al. , 2012 ) or bounded concentrability coefficients ( Munos , 2007 ) . Our mean-field MARL problem is similar to the inner-loop problem of finding the optimal local policy in mean-field games . In contrast to these existing works , our algorithm and theory can be applied to datasets that are possibly not well-explored . Moreover , as mean-field MARL and mean-field games are different models , our work is not directly comparable to these works . • Offline Single-Agent RL . Our work is also closely related to the literature on offline single-agent RL , which often focuses on either policy evaluation or policy optimization . In particular , in policy evaluation , the goal is to estimate the value function of a target policy , whereas in policy optimization , we aims to learn the optimal policy , which can be achieved via estimating the optimal value function . For both these tasks , in the offline setting , due to the lack of continuing exploration ( Szepesvári , 2010 ) , the distribution shift ( Levine et al. , 2020 ) is a fundamental challenge . That is , the trajectories in the dataset and those induced by the target policy or the optimal policy might have diverse distributions . Such a challenge is further exacerbated when function approximators are adopted to represent the desired value functions . To overcome such a challenge , most of the existing theoretical works imposes certain well-exploration assumptions on the dataset . Some of commonly made assumptions include uniformly lower bounded visitation measure of the behavior policy , uniformly upper bounded importance sampling ratio , and bounded concentrability coefficients . See , e.g. , Antos et al . ( 2007 , 2008 ) ; Munos and Szepesvári ( 2008 ) ; Farahmand et al . ( 2010 , 2016 ) ; Scherrer et al . ( 2015 ) ; Jiang and Li ( 2016 ) ; Thomas and Brunskill ( 2016 ) ; Farajtabar et al . ( 2018 ) ; Liu et al . ( 2018 ) ; Xie et al . ( 2019 ) ; Nachum et al . ( 2019a , b ) ; Tang et al . ( 2019 ) ; Zhang et al . ( 2020b ) ; Chen and Jiang ( 2019 ) ; Kallus and Uehara ( 2019 , 2020 ) ; Jiang and Huang ( 2020 ) ; Uehara et al . ( 2020 ) ; Duan et al . ( 2020 ) ; Yin and Wang ( 2020 ) ; Yin et al . ( 2020 ) ; Nachum and Dai ( 2020 ) ; Yang et al . ( 2020a ) ; Fu et al . ( 2020b ) ; Fan et al . ( 2020 ) ; Xie and Jiang ( 2020a , b ) ; Liao et al . ( 2020 ) ; Zhang et al . ( 2020a ) ; Ren et al . ( 2021 ) and the references therein . However , in practice , such assumptions on the dataset often fail to hold ( Fujimoto et al. , 2019 ; Agarwal et al. , 2020 ; Fu et al. , 2020a ; Gulcehre et al. , 2020 ) . In light of this , there is a line of recent works that proposes various pessimism-based offline single-agent RL algorithms with empirical evidence or theoretical guarantees ( Yu et al. , 2020 ; Kidambi et al. , 2020 ; Kumar et al. , 2020 ; Liu et al. , 2020b ; Buckman et al. , 2020 ; Jin et al. , 2020b ; Xiao et al. , 2021 ) . In particular , Liu et al . ( 2020b ) propose a regularized variant of fitted Q-iteration ( Antos et al. , 2007 , 2008 ; Munos and Szepesvári , 2008 ) , which is shown to attain the optimal policy within a restricted policy class without assuming the dataset is well-explored . Moreover , with an arbitrary dataset , Buckman et al . ( 2020 ) ; Jin et al . ( 2020b ) ; Xiao et al . ( 2021 ) identify the critical role of pessimism in achieving offline sample efficiency . Among these works , our work is particularly related to Jin et al . ( 2020b ) , which develops a pessimistic variant of the value iteration algorithm with finite-dimensional linear function approximation . In comparison , our SAFARI algorithm extends such an algorithm to mean-field MARL and we propose to employ RKHS mean embedding for handling the difference between finite-agent empirical mean-field state and its infinite-agent counterpart . Moreover , our algorithm and analysis involve infinite-dimensional RKHS , which strictly generalizes those in Jin et al . ( 2020b ) . Notation : Given a space X , we denote M ( X ) as the collection of probability distributions supported on X . Let u , v , w 2 H be elements in a Hilbert space , we denote hu , vi as the inner product , and u⌦v as the outer product satisfying ( u⌦ v ) w = u hv , wi . For a scalar a , we denote { a } + = max { 0 , a } . We use O ( · ) to hide absolute constants and log factors .

Summary: This paper studies the Mean-Field Multi-Agent RL problem in the offline regime where pre-collected data is available. Under the assumption that data provides “sufficient” information regarding the optimal policy, the authors propose SAFARI, a pessimistic algorithm, and prove sample complexity guarantees to achieve a certain sub-optimality gap. The obtained data-dependent guarantees do not depend on the (large) number of agents and scale linearly with the effective dimension of the function class (used to parametrize the value function). Moreover, the authors propose another algorithm OMPPO for the online setting and obtain similar sublinear (in $N$; for some kernels) regret guarantees.