Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior
Authors: Kai Cui
Sascha H. Hauck
Christian Fabian
Heinz Koeppl
Venue: ICLR 2024 poster (2024)
Keywords: Mean Field Control
Multi-Agent Reinforcement Learning
Partial Observability
Collective Behavior
Primary area: reinforcement learning
TL;DR: We propose a MARL algorithm via MFC that scales cooperative problems to many agents and partial information by reducing to a single-agent MDP, with many applications in designing collective behavior.

Abstract: Recent reinforcement learning (RL) methods have achieved success in various domains. However, multi-agent RL (MARL) remains a challenge in terms of decentralization, partial observability and scalability to many agents. Meanwhile, collective behavior requires resolution of the aforementioned challenges, and remains of importance to many state-of-the-art applications such as active matter physics, self-organizing systems, opinion dynamics, and biological or robotic swarms. Here, MARL via mean field control (MFC) offers a potential solution to scalability, but fails to consider decentralized and partially observable systems. In this paper, we enable decentralized behavior of agents under partial information by proposing novel models for decentralized partially observable MFC (Dec-POMFC), a broad class of problems with permutation-invariant agents allowing for reduction to tractable single-agent Markov decision processes (MDP) with single-agent RL solution. We provide rigorous theoretical results, including a dynamic programming principle, together with optimality guarantees for Dec-POMFC solutions applied to finite swarms of interest. Algorithmically, we propose Dec-POMFC-based policy gradient methods for MARL via centralized training and decentralized execution, together with policy gradient approximation guarantees. In addition, we improve upon state-of-the-art histogram-based MFC by kernel methods, which is of separate interest also for fully observable MFC. We evaluate numerically on representative collective behavior tasks such as adapted Kuramoto and Vicsek swarming models, being on par with state-of-the-art MARL. Overall, our framework takes a step towards RL-based engineering of artificial collective behavior via MFC.

Published as a conference paper at ICLR 2024
LEARNING DECENTRALIZED PARTIALLY OBSERVABLE MEAN FIELD CONTROL FOR ARTIFICIAL COLLECTIVE BEHAVIOR
Kai Cui, Sascha Hauck, Christian Fabian, Heinz Koeppl
Dept. of Electrical Engineering and Information Technology, Technische Universität Darmstadt (kai. cui, heinz. koeppl (utee:uuuuuuaeeeeeee
ABSTRACT
Recent reinforcement learning (RL) methods have achieved success in various domains. However, multi-agent RL (MARL) remains challenge in terms of decentralization, partial observability and scalability to many agents. Mcanwhile, collective behavior requires resolution of the aforementioned challenges, and remains of importance to many state-of-the-art applications such as active matter physics, self-organizing systems, opinion dynamics, and biological or robotic swarms. Here, MARL via mean field control (MFC) offers potential solution to scalability, but fails to consider decentralized and partially observable systems. this paper, enable decentralized behavior of agents under partial information by proposing novel models for decentralized partially observable MFC POMFC), broad class of problems with permutation invariant agents allowing for reduction to tractable single-agent Markov decision processes (MDP) with single-agent RI. solution. We provide rigorous theoretical results, including dynamic programming principle, together with optimality guarantees for Dec-POMFC solutions applied to finite swarms interest. Algorithmically, we propose Dec-POMFC-based policy gradient methods for MARL via centralized training and decentralized execution, together with policy gradient approximation guarantees. In addition, we improve upon state-of off-the-art art histogram- interest also for fully observable MFC. We evaluate numerically on representative collective behavior tasks such as adapted Kuramoto and Vicsek swarming models, being on par with state-of-the-att MARL. Overall, our framework takes a step towards RL-based engineering of artificial collective behavior via MFC.
MFC by kernel methods, which is of separate
INTRODUCTION
Reinforcement learning (RL) and multi-agent RL (MARL) has found success in varied domains with few agents, including c.g. robotics (Polydoros & Nalpantidis, 2017), language models (Ouyang et al., 2022) or transportation (Haydari & Yılmaz, 2020). However, tractability issues remain for systems with many agents, especially under partial observability (Zhang et al., 2021b). Here, specialized approaches give tractable solutions, e.g. via factorizations (Qu et al., 2020; Zhang et al., 2021a). We propose a general, tractable approach for broad range of decentralized, partially observable systems.
Collective behavior & partial observability. Of practical interest the design of simple local interaction rules to fulfill global, cooperative objectives by emergence of global behavior (Vicsek & Zafeiris, 2012). For example, intelligent self-organizing robotic swarms provide many applications such as farming, and general design frameworks remains elusive (Hrabia et al., 2018: Schranz et al., 2021). Other domains include group decision-makinn making and opinion dynamics (Zha et al., 2020), biomolecular self-assembly (Yin et al., 2008), and active matter (Cichos et al., 2020; Kruk et al., 2020), e.g. nano-pan anno-particles (Nasiri Liebchen, 2022) or microswimmers (Narinder et al., 2018). Overall, there is need for scalable MARL under decentralization and partial information.
Scalable and partially observable MARL. Despite its many applications, decentralized coopera- tive control remains difficult problem even in MARL. (Zhang et al., 2021b), especially if coupled

---

Published as a conference paper at ICLR 2024
A
Liviigg Dec-Mro MOP
O
O
Uncertrallec enecuson
MPC--type Dec-POMOF)
Centralizd training
Figure I: A: Partially-observabee Vicsek problem: agents must align headings (arrows), but observe only partial information (e.g. heading distribution in grey circle for orange agent). B: The decen- tralized model as a graphical model (grey: observed variables). C: In centralized training, we also observe the mean field, guiding the learning of upper-level actions म. D: The solved limiting MDP.
with the simultaneous requirement of scalability. Recent scalable MARL methods include graphical decompositions (Qu et al., 2020; Zhang et al., 2021a) amongst others (Zhang et al., 2021b). However, most remain limited to full observability (Zhang et al., 2021a). One line of algorithms applies pairwise mean field (MF) approximations over neighbors (Yang et al., 2018), which has yielded decentralized, partially observable extensions (Subramanian et al., 2021; 2022). Relatedly, MARL based mean field games (MFG, non-cooperative) and mean field control (MFC, cooperative) focus on a broad class of systems with many exchangeable agents. While the theory for MFG is developed (Huang et al., 2006; Şen & Caines, 2019; Saldi et al., 2019). to the best of our knowledge, neither MFC-based MARL algorithms nor discrete-time MFC have been proposed under partial information and decentralizatiin, except in special linear- quuarr-ua cases (Tottori & Kobayashi, 2022; Wang et al., 2021). I. Further, MFGs have been useful for analyzing emergence of collective behavior (Perrin et al., 2021; Carmona et al., 2022), but less for "engineering" collective behavior to achieve global objectives as in MFC, which is our focus. This is in contrast to rational, selfish agents, as decomposition of global objectives into per-agent rewards is non-trivial (Waelchli et al., 2023; Kwon et al., 2023). Beyond scalability to many agents, general MFC for MARL is also not yet scalable to high-dimensional state-actions due to discrctization of the simplex (Carmona et al., 2019b; Gu al., 2021), except in linear-quadratie models (Fu et al., 2019; Carmona et al., 2019a). Instead, consider general discrete-time MFC and scale to higher dimensions via kernels. We note that our model has similar flavor to TD-POMDPs (Witwicki & Durfee, 2010), as the MF also abstracts influence from all other agents. However, TD-POMDP addresses different types of problems, as it considers local per-agent states, while the MF is both globally shared and influenced by all agents.
Our contribution. tractable framework for cooperative control, that can handle decentralized, partially observable systems, is missing. By the preceding motivation, we propose such a framework as illustrated in Figure 1. Our contributions may be summarized as (i) proposing the first discrete- time MFC model with decentralized and partially observing agents: (ii) providing accompanying approximation theorems. reformulations to tractable single-agent Markov decision process (MDP), and novel optimality results over equi-Lipschiz policies; (iii) establishing MARI. algorithm with policy gradient guarantees; and (iv) presenting kernel-based MFC parametrizations of separate interest for general, higher-dimennsioall MFC. The algorithm is verified on classical collective swarming behavior models, and compared against standard MARL. Overall, our framework steps toward tractable RL-based engineering of artificial collective behavior for large-scale multi-agent systems.
DECENTRALIZED PARTIALLY OBSERVABLE MFC
In this section, we introduce the motivating finite MFC-type decentralized partially observable control problem, as special case of cooperative, general decentralized partially observable Markov decision processes
(Bernstein et al., 2002; Olichoek & Amato, 2016)). We then proceed to simplify in three steps of (i) taking the agentt limit, (ii) relaxing partial observability during training, and (iii) correlating agent actions during training, in order to arrive at a tractable MDP with optimality guarantees, see also Figures! 1 and 2. Proofs are found in Appendices D-S.
In a nutshell, Dec-POMDPs are hard, and hence we reformulate into the Dec-POMFC, for which we develop a new theory for optimality of Dec-POMIC solutions in the finite Dec-POMDP. The
2

---

Published as a conference paper at ICLR 2024
Algorithmic
Dec-MFC MDP (Centralized, observable). Section
selves via didermnntte appectiest MFC poicy (Proposation 2)
Dec-MFC (Decontralized, moan obsorvablo) Section
reformutate MDP
open-loop sequercca .... aeceeeeeee s.c.............................
(Propssiion
Dec-POMFC (Desentralzzdd partilly obsovvable) Section 2.2
solves propagation of choos feld fint. Theorem
System of interest
Finite MFC-type Dec-POMDP (Decentralized, parbally observable). Section 2.1
Figure Three steps of approximation (mean field limit, open-loop control, and MDP reformulation) allow us to reformulate the broad class of MFC-type Dec-POMDP to a tractable Dec-MFC MDP.
solution of Dec--oMMFF itself also remains hard, because its MDP is not just continuous, but infinite- dimensional for continuous state-actions. The MDP is later addressed in Section 3 by (i) kernel parametrizations and (ii) approximate policy gradients on the finite Dec-POMDP (Theorem 3).
2.1 MFC-TYPE COOPERATIVE MULTI-AGENT CONTROL
To begin, define the finite Dec-POMDP of interest, which is assumed to be MFC-type. In other words, (i) agents are permutation invariant, i.e. only the overall distribution of agent states matters, and (ii) agents observe only part of the system. We assume agents i [N] NN. endowed with random states x), observations yf and actions ui at times t E T = N from compact metric state, observation and action spaces X. y.u (finite or continuous). Agents depend on other agents only via the empirical mean field ju == EiE[N] d2g- Policies are memory-less and shared by all agents, archetypal of collective behavior under simple rules (Hamann, 2018), and of interest to compute-constrrinedd agents, including e-g. nano-particles or small robots. Optionally, memory and history-dependence can be integrated into the state, Appendix E. Agents act according policy ı E from a class II C "X of policies, with spaces of probability measures P(-), equipped with the 1-Wasserstein metric W, (Villani, 2009). Starting with initial distribution No. zo Mo. the MFC-type Dec-POMDP dynamics are
di~m((uf vf), r/41 ~ P(Zt41 AL for all (i, f) [N]x T, with transition kernels P: X'XUxx(X) P(X), Ph; X'x((X) - P()). ExeT>~>(ri
y/ ~ PV oi,
(1)
objective N (T)
to maximize over æ E under reward function r: P(X)
IR, and discount factor 7 (0, 1). Results generalize finite horizons, average per-agent rewards :per: X + R, r(4") = dpe and joint titte-bbeevaaton--ace
MFs via enlarged state space.
Since general Dec-POMDPs are hard (Bernstein et al., 2002), our model establishes tractable special case of high generality. Standard MFC already covers broad range of applications, e.g. see surveys for finance (Carmona, 2020) and engineering (Djehiche al., 2017) applications, which can now be handled under partial information. In addition, many classical, inherently partially observable models are covered by MFC-type Dec-POMDPs, such as the Kuramoto or Vicsek models in Section 4, where many-agent convergence is known as propagation of chaos (Chaintron & Diez, 2022).
2.2 LIMITING MFC SYSTEM
In order to achieve tractability for large multi-agent systems, the first step to take the infinite-agent limit. By law of large numbers (LLN), this allows to describe large systems only by the MF μ4. Consider a representative agent as in (1) with states To ~ Po. Ft+1 ~ P 22, Ue mt). observations yt (" Zz, 44) and actions u, n,(u, v:). Then, its state probability law replaces the empirical state distribution, informally μ4 = L(x+) = lim 14f Looking only at the MF, we hence obtain the decentralized partially observable MFC (Dec-POMFC) system
Me+1 L(x++1) T(M Re) = [I] P(x, u, Me y)) P" æx, x,, (c)
by deterministic transitions T: P(X) P(U)" + P(X) and objective J(T) o (T()))
Approximation guarantees. Under mild continuity assumptions, the Dec-POMFC model in (2) constitutes good approximation of large-scale MFC-type Dec-POMDP in (1) with many agents.
3

---

Published as a conference paper at ICLR 2024
Assumption la. The transitions P, Pd and rewards are Lipschitz with constants Lp, Lpv, Lr- Assumption 1b. The class of policies ll is the set of all Lu-Lipschitzz policies for some Lı > O, i.e. for all t E T and π E II, we have that Te: y - P(U) is Ln-Lipschitz Alteraatively, we may assume unrestricted policies if (i) observations only depend on an agent's 's state, and (ii) |X] < co.
Lipschitz continuity of the model commonly assumed (Huang et al., 2006; Gu et al., 2021; Mondal al., 2022), and in general at least (uniform) continuity is required: Consider counterexample with uniform initial po over states A, B. If dynamics, observations, or rewards jump between regimes at 0.5, the finite system will randomly experience all regimes, while limiting MFC
() B) experiences only the regime at ()) = pe( B) = 0.5. Meanwhile, Lipschitz policies are not only stan- dard in MFC literature (Pasztor al., 2021; Mondal al., 2022) by neural networks (NNs) (Araujo et al., 2023), but also fulfilled for finite Y trivially without Ioss of generality (LI == diam)(4)) ) and for continuous y by kernel parametrizations in Section 3. We extend MFC approximation theorems (Gu et al., 2021; Mondal et al., 2022; Cui et al., 2023) to partial observations and compact spaces.
(u(A) ju(
Theorem 1. Fix an equicontinuous family of functions F C RP(2), Under Assumptions la-lb, the MF converges in the sense of supxen SuP EE [[](m]]) [m()]]] + +0 all times & E T.
The approximation rate is (1/V/)) for finite state-actions, using equi-Lipschitz F (Appendix D). Hence, the easier Dec-POMFC simplifies otherwise hard Rec-POMDPs. Indeed, we later show that such optimal Lipschitz Dec-POMIC policies are guaranteed to exist via closedness of joint-measures under equi-Lipschitz kernels (Appendix K), see Propositions 1, and Theorem 2 later.
Corollary 1. Under Assumptions la-lb, any optimal Dec-POMFC policy π E arg m (TT) e-optimal the MFC-type Dec-POMDP, J (T) Z se JN (x') - E, with & + +0ss - 00.
2.3 REWRITING POLICIES WITH MEAN FIELD OBSERVATIONS
Now introducing the next system for reduction to MDP, writing P. etc., let policies depend also on j4, i.e. policies "observe" the mean field. While we could reason that agents might observe the MF or use filtering to estimate it (Aström, 1965), more importantly, the limiting MF is deterministic. Therefore, w.l.o.g. we obtain the decentralized mean field observable MFC (Dec-MFC) dynamics
Pi+1 = TTae, (w,)) = /I] P(x, u, (edu v, Po) x. (d)))
(3)
with shorthand (Pe) # Pe), initial jo po and according objective J(#) 2im00'0(i 7'T(pe) to optimize over (now MF-dependent) policies म E πT = P(U))xx(8xxxT
Deterministic open-loop control transforms optimal Dec-MFC policies π E arg ma J(z') into optimal Dec-POMFC policies и E arg max J(x) with decentralized execution, and vice versa: For given R. compute deterministic MFs (ão, ......) via (3) and let (TF) by Ti v,(du y) =
a [du y, Pt). Analogously, represent π E 11 by ã E 11 with constant म. = ne for all v.
Proposition 1. For any ग E I1, define (āo- P1, as in (3). Then, for = 2 (t) 11, we have ((x) = J (T)) Inversely, for any π E 11, let 7. (D) = for all D, then again ( (TT).
Corollary 2. Optimal Dec-MFC policies मे arg max J(7') yield optimal Dec-POMFC policies ( i.e. J()(T)) = sup (
Knowing initial is often realistic, as deployment is commonly for well-defined problems of interest. Even then, knowing no is not strictly necessary (Section 4). In contrast to standard deterministic open-loop control, (i) agents have stochastic dynamics and observations, and (ii) agents randomize actions instead of playing a trajectory, still leading to quasi determmtetiiiiitiic MFs by the LLN.
2.4 REDUCTION TO DEC-MFC MDP
MDP with more tractable theory and algorithms, writing jú, â etc. The recent MFC MDP (Pham & Wei, 2018; Carmona et al., 2019b; Gu et al., 2019) reformulates fully observable MFC as MDPs with higher-dimensional acttinnt Similarly, we reduce Dec-MFC to an MDP with joint otste-observation. action distributions as its MDP actions. The Dec-MFC MDP has states (4 P(X)) and actions by H(f)) C P(X's y x x() in the set of joint hy ja X PM (f+ 60 it.
Lastly, we reformulate
4

---

Published as a conference paper at ICLR 2024
under any Lu Lip-iisshitz policy #1 E P(U))> Here, @ K is the product measure of measure v and kernel K, and yK is the measure yK = K(- z))(dz). For n; E P(4)) May E P(X x )), we write May @ &t by letting n, constant on X'. In other words, the desired joint h, results from all agents replacing the previous system's 's policy मe by lower-level policy Rt, which may be reobtained from hy (Appendix K, disintegration (Kallenberg, 2021)). Equivalently, identify H(1) with J and classes of te yielding the same joint, and in practice we parametrize म.. Thus, we obtain the MDP dynamics
//] P(z, (e du)
h + (ix),
pu+1 = T(i, he)
(4)
for DeC-MFC MDP policy a E fî and objective ((t) E EE (r((e))] The Dec-MFC MDP policy स is "upper-level", as we sample he from म. to apply the lower-level policy [e[ to all agents.
Guidance by mean field dependence. Intuitively, the MF guides policy search in potentially hard, decentralized problems, and reduces to a single-agent MDP where we make some existing theory compatible. First, we formulate a dynamic programming principle (DPP), i.e. exact solutions by Bellman's ' equation for the value function ((u) = SuPhEM(H) r(n) + 7V (H,, ft) (Hernández- Lerma & Lasserre, 2012). Here, a central theoretical novelty is closedness of joint measures under equi-Lipschitz policies (Appendix K). Concomitantly, we obtain optimality of stationary deterministic f. For technical reasons, only here we assume Hilbertian y (e.g. finite or Euclidean) and finite U. Assumption 2. The observations y are metric subspace of a Hilbert space. Actions I are finite. Theorem 2. Under Assumptions la-lb and 2, there exists an optimal stationary, deterministic policy it for the Dec-MFC MDP, with (()) E arg max)eH()) r(u) + (T,,
Decentralized execution. Importantly, guidance by MF is only for training and not execution. An optimal upper-level policy á E arg max ((t) is optimal also for the initial system, if it is deterministic, and an optimal one exists by Theorem 2. The lover-level policies म. = #. are obtained by inserting the sequence of MFs Pao, pa,... into ât, and remain non-stationary stochastic policies.
Proposition 2. For deterministic ît E fT, let fú as in (4) and स = (ff) by (v) = # for all v, then j(#) (T). Inversely, for ā E II, let fte = & (v) @ (z(v) for all v, then J(+)) = J(()).
Note that the determinism of the upper-level policy is strictly necessary: A simple counterexample is a problem where agents should choose to aggregate to one state. If the upper-level policy randomly chooses between moving all agents to either A B, then a corresponding random agent policy splits agents and fails to aggregate. At the same time, randomization of agent actions remains necessary for optimality, as the problem of equally spreading would require uniformly random agent actions.
Complexity. Tractability of multi-agent control heavily depends on information structure (Mahajan et al., 2012). General POMDPs have coubly-exponential complexity (NEXP, Bernstein et al. (2002)) and are harder than fully observable control (PSPACE, Papadimitriou & Tsitsiklis (1987)). In contrast, Dec-POMFC surprisingly imposes little additional complexity over standard MFC, as the MFC MDP remains deterministic in the absence of common noise correlating agents (Carmona et al., 2016). An analysis with common noise is possible, c.g. if observing the mean field, but out of scope.
DEC-POMFC POLICY GRADIENT METHODS
All that remains is to solve Dec-MFC MDPs. As we obtain continuous Dec-MFC MDP states and actions even for finite X. y. Ut and infiitt-diimensional ones for continuous X, y. 24 a value-based approach can be hard. Our policy gradient (PG) approach allows finding simple policies for collective behavior, with emergence of global intelligent behavior described by rewards r, under arbitrary (Lipschitzz) policies. For generality, lower-level (Lipschitz, (Araujo et al., 2023)) NNs policies could be considered akin to hypernetworks (Ha et al., 2016), the resulting distributions over NN parameters as MDP actions are too high- dimensional and failed in our experiments. We directly solve finite-agent MFC-type Dec-POMDPs by solving the DFC MDP in the background. Indeed, the theoretical optimality of Dec-MFC MDP solutions is guaranteed over Lipschitz policies in II.
NN upper-level and kernel lower-level policies. While
Corollary 3. Under Assumptions la-lb, a deterministic Dec-MFC solution के E arg max. J(*') is e-optimal in the Dec-POMDP, J (a (+))) 2 supx'en JJ (T') 6, with + 0 as N + 00.
5

---

Published as a conference paper at ICLR 2024
Histogram vs. kernel parametrizations. Except for linear-quadrate algorithms (Wang al., 2021; Fu et al., 2019; Carmona et al., 2019a), the only approach to learning MFC in continuous spaces X' _ IR", n E N (and here ))) is by partitioning and "discretizing" (Carmona et al., 2019b; Gu et al., 2021). 1). Unfortunately, partitions fail Lipschitzness and hence approximation guarantees, even in standard MFC. Instead, we kernel representations for MFs 14 and lower-level policies #t.
We represent -valuu MDP states 148' not by counting agents in each bin, but instead mollify around each center 2% E X of Mx bins b [Mx] using kernels. The result is Lipschitz and approximates histograms arbitrarily well (Miculescu, 2000, Theorem 1). Hence, we obtain input logits I = f (=(2,,)ddp = N ΣiE[N] x(116. 2z.) for some kernel w: X' X + R and b E [Mx]] Output logits constitute mean and log-standard deviation of diagonal Gaussian over parameter representations Ç E of #t. We obtain Lipschitz #e by representing # via My points ys E ) such that #t (u v) = 2 k(86, y)pp(u))) (u) Ene[M)) (:(6..)) Here, we consider Lx-Lipschitz maps As from parameters & E E to distributions ps = () E P((t) with compact parameter space E, and for kernels choose RBF kernels s(2, v) exp(- 1|= y / (2o²)) with some bandwidth o² O.
Proposition 3. Under RBF kernels K. for any & and Euclidean Vi lower-level policies A(E)(- y) = Eee(My) K (yy,, /))o(E// Eme(Mry] k(y6,3) are Lny-Lipchitzz in y as in Assumption 1b, whenever o² expp
2 d)am()))) Z In diam(Y) diam(U)) max llyll. and such o² > 0 always exists.
Proposition 3 ensures Assumption lb if needed. To achieve optimality by Corollary 3, deterministic policies commonly result from convergence of stochastic PGs, taking mean actions, or are guaranteed by deterministic PGs (Silver et al., 2014; Lillicrap et al., 2016). Beyond allowing for (i) Lipschitz guarantees, and (ii) finer control over agent actions, another advantage of kernels is (iii) the improved complexity over histograms. Even a histogram with only 2 bins per dimension requires 2d bins in d-dimensional spaces, while kernel representations may place e.g. points per dimension, improving upon the otherwise necessarily exponential complexity, see also Appendix A for empirical support.
Direct multi-agent reinforcement learning algorithm. Applying RL directly to the Dec-MFC MDP would be satisfactory only under known MFC models. Importantly, (i) we do not always have access the model, and (ii) even if we do, parametrizing MFs in arbitrary compact X' is hard. Instead, it is more practical and tractable to train on a finite system. Our direct MARL approach hence trains on a finite N-agent MFC-type Dec-POMDP of interest, in a model-free manner. In order to exploit the underlying MDP, our algorithm assumes during training that (i) the MF is observed, and (ii) agents can correlate actions (e.g. centrally, or sharing seeds). Therefore, the finite system (1) is adjusted for training by correlating agent actions on a single centrally sampled lower-level policy म.. Now write (E, (A as density over parameters & E E under a base measure (discrete, I Lebesgue). Substituting & as actions parametrizing h, in the MDP (4), e.g. by using RBF kernels, yields the centralized training system as seen in Figure for stationary policy 2 parametrized by 0,
#. ((E), (
& * (7t"), )
(5)
/ ~
P. (
ut
vt),
El41 ~ P( 3 AM), Vi [N].
Policy gradient approximation. Since we train on a finite system, it is not immediately clear whether centralized training really yields the PG for the underlying Dec-MFC MDP, also in existing literature for learning MFC. We will show this practically relevant fact up to an approximation. The general PG for stationary 4 (Sutton et al., 1999; Peters & Schaal, 2008) is JoJ(
(1 ----)>--eaaa - (n)) [Q (M, VV log #0 [i] with (p. = E (rr(()) | io 14. Eo = <) under parametrized actions& & in (4), and using sums daa = (1 - 7) Eier>' (pe) of laws of ie under 2²_ Our approximation motivates MFC for MARL by showing that the underlying background MeC-MFC MDP is approximately solved under Lipschitz parametrizations, e.g. normalize parameters& & to finite action probabilities, or use bounded diagonal Gaussian parameters.
Assumption 3. The policy # (4) and its log gradient Vo log ( μ)) are Lip I -ipschitz in yu and & (or alternatively in p. for any &, and uniformly bounded). The A(E) v) == Es k(Bb, y) ()) 2i k(yb, 2). with kernels k and Lx-iischiitz >o; E + P(U).
map is
Existtng Q-L g-LLeanning with kernel regression (Gu et al., 2021) is for finite states X' with kernels on P(22), and learns on the MFC MDP. We allow continuous y by kernels on itself, and leam on the finite-agent system.

---

Published as a conference paper at ICLR 2024
Algorithm 1 Dec-POMIFPPO (during centralized training)
1: for iteration 71 1, 2, ... do
2: for time f = 0,
Be do 3: Sample central MeC MDP action #e = A(Er). ~ # (p.))
4: for agent = . do
5:
Sample per-agent action ūt ~ #r (ū) Di) for observation o/-
6:
Perform actions, observe reward r r( ), next MF pi termination flag de+1 E {0, 1].
7:
for updates i = =1,,... 1,.... NNPr do
8:
Sample mini-batch b. 66] = bien from data B = (( &t, rf" de+1 (441)
9:
Update policy #6 via PPO loss VoLg on b, using GAE (Schulman al., 2016).
10:
Update critic V via critic Lz-loss Vo Lay on b.
Theorem 3. Centralized training on system (5) approximates the true gradient of the underlying Dec-MFCC MDP, i.e. under RBF kernels K. as in Proposition 3, Assumptions la-lb and 3, as N 0o,
(|0 -) ( (H) (Ô0 (v.
log # m)] VoJ( |||
with d (1 -1) Et and Q (v. = 1'r((M))) Mo 1. = E)-
The value function Q² the finite system then substituted actor-critic manner by on-policy and critic estimates. The Lipschitz conditions of H in Assumption 3 are fulfilled by Lipschitz NNs (Pasztor et al., 2021; Mondal et al., 2022; Araujo et al., 2023) and our parametrizations. The approximation is novel, building foundation for MARL via MFC directly problem. Our results also apply to fully observable MFC by 3/- = Lt. Though gradient estimates allow convergence guarantees in finite MDPs (e.g. Qu et al. (2020, Theorem 5)), Dec-MFC MDP state-actions are always non non-finite. In practice, we use empirically more efficient proximal policy optimization (PPO, Schulman et al. (2017): Yu al. (2022)) obtain the decentralized partially observable mean field PPO algorithm POMPPO,, Algorithm 1).
finite MARL
By Theorem 3, we may learn directly on the MFC-type Dec-POMDP system (1). During training, the algorithm (i) assumes to observe the MF, and (ii) samples only one centralized ht. Knowledge of the MF during training aligns our framework with the popular centralized training, decentralized execution (CTDE) paradigm. During execution, decentralized policies suffice for near-optimality by Corollary without agents knowing the MF or coordinating centrally. Decentralized training can also be achieved, if the MF is observable and all agents use the same seed to correlate their actions.
4 EVALUATION
In this section, we empirically evaluate our algorithm, comparing against independent and multi-agent PPO (IPPO, MAPPO) with state-of-the- performance (Yu et al., 2022; Papoudakis al., 2021). For comparison, we share hyperparameters and architectures between algorithms, see Appendices A-C.
Problems. In the Aggregation problem we consider a typical continuous single integrator model, commonly used in the study of swarm robotics (Soysal & Sahin, 2005; Bahgeçi & Sahin, 2005). Agents observe their own position noisily and should aggregate. The classical Kuramoto model is used to study synchronization of coupled oscillators, finding application not only in physics, including quantum computation and laser arrays (Acebrón al., 2005), but also in diverse biological systems, such as neuroscience and pattern formation in self-organizing systems (Breakspear et al., 2010; Kruk et al., 2020). Here, via partial observability, we consider version where each oscillator can see the distribution of relative phases of its neighbors. Finally, we implement the Kuramoto model on a random geometric graph (e.g. (Diaz-Guilera et al., 2009)) via omitting movement in its independent generalization, the Viesek model (Vicsek et al., 1995; Viesek & Zafeiris, 2012). Agents j have two-dimensional position P and current headings df. be controlled by their actions. The key metric of interest for both Kuramoto and Vicsek is polarization via the polar order parameter R = 1 E, exp(id)), ). Here, / ranges from 0 - fully unsynchronized - to 1 perfect alignment of agents. Experimentally, we consider various environments, such as the torus, Möbius strip, projective plane and Klein bottle. Importantly, agents only observe relative headings of others.
7

---

Published as a conference paper at ICLR 2024
107
100
200
(d)
(8)
-120
Stops X30
Stops XMI
Steps XXU
Steps x11
X1I Steps
Figure 3: Dec-POMFPPO training curves (episode return) with shaded standard deviation over 3 seeds for N 200 (a) Aggregation; Viesek on a (b): torus; (c): Möbius strip; (d): projective plane; (e): Klein bottle; and (f) Kuramoto on a torus.
IPPO
200
100
MAPPO
100
(2)
(c)
MF
1..00
0.25
0.50
0.75
1.00
0,00
1.25
0.50
0,15
1,00
0.00
0.25
0.50
0,75
1,00
Steps
XIP
Steps
XII
Staps
KIIP
Figure 4: Training curves (episode return) with shaded standard deviation over 3 seeds and N = 200, in (a) Aggregation (box), (b) Viesek (torus), (c) Kuramoto (torus). For comparison, we also plot the best return averaged over seeds for Rec-POMFPPO in Figure (MF).
Training results. In Figure 3 it is evident that the training process of MFC for many agents is relatively stable by guidance via MF and reduction to single-agent RL. In Appendix A, we also see similar results with significantly fewer agents and comparable to the results obtained with larger number of agents. This observation highlights that the training procedure yields satisfactory outcomes, even in scenarios where the mean field approximation may not yet be perfectly exact. These findings underscore the generality of the proposed framework and ability to adapt across different regimes. On the same note, we see by comparison with Figure 4, that our method is usually on par with state-of-the-art IPPO and MAPPO for many agents, e.g. here
200.
Verification of theory. In Figure 5, as the number of agents rises, the performance quickly tends to its limit, i.e. the objective converges, supporting Theorem I and Corollary well applicability to arbitrarily many agents. Analogously, conducting open-loop experiments on our closed-loop trained system in Figure demonstrates the robust generality of learned collective behavior with respect to the randomly sampled initial agent states, supporting Theorem 3 and Corollary 2.
Qualitative analysis. In the Viesek model, as seen exemplarily in Figure 6 and Appendix A, the algorithm learns to align in various topological spaces. In all considered topologies, the polar order parameter surpasses 0.9, with the torus system even reaching value close to 0.99. As for the angles at different iterations of the training process, as depicted in Figure 7, the algorithm gradually learns form concentrated cluster of angles. Note that the cluster center angle is not fixed, but rather changes over time. This behavior can not be observed in the classical Vicsek model, though extensions using more sophisticated equations of motion for angles have reported similar results (Kruk et al., 2020). For more details and further experiments or visualizations, we refer the reader to Appendices A-C. Figure 7 and additional figures, with similar results for other topologies in Appendix A, e.g. Figures 18-2,, illustrate the qualitative behavior observed across the different manifolds. Agents on the continuous torus demonstrate no preference for a specific direction across consccutive training rums. Conversely, agents trained other manifolds exhibit tendency to avoid the direction that leads to an angle flip when crossing the corresponding boundary. Especially for the projective plane topology, the agents tend to aggregate another reward for aggregation. For Aggregation in Figure 8, we also find successful aggregation
while aligning, even without adding
Reo
(b)
M
-50
100-
100
100
(b)
150
(0)
200
(5)
100
(6)
25
50
130
152
50
100
150
30
102
250
30
100
150
50
100
150
Num agomss
Num agentss 37
Num agerts
N
Num agants
N
Num agents
N
Num agents
Figure 5: The performance of the best of 3 DeMMPOO policies transferred N-agent systems (in blue, error bars for 95% confidence interval), averaged over 50 episodes, and compared against the performance the training system (in red). Problems (a)-(f) and training are as Figure 3.
8

---

Published as a conference paper at ICLR 2024
1= 26
2
2r
t=
t= 30
0.528
0.999
0.54
0.868
Figure 6: A, B: For the Viesek (torus) problem with forward velocity control, the open-loop behavior (B) shows little difference in performance of agents (rods, color indicating heading) over the closed- loop behavior (A). C: Visualization of agents (triangles) under the Vicsek model on the torus.
Iocetin
1:5
1001
120
2391
150
:001
0
Figure 7: A: Agent angle alignment in the Viesek model the torus, plotted as density over time; B: Alignment of agents the Vicsek model on the projective plane, as in Figure 6.
of agents in the middle. In practice, one may define any objective of interest. For example, we can achieve misalignment in Figure 8, resulting in polar order parameters on the order of magnitude of 10-2, and showing the generality of the framework.
Additional experiments. Some other experiments are discussed Appendix A, including the generalization of our learned policies to different starting conditions, a comparison of the Vicsek model trained or transferred to different numbers of agents, additional interpretative visualizations, similar success for the Kuramoto model, and favorable comparison between RBFs and histograms for higher dimensions, showing the generality of the framework and supporting our claims.
5 CONCLUSION AND DISCUSSION
Our framework provides novel methodology for engineering artificial collective behavior in rigor- ous and tractable manner, whereas existing scalable learning frameworks often focus on competitive or fully observable models (Guo al., 2023; Zheng et al., 2018). 8). We hope work opens up new applications of partially-observable swarm systems. Our method could be of interest due to (i) its theoretical optimality guarantees while covering a large class of problems, and (ii) its surprising simplicity in rigorously reducing complex Dec-POMDPs to MDPs, with same complexity as MDPs from fully observable MFC, thus allowing analysis of Dec- POMDPs via a tractable MDP.
The current theory remains limited to non-stochastic MFs, which in the future could be analyzed for stochastic MFs via common noise (Perrin et al., 2020; Cui et al., 2023; Dayanikli al., 2023). Further, sample efficiency could be analyzed (Huang et al., 2023), and parametrizations for history-dependent policies using more general NNs could be considered, e.g. via hypernetworks (Ha et al., 2016; Li al., 2023). Lastly, extending the framework to consider additional practical constraints and sparser interactions, such as physical collisions or via graphical decompositions, may fruitful.
1-
50
2T
50
I= 125
1.0
0.5 P
01003
Figure 8: A: Qualitative behavior for misalignment of agents in the Vicsek (torus) problem. B: The two-dimensional Aggregation problem, with agent distances to mean as colors.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work has been co-funded by the LOEWE initiative (Hesse, Germany) within the emergenCITY center and the FlowForLife project, and the Hessian Ministry of Science and the Arts (HMWK) within the projects "The Third Wave of Artificial Intelligence 3AI" and hessian.AL. The authors acknowledge the Lichtenberg high performance computing cluster of the TU Darmstadt for providing computational facilities for the calculations of this research.
REFERENCES
Juan A. Acebrón, L. Bonilla, Conrad J. Pérez, Félix Ritort, and Renato Spigler. The Kuramoto model: A simple paradigm for synchronization phenomena. Rev. Mod. Phys., 77(1):337-185, 2005.
Alexandre Araujo, Aaron Havens, Blaise Delattre, Alexandre Allauzen, and Bin Hu. A unified algebraic perspective on Lipschitz neural networks. In Proc. ICLR, pp. 1-15,2023 2023.
Karl Johan Åström. Optimal control of Markov processes with incomplete state information. Math. Anal. Appl., 10(1):174-205, 1965.
Erkin Bahgeçi and Erol Sahin. Evolving aggregation behaviors for swarm robotic systems: A systematic case study. IEEE Swarm Intell. Symp., pp. 333-340, 2005.
Lucas Barberis. Emergence of a single cluster in Viesek's model at very low noise. Phys. Rev. E, 98 (3), 2017.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of Markov decision processes. Math. Oper. Res., 27(4):819 04488999,00, 2002.
Patrick Billingsley. Convergence of probability measures. John Wiley Sons, 2013.
Michael Breakspear, Stewart Heitmann, and Andreas Daffertshofer. Generative models cortical oscillations: neurobiological implications of the Kuramoto model. Front. Hum. Neurosci,, 4:190, 2010.
René Carmona. Applications of mean field games in financial engineering and economic theory. arXiv:2012.05237. 2020.
René Carmona, François Delarue, and Daniel Lacker. Mean field games with common noise. The Annals of Probability, 446))3374-33833, 2016.
René Carmona, Mathicu Laurière, and Zongjun Tan. Linear- mean-field reinforcement learning: convergence of policy gradient methods. 1rXhv:1910.04295, 2019a.
René Carmona, Mathieu Laurière, and Zongjun Tan. Model-free mean-field reinforcement learning: mean-field MDP and mean-field Q-learning. arXiv:1910. /2802, 2019b.
René Carmona, Quentin Cormier, and H Mete Soner. Synchronization in Kuramoto mean field game. arXiv:2210. /2912, 2022.
Louis-Pierre Chaintron and Antoine Diez. Propagation of chaos: A review of models, methods and applications. models and methods. Kinet. Relat. Models, 1956) 2022.
Frank Cichos, Kristian Gustavsson, Bernhard Mehlig, and Giovanni Volpe. Machine learning for active matter. Nat. Mach. Intell., 2(2):94-033, 2020.
Ştefan Cobzaş, Radu Miculescu, and Adriana Nicolac. Lipschitz,functions Springer, 2019.
Kai Cui, Christian Fabian, and Heinz Koeppl. Multi-agent reinforcement learning via mean field control: Common noise, major agents and approximation properties. arXiv:2303. /0665, 2023.
Gokce Dayanikli, Mathieu Laurière, and Jiacheng Zhang. Deep learning for population-deepenent controls in mean field control problems. 206.78,300 2023.
10

---

Published as a conference paper at ICLR 2024
Ronald A Vore and George G Lorentz. Constructive approximation, volume 303. Springer Science & Business Media, 1993.
Albert Diaz-Guilera, Jesús Gómez-Gardenes, Yamir Moreno, and Maziar Nekovee. Synchronization in random geometric graphs. Int. J. Bifurcation Chaos, 19(02)x887-633, 2009.
Boualem Djehiche, Alain Tcheukam, and Hamidou Tembine. Mean-field-type games in engineering. AIMS Electronics and Electrical Engineering, 1(1):18-73, 2017.
Zuyue Fu, Zhuoran Yang, Yongxin Chen, and Zhaoran Wang. Actor-critic provably finds nash equilibria of linear-quadratic mean-field field games. Proc. ICLR, pp. 1-82, 2019.
Alison L Gibbs and Francis Edward Su. On choosing and bounding probability metrics. Int. Stat. Rev., 70(3):419 435, 2002.
Haotian Gu, Xin Guo, Xiaoli Wei, and Renyuan Xu. Dynamic programming principles for mean-field controls with learning. arXiv:191..07311 2019.
Haotian Gu, Xin Guo, Xiaoli Wei, and Renyuan Xu. Mean-field controls with Q-learning for cooperative MARL: convergence and complexity analysis. STAM J. Math. Data Sci., 3(4):1168 1196, 2021.
Xin Guo, Anran Hu, Matteo Santamaria, Mahan Tajrobehkar, and Junzi Zhang. MFGLib: A library for mean-field games. arXiv:2304.086300 2023.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv:1609.09100 2016.
Heiko Hamann. Swarm Robotics: A Formal Approach. Springer, 2018.
Ammar Haydari and Yasin Yilmaz. Deep reinforcement learning for intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(1): 2020.
Onésimo Bernández-Lerma and Jean B Lasserre. Discrete-time Markov control processes: basic optimality criteria, volume 30, Springer Science & Business Media, 2012.
Onésimo Hernández-Leerme and Myriam Muñoz de Ozak. Discrete-time Markov control processes with discounted unbounded costs: optimality criteria. Kybernetika, 28(3):191-21,, 1992.
Christopher-Fyy Hrabia, Marco Lützenberger, and Sahin Albayrak. Towards adaptive multi-robot systems: Self-organization and self-adaptation. Knowl. Eng. Rev., 33:e16, 2018
Siyi Hu, Yifan Zhong, Minquan Gao, Weixun Wang, Hao Dong, Xiaodan Liang, Zhihui Li, Xi- aojun Chang, and Yaodong Yang. MARLlib: A scalable and efficient library for multi-agent reinforcement learning. JMLR, 24:1-23, 2023.
Jiawei Huang. Batuhan Yardim, and Niao He. On the statistical efficiency of mean field reinforcement learning with general function approximation. arXiv:2305. /283, 2023.
Minyi Huang, Peter E Caines, Roland P Malhamé, al. Distributed multi-agent decision-making with partial observations: asymptotic Nash equilibria. In Proc. 17th Internat. Symp. MTNS, pp. 2725-2730 2006.
Olav Kallenberg, Foundations of Modern Probability. Springer, 2021.
Nikita Kruk, José Carrillo, and Heinz Koeppl. Traveling bands, clouds, and vortices of chiral active matter. Phys. Rev. E, 102(():022604, 2020.
Minae Kwon, John Agapiou, Edgar luéñez-Guzzál Romuald Elie, Georgios Piliouras, Kalesha Bullard, and Ian Gemp. Auto-aligning multiagent incentives with global objectives. In ALA Workshop, AAMAS, pp. 1-9, 2023.
Pengdeng Li, Xinrun Wang, Shuxin Li, Hau Chan, and Bo An. Population-sszz-awaar policy optimization for field games. In Proc. ICLR, pp. 1-32, 2023.
11

---

Published as a conference paper at ICLR 2024
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In Proc. ICML, pp. 3053-3062, 2018.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proc. ICLR, pp. 1-14, 2016.
Aditya Mahajan, Nuno C Martins, Michael Rotkowitz, and Serdar Yüksel. Information structures in optimal decentralized control. In Proc. IEEE CDC, pp. 1291-1306, 2012.
Radu Miculescu. Approximation of continuous functions by Lipschitz functions. Real Analysis Exchange, pp. 449-452, 2000.
Washim Uddin Mondal, Mridul Agarwal, Vaneet Aggarwal, and Satish V Ukkusuri. On the approx- imation of cooperative heterogeneous multi-agent reinforcement learning (MARL) using mean field control (MFC). J. Mach. Learn. Res., 23(129):1-46, 2022.
Narinder Narinder, Clemens Bechinger, and Juan Ruben Gomez-Solano. Memory-induced transition from a persistent random walk to circular motion for achiral microswimmers. Phys. Rev. Lett., 121 (7):078003, 2018.
Mahdi Nasiri and Benno Liebchen. Reinforcement learning of optimal active particle navigation. New Journal of Physics, 24(7):073042, 2022.
Frans A Oliehoek and Christopher Amato. A Concise Introduction Decentralized POMDPs. Springer, 2016.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv:2203.02/55 2022.
Christos H Papadimitriou and John N Tsitsiklis. The complexity of Markov decision processes. Math. Oper. Res., 12(3):441-450, 1987.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. Proc. NeurlPS Track Datasets Benchmarks, 2021.
Kalyanapuram Rangachari Parthasarathy. Probability measures on metric spaces, volume 352.
American Mathematical Soc., 2005.
Barna Pasztor, Ilija Bogunovic. and Andreas Krause. Efficient model-based multi-agent mean-field reinforcement learning. 107i::20770045000 2021.
Sarah Perrin, Julien Pérolat, Mathieu Laurière, Matthieu Geist, Romuald Elic, and Olivier Pietquin. Fictitious play for mean field games: Continuous time analysis and applications. In Proc. NeurIPS. volume 33, pp. 13213, 2020.
Sarah Perrin, Mathieu Laurière, Julien Pérolat, Matthieu Geist, Romuald Élie,, and Olivier Pietquin. Mean field games flock! The reinforcement learning way. In Proc. IJCAI, pp. 356-362, 2021.
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputing, 71(7-9): 11011890,,,,1111111 2008.
Huyên Pham and Xiaoli Wei. Bellman equation and viscosity solutions for mean-field stochastic control problem. ESAIM Contr. Optim. Calc. Var., 24(1):437-461, 2018.
Athanasios S Polydoros and Lazaros Nalpantidis. Survey of model-based reinforcement learning: Applications on robotics. J. Intell. Robot. Syst., 86)):153-13333 2017.
Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning of localized policies for multi-agent networked systems. In Proc. Learn. Dyn. Contr. pp. 256-266, 2020.
Walter Rudin. Principles of Mathematical Analysis, volume 3. McGraw-hill New York, 1976.
12

---

Published as a conference paper at ICLR 2024
Naci Saldi, Tamer Başar, and Maxim Raginsky. Partially-observed discrete- time risk-sensitive mean-field games. In Proc. IEEE CDC, pp. 317-322, 2019.
Melanie Schranz, Gianni A Di Caro, Thomas Schmickl, Wilfried Elmenreich, Farshad Arvin, Ahmet Şekercioğlu, and Micha Sende. Swarm intelligence and cyber-physical systems: Concepts, challenges and future trends. Swarm Evol. Compul., 60:10006:2 2021.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proc. ICLR, pp. 1-14, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXivv1707.0663 2017.
Nevroz Şen and Peter E Caines. Mean field games with partial observation. SIAM J. Contr. Optim., 2703):2064-0091, 2019.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proc. ICML, pp. 387-395, 2014.
Onur Soysal and Erol Sahin. Probabilistic aggregation strategies in swarm robotic systems. In IEEE Swarm Intell. Symp., pp. 325-332, 2005.
Sriram Ganapathi Subramanian, Matthew E Taylor, Mark Crowley, and Pascal Poupart. Partially observable mean field reinforcement learning. In Proc. AAMAS, volume 20, pp. 537-545, 2021.
Sriram Ganapathi Subramanian, Matthew Taylor, Mark Crowley, and Pascal Poupart. Decentralized mean field games. In Proc. AAAI, volume 36, pp. 9439-9447 2022.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Proc. NIPS, pp. 1057-1063, 1999.
Takehiro Tottori and Tetsuya J Kobayashi. Memory-limited partially observable stochastic control and its mean-field control approach. Entropy, 24(11):1599, 2022.
Tamás Vicsek and Anna Zafeiris. Collective motion. Physics reports, 517(3-4):71-140, 2012.
Tamás Vicsek, András Czirók, Eshel Ben-Jacob, Inon Cohen, and Ofer Shochet. Novel type of phase transition in a system of self-driven particles. Phys. Rev. Lett., 75(6) 1995.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Daniel Waelchli, Pascal Weber, and Petros Koumoutsakos. Discovering individual rewards in collective behavior through inverse multi-agent reinforcement learning. arXiv:2305. 10548, 2023.
Weichen Wang, Jiequn Han, Zhuoran Yang, and Zhaoran Wang. Global convergence of policy gradient for linear-quadratic mean-field control/game in continuous time. In Proc. ICML, pp. 10772-10782. PMLR, 2021.
Stefan Witwicki and Edmund Durfee. Influence-baeed policy abstraction for weakly-coupled dec- pomdps. In Proc. ICAPS, volume 20, pp. 185-192, 2010.
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In Proc. ICML, pp. 5571-5580, 2018.
Peng Yin, Harry MT Choi, Colby R Calvert, and Niles A Pierce. Programming biomolccular self-assembly pathways. Nature, 451(7176):318-33 322, 2008
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Proc. NeurIPS Datasets and Benchmarks, 2022.
Jorge L Zapotecatl, Angélica Munoz-Meléndee Meléndez, and Carlos Gershenson. Performance metrics of collective coordinated motion in flocks. In Proc. ALIFE XV, pp. 322-329. MIT Press, 2016.
13

---

Published as a conference paper at ICLR 2024
Quanbo Zha, Gang Kou, Hengjie Zhang, Haiming Liang, Xia Chen, Cong- Li, and Yucheng Dong. Opinion dynamics in finance and business: a literature review and research opportunities. Financial Innovation, 6:1-22, 2020.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Decentralized multi-agent reinforcement learning with networked agents: Recent advances. Frontiers of Information Technology & Electronic Engineering. 22(6):802-81) 814, 2021a.
Kaiqing Zhang, Zhuoran Yang. and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. In Kyriakos G. Vamvoudakiss Yan Wan, Frank L. Lewis, and Derya Cansever (eds.), Handbook of Reinforcement Leurning and Control, pp. 321-384. Springer International Publishing, Cham, 2021b.
Lianmin Zheng, Jiacheng Yang, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. Magent: many-agent reinforcement learning platform for artificial collective intelligence. In Proc. AAAI, pp. 8222-8223, 2018.
14

---

Published as a conference paper at ICLR 2024
A ADDITIONAL EXPERIMENTS
In this section, we give additional details on experiments. The mathematical description of problems can be found in Appendix C.
We use the manifolds as depicted in Figure 9 and as described in the following. Here, we visualize the qualitative results as the main text for the remaining topologies. Due technical limitations, all agents are drawn, including the ones behind surface. To indicate where an agent belongs, we colorize the inside of the agent with the color of corresponding surface.
2
0.25
0.00
0
0.25
2
2.5
1
2.5
0y
0.00
30
00
0
2.5
0,0
2.5
2.5
1
9
-2
2.
2.5
0
2.5
Figure 9: Two-dimensional manifolds visualized three-dimensional space. order from left to right: Möbius strip, torus, projective plane (Boy's surface), and Klein bottle (pinched torus).
Torus manifold. The (flat) torus manifold is obtained from the square [-1, by identifying (z, -1) 1x and (-1,y) ~ (1,y) for all x, y [-1, For the metric, we use the toroidal distance inherited from the Euclidean distance d2, which be computed
min d(r,y) 61.20(11001)
d2 y 2 + 2tye2)
where e1, e2 denote unit vectors. Figure 9, we visualize the torus by mapping each point (æ; y) [-1, 1] to point (X, Y, 2) in 3D space, given by
X (2 0.75 cos (π(x + 1))) cos (T(y 1)),
Y (2 0.75 cos (r + 1))) sin (=(y 1)),
Z 0.75 sin (ι(x 1)).
The results have been described in the main text in Figure 7. Here, also note that the torus by periodicity and periodic boundary conditions can essentially be understood as case of an infinite plane, consisting of infinitely many copies of the square [-1, 1) laid next to each other.
Möbius strip. The Möbius strip is obtained from the square [-1, 1] 2 by instead only identifying (-1.-W) (1. 3) for y 1], i., i.e. only the top and bottom side of the square, where directions are flipped. We then use the inherited distance
d(x, v)
min dz(a,, 2- 1,200, 1)TH O Oy+ 26262)
12,0(1,0,1)
where 3 denotes the elementwise (Hadamard) product.
We visualize the Möbius strip in Figure 9 by mapping each point (x, v) E [-1, 1) to
G (y + 1))) cos (T + 1)),
X +
Y 1 cos
( sin (т( 1))
Z sin
T
( +1)+
2
As we can see in Figure 10, the behavior of agents is learned as expected: Agents learn to align along one direction the Möbius strip.
15

---

Published as a conference paper at ICLR 2024
t= 2
t t= 10
30
Figure 10: Qualitative visualization Viesek behavior on the Möbius strip manifold for uniform initialization. The 300 agents (triangles) align into one direction the Möbius strip.
Projective plane. Analogously, the projective plane is obtained by identifying and flipping both sides of the square [-,, i.e. (-I, -1) ~ (ar, 1) and (-1, ~ (1,9) for all v,y 11,, We
use the inherited distance
min dxx v) (0,11)
d2(2, (1 21--- 2- 1,20, 2 10) o y + 21 + 21222)
and though accurate visualization in less than four dimensions is diffieult, visualize in Figure 11 by mapping each point (x. y) E 1| to a point (X, Y, Z) on the so-called Boy's surface, with
+
exp (im (y + 1))
3
2(1 24)
Im
2
2 + V523
3 Re
x(1 - 24)
92 =
2
6 + V523
11
1+ 6
Im
26 + V523
2'
91
X
9
2
+ 92
+ 03 7
92
Y
91 + 9 + 93
2
2
2'
92
Z
9 + 02 + 93
As we can see in Figure 11, under the inherited metric and radial parametrization, agents tend to gather at the bottom of the surface.
t=2
t= 10
t t= 30
Figure 11: Qualitative visualization of Viesek behavior on the projective plane manifold for uniform initialization. The 300 agents (triangles) align over time by gathering at the bottom right.
Klein bottle. Similarly, the Klein bottle is obtained by identifying both sides of the square [-1, and flipping one side, i.e. (x. -1) ~ (xr, 1) and (-1, -w) (1,y) for all x, y [-1, 1]. We the
16

---

Published as a conference paper at ICLR 2024
inherited distance
d(r, v)
min 6.66((110011
d2(2, (1 - 2. 1 1970, 137
1)" oy y 2/161 + 2/202)
and visualize in Figure 12 by the pinched torus, i.c. mapping each (z, E [-1,]] to point (X, Y, with
X (2 + 0.75 cos (π + 1))) cos (T 1)),
Y = (2 + 0.75 cos (अ(x + 1))) sin (5 1)),
= 0.75 sin (α( + 1))) cos 2 + 1))
As we can see in Figure 12, agents may align by aggregating on the inner and outer ring, such that they may avoid switching sides at the pinch.
t=2
t=
30
Q
Figure 12: Qualitative visualization of behavior Klein bottle topology for uniform initialization: The visualization in 3D is limited. Here, we use a pinched torus visualization that inverts itself at the flat pinch (i.e. there is no "connection" between blue and red surfaces the bottom). Over time, 300 agents (triangles) sometimes align by aggregating on the inner, avoiding side switches at the pinch.
Box. Lastly, the box manifold is the square [-1, 1| equipped with the standard Euclidean topology, i.c. distances between two points x, y E 1|2 are given by
d(x, v)
(x) ツı (x2-
while the sides of the square are not connected to anything else. We use the box manifold for the following experiments Aggregation, and mention it here for sake of completeness.
Ablation number of agents. As seen in Figures 13 and 14, we can successfully train on various numbers of agents, despite the inaccuracy of the mean field approximation for fewer agents as inferred from Figure 5. This indicates that our algorithm is general and at least in the considered problems scales to arbitrary numbers of agents.
Return (())
-150
100
100
100
200
(a)
200
(b)
200
(c)
200
(d)
-250
0
2
2
Steps t tx10
Steps x1
Steps t xx10
Steps 1x
Figure 13: Training curves for Vicsek (torus), using RBF-based or discretization-bass solutions. (a): RBF, N 25; (b): Discretization. N 25; (c): RBF, N 50; (d) Discretization, N 50.
Qualitative results for Kuramoto. The Kuramoto model, see Figure 15, demonstrates instability during training and subsequent lower-grade qualitative behavior compared to the Vicsek model. This disparity persists even when considering more intricate topologies, despite being a specialization of the Viesek model. One explanation is that the added movement makes it easier to align agents over time. Another general explanation could be that, despite initially distributing agents uniformly across the region of interest, the learned policy causes the agents to aggregate into few even
17

---

Published as a conference paper at ICLR 2024
Return J(x)
-100
100
100
100
200
(a)
200
(b)
200
(c)
2001
0
2
2
2
Steps tx10
Steps xx10 10
Steps 1 xx10
Steps /x
Figure 14: Training curves for Vicsek (torus), using RBF-based or discretization- solutions. (a):
RBF, N 100: (b): Discretization, N 100; (c): RBF, N 150; (d) Discretization, N 150.
single cluster (though we do not observe such behavior in Figure 15). The closer particles are to each other, the greater the likelihood that they perceive similar identical mean field, prompting aligmment only in local clusters. similar behavior is observed in the classical Viesek model, where agents tend move in the same direction after interaction. Consequently, they remain within each other's interaction region and have the potential to form compounds provided there are no major disturbances. These can come from either other particles or excessively high levels of noise (Barberis, 2017). Although agents are able to align, the desired alignment remains to be improved, either via more parameter tuning or improved algorithms.
t=0
150
2T
2T
0
20
0.146
0.916
π
0
Figure 15:
Qualitative behavior of the learned behavior in the Kuramoto model with histogram over angles, where in contrast to Vicsek, agents remain fixed in their initial position.
Effect of kernel method. While for low dimensions, the effect of kernel methods is not pro- nounced and mostly ensure theoretical guarantees, in Figures 16 and 17 we can see that training via RBF-based methods outperforms discretization-ban compared to a simple gridding of the probability simplex with associated histogram representation of the mean field. Here, the RBF method in Aggregation, we place equidistant points yb on the axis of each dimension. This is also the reason for why the discretization- based approach is better for low dimensions d 2 or d 3, as more points will have more control over actions of agents, and can therefore achieve better results, in exchange for tractability in high dimensions. This shows the ad- vantage RBF-based methods more complex, ding--iimessional problems. While the RBF-based method continues to learn even for higher dimensions up eventually stops learning due to very large action spaces leading to increased noise on the gradient. The advantage is not just in terms of sample complexity. but also in terms of wall clock time, the computation over exponentially many bins also takes more CPU time as shown in Table 1.
methods for dimensions higher than 3 as
5, the discretization-bant
solution
Table 1: Wall clock time for one training step averaged over 500 iterations in d-dimensional Aggrega- tion, for 50 agents.
Dimensionality
RBF MFC [s]
Discretization MFC [s]
MARI (IPPO) [s]
2
5.64
5.69
146.58
3
6.16
7.96
147.03
4
6.97
17.26
147.29
5
8.31
76.33
146.91
18

---

Published as a conference paper at ICLR 2024
Return ((r)
100
-75
75
100
100
(a)
100
(b)
-125
(c)
125H
(d)
0
Steps x x10
Steps x x10$
Steps x10
Steps x10
Figure 16: Training curves for d-dimensional Aggregation, using RBF-based vs. discretization- based solutions. (a): RBF, d = 2; (b): Discretization, d = 2; (c): RBF, = 3; (d) Discretization, d = 3.
Return J(x)
-125
155
167
140
168
(a)
(b)
165
(c)
(d)
-150-
150
0
2.5 Steps x100
Steps x x10
Steps x x10
Steps x10
Figure 17: Training curves for d-dimensional Aggregation, using RBF-based vs. discretization-basee solutions. (a): RBF, 4; (b): Discretization, d = 4; (c): RBF, 5; (d) Discretization, d == 5.
Ablations time dependency and starting conditions. As discussed in the main text, also verify the effect of using a non-time-dependent open-loop sequence of lower-level policies, and also ablation different starting conditions. In particular, for starting conditions, to begin will consider the uniform initialization as well as the beta-1, beta-2 and beta-3 initializations with beta distribution over each dimension of the states, using a respectively.
0.5, 8 0.25 and a 0.75
in Figure 18, the behavior learned for the Vicsek problem the torus with N 200 agents allows for using the first lower-level policy to at all times t under the Gaussian initialization used in training to nonetheless achieve aligmment. This validates the fact that a time-variam open- sequence of lower-level policies is not always needed, and the results even hold for slightly different initial conditions from the ones used training.
As
25
150
25
150
0.048
0.907
t 25
= 150
2x
25
I= 150
2n
T
0.259
5955
0.138
0 800
Ri 0.998
0
1
0
0
0
20
20
Figure 18: Open-loop behavior by using the lower-level decision policy at time = 0 for all times, on Vicsek (torus) with N 200 agents and various initializations. A: uniform initialization; B: beta-2 initialization; C: beta-1 initialization; D: beta-3 initialization.
Analogously, consider peak-normal initialization is given by more concentrated zero-centeres diagonal Gaussian with covariance 2 0.1. The squeezed-normal is the same initialization as in training, except for
more strongly concentrated and heterogeneous initializations: The
19

---

Published as a conference paper at ICLR 2024
dividing the variance in the y-axis by 10. The multiheaded-normee initialization is mixture of two equisized Gaussian distributions in the upper-right and lower-left quadrant, where in comparison to the training initialization, position variances are halved. Finally, the crmuulll-muliihhadd----nnoe additionally changes the weights of two Gaussians to 0.75 and 0.25 respectively.
As seen in Figure 19, the lower-level policy No for Gaussian initialization from training easily transfers and generalizes to more complex initializations. However, the behavior may naturally be more suboptimal due the training process likely never seeing more strongly concentrated and heterogeneous distributions of agents. For example, in the peak-normal initialization in Figure 19, we see that the agents begin relatively aligned, but will first misalign in order to align again, as the learned policy was trained to handle only the wider Gaussian initialization.
A t= 0
25
= 150
2x
B
25
t= 150
2x
Ø
=
0.943
15
0.487
0.994
= 0.151
0.967
0 1.9
0
0
1
0
0
0
150
2m
t=
t 25
t= 150
2m
R 0.028
Bi
R 0.997
0
0
20
Figure 19: Open-loop behavior by using the lower-level decision policy at time t 0 for all times, on Vicsek (torus) with N = 200 agents and various initializations. A: peak-normal initialization; B: squeezed- normal initialization; C: multhheaded-nome initialization; D: normal initialization.
No observations. As additional verification of the positive effect of mean field guidance PG training, we also perform experiments for training PPO without any RI. observations, as in the previous paragraph we verified the applicability of learned behavior even without observing the MF that is observed by RL during training. In Figure 20 we see that PPO is unable to learn useful behavior, despite the existence of such time-invariant lower-level policy from the preceding paragraph, underlining the empirical importance of mean field guidance that we derived.
0
= 50
150
2mT
0 044
0
474
474
Figure 20: Qualitative behavior after training without observations for Vicsek (torus).
Transfer to differing agent counts. In Figure 21, we see qualitatively that the behavior learned for N 200 agents transfers to different, lower numbers of agents well. The result is congruent with the results shown in the main text, such as Figure 5, and further supports the fact that our method scales to nearly arbitrary numbers of agents.
Forward velocity control. We also allow agents alternatively control their maximum velocity in the range [0, vo]. Forward velocity can similarly be controlled, and allows for more uniform spreading of agents in contrast to the case where velocity cannot be controlled. This shows some additional
20

---

Published as a conference paper at ICLR 2024
0
25
== 150
= 25
t= 150
0.509
0.889
0.821
0.589
0.962
0.975
0
0
0
t
t = 25
t= 150
2m
25
150
2m
- 0.519
[1 0.969
R 0.993
0.518
= 0.975
H 0.9966
0
0
x
Figure 21: Qualitative behavior of the policy learned for N 200 Vicsek (torus), transferred to different numbers of agents N. A: N 25; B: N 50; C: N = 100; D: = 150.
generalization of our algorithm variants of collective behavior problems. The corresponding final qualitative behavior is depicted in Figure 22.
A
2m
25
2m
R
0:487
0.007
= 25
150
2x
1= 150
2T
Ø
T
[R 0.016
0.3
0.338
R 0.974
0
1
20
Figure 22: Qualitative behavior on Vicsek (torus) with = 200 agents, additional forward velocity control, and various initializations. A: peak-normal initialization; B: squeezed-normal initialization; C: multiheaded- normal initialization; D:
initialization.
Comparison of IPPO and MAPPO for low numbers of agents. Lastly, for completeness show the comparison of IPPO and MAPPO training results for various numbers agents in Figures 23 and 24. The overall achieved performances overall comparable to the results of the Dec-POMFPPO method in Figure
Return Jim)
50
N-5
-75
N=15
200
00
N-50
-100
[8]
(b)
N-200
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.32
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Steps
K1P
Steps
x11"
Steps
x10"
MF
Figure 23: IPPO training curves (episode return) with shaded standard deviation over 3 seeds and various N, in (a) Aggregation (box), (b) Vicsek (torus), (c) Kuramoto (torus). For comparison, we also plot the best return averaged over 3 seeds for Dec-POMFPPO in Figure 3 (MF).
21

---

Published as a conference paper at ICLR 2024
Return J(r)
N=5
20
-75
-00
N-15
2.10
N=50
-100
(a)
(b)
-125
[c]
N=200
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0,30
0.75
1,00
0.00
0.25
0.50
0.75
1,50
Steps
Steps
X10
Steps
110
MF
Figure 24; MAPPO training curves (episode return) with shaded standard deviation over seeds and various N, in (a) Aggregation (box), (b) Viesek (torus), (c) Kuramoto (torus). For comparison, we also plot the best return averaged over 3 seeds for Dec-POMFPPO in Figure 3 (MF).
B EXPERIMENTAL DETAILS
We use the RLlib 2.0.1 (Apache-2.0) license) (Liang et al., 2018) implementation of PPO (Schulman al., 2017) for both MARL via IPPO, and Dec-POMFPPO. For MAPPO, used the MARLlib 1.0.0 framework (Hu et al., 2023), which builds upon RLlib. For our experiments, we used no GPUs and around 60 000 Intel Xeon Platinum 9242 CPU core hours, and each training usually took at most three days by training on up to 96 CPU cores. Implementation-wicc for the upper-level policy NNs learned by PPO, we use two hidden layers with 256 nodes and tanh activations, parametrizing diagonal Gaussians over the MDP actions 5 E =\t (parametrizations of lower-level policies).
In Aggregation, we define the parameters ę E E for continuous spaces X, X, M C Rd by values in E == -1,1 Each component of & is then mapped affinely to mean in l or diagonal covariance in le, 0.5 + e| with = 10 10 /4, of each dimension. Meanwhile, in Viesek and Kuramoto, we pursue "discrete action space" approach, letting E = 1}3. We then affinely map components of E = [c, 0.5 c). which are normalized to constitute probabilities of actions in (-1,0,1)
For the kernel-based representation of mean fields in d-dimensional state spaces X, we define points Ia by the center points of d-dimensional gridding of spaces via equisized (Mx 3 hypercubes) partitions. For the histogram, we similarly use the equisized hypercube partitions. For observation spaces y and the kernel-based representation the nigh-dimensional experiments below, where
lower-level policies, unless noted otherwise (e.g. in less than exponentially many points), do the same but additionally rescale the center points l/b around zero, giving y/ = cýo for some 0.75 for Aggregation and
0 and My 5d. We For the (diagonal) bandwidths RBF kernels, in Aggregation we use a = 0.12/ /Mx for states and a = 0.12c for observations. In Vicsek and Kuramoto, we use a = 0.12/v / for state positions, 0.12 for state angles, and a 0.06c a 0.12 for the first second component of observations respectively. For IPPO and MAPPO, we observe the observations y: directly. For hyperparameterss of PPO, see Table
0.1 for Vicsek and Kuramoto.
Table 2: PPO hyperparameter values.
Hyperparameter
Value
Discount factor
0.99
GAF lambda
1
KI. coefficient
0.03
Clip parameter
0.2
Learning rate
0.00005
Training batch size Bien
4000
Mini-bated size bien
1000
Steps per batch Nppo
5
C
PROBLEM DETAILS
In this section, we will discuss in more detail the problems considered in our experiments.
Aggregation. The Aggregation problem is a problem where agents must aggregate into a single point. Here, X = = [-1, C Rdd for some dimensionality parameter d E N, and analogously
22

---

Published as a conference paper at ICLR 2024
4 -=1-1,11 for per-dimension movement actions. Observations are the own, noisily observed position, and movements are similarly noisy, using Gaussian noise. Overall, the dynamics are given by
yo N (Iz, diag .......
J++1 N
+ 00
U
diag(o go....
o
max [|u)]l))
for some velocity vo, where additionally, observations and states that are outside of the box [-1, 1] are projected back to the box.
The reward function for aggregation of agents is defined as
r(p)) cca /ʃ |a (dl
// max 1 [12l22)
xe(du (dx).
for some disaggregation cost ca > 0 and action cost Cu O. where we allow the dependence of rewards on actions as well: Note that our framework still applies to the above dependence on actions, as discussed in Section 2, by rewriting the system in the following way. At any even time 2t, the agents transition from state x E 2' to state-actions (at, u) E X' x 24, which will constitute the states of the new system. At the following odd times 21 + 1, the transition is sampled for the given state-actions. In this way, the mean field is over X' J (X x 2) and allows description of dependencies on the state-action distributions instead of only the state distribution.
For the experiments, we use o = 0.04, o 0.04 and vo - 0.1. The initial distribution of agent positions is Gaussian centered around zero, with variance 0.4. The cost coefficients are Cd 1 and Cu = 0.1. For simulation purposes, we consider episodes with length T 100.
Vicsek. In classical Vicsek models, each agent coupled every other agent within predefiedd interaction region. The agents have a fixed maximum velocity Un > 0, and attempt to align themselves with the neighboring particles within their interaction range > The equations governing the dynamics of the i-th agent in the classical Vicsek model are given in continuous time by
dp' = (vo sin(""), by cos(o"))
dø =
1
sin (6' - -²) dt + odW Σ JEN;
N
for all agents i, where N; denotes the set of agents within the interaction region, N₂ {j E [M] d(x;, x²) < D), and W denotes Brownian motion.
We consider a discrete-ting time variant where agents may control independently how to adjust their angles in order achieve global objective (e.g. alignment, misalignment, aggregation). For states x = (pr, ).), actions un and observations yr, we have
xe = (/f sin(o Φe dd)
/f (00) 1d). m) de)
T
yt = (z, g)² 112, atan2 (ū, 3))²
P++1 = (pr.1 + sin (6:). Pt. + cos())
0:+1 (d, + woree o3)
for some maximum angular velocity wp > and noise covariance of > O, where atan2(x; y) is the
angle from the positive z-axis to the vector y)"" Therefore, we have X [-1, 1|2 x [0, 2m), where positions are equipped with the corresponding topologies discussed in Appendix A, and standard Euclidean spaces y i-1, 1]2 and U = [-1, 1]. Importantly, agents only observe the relative headings of other agents within the interaction region. As result, is impossible to model such a system using standard MFC techniques.
As cost functions, we consider rewards via the polarization, plus action cost as in Aggregation. Defining polarization similarly to e.g. (Zapotecatl al., 2016).
pol, = //
2. 2z, dø),
23

---

Published as a conference paper at ICLR 2024
((x,)) y) = arccos (ccos()), sin(o)) y/
)
5t // (cos(())) sin())) pe (dp, dé)
where high values of pol, indicate misalignment, we define the rewards for alignment
r(p) c.pol, Cu // | |ulsza(duu x)),(dx),
and analogously for misalignment
r(w) = +c,pol, - Cu // |u/mz((uu ( ()))(dr).
For our training, unless noted otherwise, we let D = 0.25, 0o = 0.075, wp 0.2, 0e = 0.02 and μoo as zero-centered (clipped) diagonal Gaussian with variance 0.4. The cost coefficients are Ca and C₂ = 0.1. For simulation purposes, we consider episodes with length T = 200.
Kuramoto. The Kuramoto model can be obtained from the Vicsek model by setting the maximal velocity Up of the above equations to zero. Hence, we obtain random geometric graph, where agents see only their neighbor's s state distribution within the interaction region, and the neighbors are static per episode. For parameters, we let D 0.25, to wi 0.2, d 0 and (Ao zero-centered (clipped) Gaussian with variance 0.4. The cost coefficients are Ca = and ea 0.1. For simulation purposes, consider episodes with length T 200.
D PROPAGATION OF CHAOS
Proof of Theorem /. As in the main text, usually equip P(X) with the 1-Wasserstein distance. In the proof, however, it is useful to also consider the uniformly equivalent metric dy (s,, = Em 2 m [ fm d(p x²)l instead. Here, m2) is fixed sequence of continuous functions fm: X + [1, see e.g. (Parthasarathy, 2005, Theorem 6.6) for details.
First, let define the measure 4Tth on X' U. defined for any measurable set X B C X x U by G. CT (A x B) = SA Jy SB xi(du Pdyy x. (d))) For notational convenience we define the MF transition operator f such that
If (,, du) M4+1-
The ST =
(6)
Continuity of T follows immediately from Assumption la and (Cui al., 2023, Lemma 2.5) which we recall here for convenience.
Proposition 4 ((Cui et al., 2023), Lemma 2.5). Under Assumption la, (Mn, Gn) - (1,, () implies T(un, (n) + (p,
The rest of the proof is similar to (Cui et al., 2023, Theorem 2.7) - though we remark that we strengthen the convergence statement from weak convergence convergence in Lı uniformly over f E F by showing via induction over that
sup sup E [[f(n()) f(w)]] + 0.
=ell fElF SEF
Note that the induction start can be verified by weak LLN argument which is also leveraged in the subsequent induction step. For the induction step we assume that (7) holds at time t. At time + 1 have
sup sup E [[ - f(H++1)l rel
sup sup E [[ { (pt. .5 ST
(8)
well
24

---

Published as a conference paper at ICLR 2024
| (i (4t (x"" -f(ci)))
sup sup E xell FEF
(9)
We start by analyzing the first term and recall that modulus of continuity wi of defined as a func- tion wiF: [0, oo) + [0, oo) with both lim=-0 (x) = 0 and [f()) [(v)- S wF(W (H, VS E Lorentz. 1993, Lemma 6.1), such a non-concave and decreasing modulus wf
F. By (DeVore exists for FF because it is uniformly equicontinuous due to the compactness of ((t). Analogously, we have that F is uniformly equicontinuous in the space (P(X), dx) as well. Recalling that P(X) is compact and the topology of weak convergence is metrized by both dy and W1, know that the identity map id: (P(X), dr:) + (P(2), Wi) is uniformly continuous. Leveraging the above findings, have that for the identity map there exists modulus of continuity such that
If(x) - f(v)] S WF (W1 (id μ. idv)) S ( z)))
holds for all v,v (P(X), dz). By (De Vore & Lorentz, 1993, Lemma 6.1), we can use the least concave majorant of Wy = wy i instead of wy: itself. Then, (8) can be bounded by
=[(wwNt)) (i (ut ST.u"
WF (ds (Mtt2: T ( Ct
I E
I عوه E ds (utt1, T ,
ST.
irrespective of both and f by the concavity of wy and Jensen's inequality. For notational conve- nience, we define x²" (z) )oE[N)- and arrive
E ds (MttI, T (N." ( (("""))
Σ 2
E
// fi d (witz F (ni² ((Taa))
f fm (ui+1 -T
F (14 ((p (3 xx""
E m21
sup
E
Finally, we require the aforementioned weak LLN argument which goes follows
|ʃ fm d (uêt1 7 ( (o("c"""))
E
(Jm(1841) - T. (m((241)) xt11
N
N
iE[N]
12
<
E
(fm( [. (x2+2) [x
x/
N
iE[N]
E
(fm(((411) [fm (xt+1)))
x
N
N2
KE[N]
Here, have used that [fml S 1, well the conditional independence of w:41 given .:N. In combination with the above results, the term (8) thus converges to zero. Moving on to the remaining second term (9), we note that the induction assumption implies that
||5 ( (N. (u""s="")) <5"")) S(Meti)]
sup sup IE sett FEF
sup sup IE [[s (i (um",st"")) (T(e *enl JEF
< sup sup E [l9(u(") s(wi)]] + rell gEg
using the function == f o T which belongs to the class G of equicontinuous functions with modulus of continuity wg = wr wi. Here, wp is the uniform modulus of continuity over all policies ग of μ4 + T (H+) = T(p.. H. The equicontinuity of (I.T) Jren is a consequence of Lemma 4 as well as the equicontinuity of functions bt CT. which in turn follows from the
25

---

Published as a conference paper at ICLR 2024
uniform Lipschitzness of II. The validation of this claim is provided in the next lines. Note that this also completes the induction and thereby the proof. For a sequence of pn. + p E P(X) we can write
sup =E
Wi(St"
An St (
S sup sup =ell | sup sup
['( w)xx(du vP (,,),) - PM (dd at, 11) (nn(())))
VJf (1 () Pd x, ()), - (((ddd))))(())))
Hell II S1
Starting with the first term, we apply Assumptions la-lb arrive
(a, (du (P xx,p,) (d I. 11) (dx)
sup sup wlll
|| 11
I sup #ell If [
sup
(' ((aa,)) v) I. Mn) pd x, w)) un (dx)
/f f'(x, u) (v) x x, - p (dy از,
< sup sup netf || |ل =EX
sup
< Ln sup Wi (P (- [, ( x,,
S Ln W1 (P H4) +
with Lipschitz. constant Li corresponding to the Lipschitz function y + ʃ [1x u)m | y). Alternatielyy, if p" is assumed independent of the mean field in Assumption 1b, the term is zero.
f'(x.)) In a similar fashion, we point out the 1-Lipschitzness of x + ff In n. (du P" x, p).
Te(du P" | 22, /] Lu + 1
f'(x, Lu + Te(du P" ar m,
$ /// ((, u) (((,, u) (elu y) PM |2, 1u)
LnLpu +1
+Iʃ [( u)
IuLpp +1 reldul () 22,14 - (dy xr mi))
1
In
$ Ln +1 d(2,x)+ + LuLp +1 (P 2,11), PM (dy x.
LnLps
11)))))) d(z, d(x,y)
InLlpp +1
Inlep +1)
for A x. Alternatielyy, if the state space is assumed finite in Assumption 1b, the Lipschitzness follows directly.
This eventually yields the convergence of the second term, i.e.
f(, u)m Pd x. (,))(
u(dx))
sup sup well II $1
(Lpv Lu (ortm++1)p -1)
f' u)
LpvLn Te(du P" æ, -d)) - m(dx))
= sup sup Hell [[ 1i
S (T.pa In + 1)Wi (1) - 0
and thus completes the proof.
In the special case of finite states and actions, the approximation rate can also be quantified to (11/ by considering equi-Lipschitz families of functions F with constant Ly. Then, there is no need consider the two different metrizations dy and Wi, as they are Lipschitz equivalent, and one
26

---

Published as a conference paper at ICLR 2024
simply use the L1 distance. The convergence in the first term (8) is then directly via the weak LLN at rate O(1/VN) by
( - (T (4 (.T.")) )
sup sup IE =E fEF
S sup L E #ell
Σ
14- (x) - T (14 St
c: (z)
Leex
1
sup
LJ
Σ
E
E
Σ
1. 1x
E
12 1
J.N (+++1)
N
N
sell
N
N
zEx
4
Nº
while for the second term (9) the induction assumption, since T uniformly Lipschitz.
E AGENTS WITH MEMORY AND HISTORY-DEPENDENE
For agents with bounded memory, we note that such memory can be analyzed by our model by adding the memory state to the usual agent state, and manipulations on the memory either to the actions or transition dynamics.
For example, let 2f E Q = {0, 1) be the Q-bit memory of an agent at any time. Then, we may consider the new x Q-valued state (ri; zf), which remains compact, and the new U x Q-valued actions (w,, w(), where w is write action that can arbitrarily rewrite the memory, 2(41 = wf Theoretical properties are preserved by discreteness of added states and actions.
Analogously, extending transition dynamics to include observations y also allows for description of history-dependent policies. This approach extends to infinite-memory states, by adding observations y also to the transition dynamics, and considering histories for states and observations. Define the observation space of histories )" == y x O x UY', and the according state space X' == X x U (Y x 2)). The new mean fields μr Me are thus ((X')-valuee The new observation- dependent dynamics then defined by
P x. y. u, ()) P(- Ii, u, margi (a) ((2.y..)
where margı maps μ to first marginal, 21 is the first component of x, and x₂ the ()) -valued past history. Here, (X2, y, u) defines the new history of an agent, which is observed by
pd
p)) = P" æi, margi μ) @ 8xy.
Clearly, Lipschitz continuity is preserved. Further, we obtain the mean field transition operator
T' 11) = [ff p' 1. y, u, (e) dy, du).
using X' y" x U--aluee actions K4 14 (14 & [M] for some Lipschitz # [M]]: >" P(U). And in particular, the proof of e.g. Theorem extends to this new case. For example, the weak LLN argument still holds by
[dx (M T
[ fm (K - (ME b2))
< sup m>!
E
< sup E m21
Σ (Im ub. ut)
N
SE[M]
E [Im wo. u....., ... V.. ut) 2(1))
2/ 4
2/
0.
N
27

---

Published as a conference paper at ICLR 2024
for appropriate sequences of functions (fm)m21, fm: & x (2 x 2) - [-1, (Parthasarathy, 2005) and
f Im Imd dT' (ul .() = / fm( Yo, u0. ,y, us) Pd Wez Mss Wes 14g )
#[K][ddu PW))) Te, p4 )u (day, dyo, duo, , ddft- dut- 1).
Analogously, we can see that the above is part of a set of equicontinuous functions, and again allows application of the induction assumption, completing the extension.
F APPROXIMATE MFC-TYPE DEC-POMDP OPTIMALITY
Proof of Corollary /. The finite-agent discounted objective converges uniformly over policies to the
MFC objective
sup
JJ (T) [[TT] + N +
(10)
#Ill
since for any & let T' E 7 such that Σi 7t [r( ) S - max, 2/r(n))
and further let E 7 E [[ - r(w) by Theorem for sufficiently large N. Therefore, approximate optimality is obtained by
JdY (T) sup (T)) inf (J (J J (T')) r'ell
r'ell
2 inf N (T) - ((T)) + inf (J - ((z)) inf (J(m') J r'ell
e'lll
e'lll
-
by the optimality of π E arg max, JTr') and (10) for sufficiently large N.
G EQUIVALENCE OF DEC-POMFC AND DEC-MFC
Proof of Proposition I. We begin by showing the first statement. The proof is by showing ir = pt at all times E T, it then follows that J(ā) Eee ('r(fx) Et= =E )r((m) J(T). At time
t = 0, we have by definition joo uoo- Assume f4: = Pt at time I, then at time + 1, by (2) and (3),
have
Ae+ = //ʃ P(z,u, u. y. p.).) æ. (d)
(11)
[/ʃ P(z,, u, (d) 1 Pdy x, ())) p++1
(12)
which is the desired statement. An analogous proof for the second statement completes the proof. O
H OPTIMALITY OF DEC-MFC SOLUTIONS
Proof of Corollary 2. Assume (()(T)) < sup J(T'). Then there exists n' E 11 such that ( < J(T'). But by Proposition 1, there exists #' E such that J(x)) = J(x') and
hence J (+) = (+)()) < <J(')) = J(Ø'), which contradicts arg max J ( Therefore,
(TT) E arg / [(r').
I DYNAMIC PROGRAMMING PRINCIPLE
Proof of Theorem 2. We verify the assumptions in (Hernánezz-Lerma & Muñoz de Ozak, 1992). First, note the (weak) continuity of transition dynamics Í.
Proposition 5. Under Assumption la, T(un h.)) - (p, h) for any sequence (pn, hw)) + (H, of MFs pa, P (X) and joint distributions hn E H(Nx), H()).
28

---

Published as a conference paper at ICLR 2024
Proof. The convergence hn + h also implies the convergence of its marginal fy :n Jy h(, The proposition then follows immediately from Proposition
Furthermore, the reward continuous and hence bounded by Assumption la. is inf-compact by
(h E H(p) -r(p) s c} (H(u) if if-((p) S
else,
where H()) is closed by (Cui et al., 2023, 2023, Appendix A.2), and Lemma 2 if considering equi-Lipschitz policies Assumption lb.
Further, by compactness of P(X x y x U), H(u) is compact as a closed subset of compact set.
Lastly, lower semicontinuity of f -> H(j)) given, since for any pa >> μ and h = u P (1) o H(u)), we can find hn H(Mn):: Let h, pn @ ("(un) @ ē, then
W Wh(hnsh h)
sup SeLip(1)
f (z, u,) (Py pa) - puy I ())
[I] f(z,y, u) y) (Py I., - Pu(dy æ. 11)) pa (dx)
s sup sELip(1)
sup feLip(1) [I] ((r,y,)) Pdy a.,)) (un(di)) - (((dd))))((d)))
/ // f(x, y, u) I v) (P" a., un) ( pd Ir v4) pa (dxr
< sup SELip(1)
f(z, y, ud y) (dy x. x. (Mm(dx) - (((((d))))d)))) +
sup fetip(1)
since the integrands are Lipschitz by Assumption la and analyzed in the proof of Theorem 1.
The proof concludes by (Hernández-Lerma & Muñoz de Ozak, 1992, Theorem 4.2).
CONVERGENCE LEMMA
Lemma 1. Assume that (X,d) is a complete metric space and that (IR)NEN is sequence of elements of X. Then, the convergence condition of the sequence (In)nE,, i.e. that
3x E X Vs 0: 3N E N Vn 2 N d(æ, kn) <E
(13)
holds, is equivalent the statement
Ve > 0: 3x E X 3N EN: Vn N d(x, <..
(14)
Proof. (13) > (14): follows immediately.
(14) = (13): Choose some strictly monotonically decreasing, positive sequence of (E:) with lim;-soe & 0. Then, by statement (14) we can define corresponding sequences (æx)ie and (No)en such that
Vn 2 N₂ d(xi, < E:
(15)
Consider i' N and assume w.l.o.g. K. We know by the triangle inequality
Vn 2 max(N,, d(x,, myy S d(x,, d(z,: xT) S 28,-
(16)
Thus, the sequence (Ii)iEN is Cauchy and therefore converges to some x E X because (X, d) is complete metric space by assumption. Specifically, this is equivalent to
3a E Ve 0 31 E N Vi 2 I d(x, 2i) <.
(17)
Finally, statements (16), (17), and the triangle inequality yield
Fx. E X V2e > 0 NN E N: Vn 2 N : d(I, xn) < (() + d(ai,ta)) < 28
which implies the desired statement (13) and concludes the proof.
29

---

Published as a conference paper at ICLR 2024
K CLOSEDNESS OF JOINT MEASURES UNDER EQUI-LIPSCHITZ KERNELS
Lemma 2. Let Hay E P(X V) be arbitrary. For any h = May @ nn + h P y x x with Li-Lichchita th P(U)) there exists Ln-Lipschitz P(U) such that h Hay @
Proof. For readability, we write My E P())) for the second marginal of May. The required म is constructed as the My-a.e. pointwise limit of + (y) E P(U), as P(U) is sequentially compact under the topology of weak convergence by Prokhorov's 's theorem (Billingsley, 2013). For the proof, we assume Hilbert y and finite actions 21 making P(4) Euclidean.
First, (i) we show that (1) must converge for py-a.e. y y to some arbitrary limit, which we define as (ty) It then follows by Egorov's theorem (e.g. (Kallenberg, 2021, Lemma 1.38)) that for any > 0, there exists measurable set A E y such that (Ay(A) < c and (y) converges uniformly on V \ A. Therefore, we obtain that ø restricted to y A is Lu-Lipschitz as a uniform limit of Ln--Lischitz functions, hence My-a.e. -ip-ichchitz (ii) We then extend स on the entire space y to be Ln-Lipschitz. (iii) All that remains is to show that indeed, the extended # fulfills h My & n. which is the desired closedness.
(i) Almost-everywhere convergence. To prove the Myra.e. convergence, we perform a proof by contradiction and assume the statement is not true. Then there exists a measurable set A S y with positive measure Hy > such that for all y E A the sequence म, (y) E P(4)) does not converge as n - oc. We show that then, My @ in does not converge to any limiting h E P(Y x U4), which is a contradiction with the premise and completes the proof.
Lemma 3. There exists y E A such that for any O, the set Br(y") n A has positive measure.
Proof of Lemma 3, Consider an open cover UyEn (y) of using balls B, with radius r, and choose a finite subcover (B.(w))==..... of A by compactness of y. Then, there exists a ball (v")") from the finite subcover around point y* y such that My (Br((")) ) n A) > as otherwise (A) = ,, (w.) < E (y4) n A) = contradicts (y(A) > 0.
By repeating the argument, there must exist y" E A for which we have for any >> 0 that the ball (,()))) n A has positive measure. More precisely, consider a sequence of radii T = 1/k, k > 1. and repeatedly choose balls Bre+1 C Brg from open cover of Br, A such that ((Bret: n Br n A) > O, starting with Br, S y such that M(B,, n A) > O. By induction, thus have for any i that p(Br, O A) > 0. The sequence (B,, )EN produces decreasing sequence of compact sets by taking the closure of the balls Br,- By Cantor's intersection theorem (Rudin, 1976, Theorem 2.36), the intersection is non-empty, MKEN Br, # 0. Choose arbitrary
y" OREN Brg. then for any r > 0 we have that By, C Br(3") for some k by Th + 0. Therefore. u(Br(y") A) m(Br, n A) 0.
Bounding difference to assumed limit from below. Choose y" according to Lemma 3. By (14) in Lemma 1, since n,, (v²) E P(4) does not converge, there exists e > such that for all r > 0, infinitely often (i.o.) in n,
1
/ #(- v)dy(dy))
Wi #n *n(-10")
My (,(((())) (.(rr))
(u yn)
#(u v)
>e
My (y(BB(u)) B,(r))
WE
where for finite U4 W, is equivalent the total variation norm (Gibbs & Su, 2002, Theorem 4), which is half the L, norm, and ßे is not necessarily Lipschitz and results from disintegration of h into h By & a (Kallenberg, 2021).
30

---

Published as a conference paper at ICLR 2024
Now fix arbitrary e' (). Then, by the prequel, define the non-empty set U(r) S u by excluding all actions where the absolute value less than c i.e.
[e]]
1
c-6'
u(r) = u EU
in 13) (y(BB(u")) B,(r))
=(u (dyn
2
1
such that
1
/
Σ
in 5²)
My (B(((())) B.(r') #(u y) >
(18)
2
uEZ((r)
since we have the bound on the value contributed by excluded actions a & U(r))
Σ
(u (3)
/ (y((Bly))) Br(())
1
#(u v)
E 6-é'
<e-é. (19)
2
2
(Ell(r)
By Ln--ichchitz #n. also have for all (y")") that (1 #n (1("))
Lnr. Hence, in
particular if we choose in min (e' 2. 2: ब/
d c-dd
then for all y E B, (yy)
#n y) #n(u | < min (<'
è' E e
(20)
2 2 4/
wEU
and in particular also
(u 3) ( b² <
2
for all actions u (r)). such that by definition of (r)). we find that the sign of the value inside the absolute value must not change on the entirety of E B,yy"),i..
S #(u v/ (ly(ddy) B.(
sgn (u v)
My (b(((
sgn #n (n(ulv)) (,((B(rr))
1
#(u s s'lyydyy))
BA(r))
which implies, since the signs must match for all y with the term for y², by integrating y
sgn (J #n(u y ('))) - S
B. a(u ( mor(d)))
sgn (1 (An(u v*) #(u v'))) pa((//))
(21)
From the triangle inequality,
1 2
Σ uE(((r)
(in(u) n²) #(u v)) pay (dw)
Voacor (#n( (1) (u v)) My (dy)
Σ
2
wEl((r)
1
(An(ul - v)) My(dy)
2
Ba(r')
(E(((r)
31

---

Published as a conference paper at ICLR 2024
it follows then that for all y B,()") by (18) and (20), i.. in n
1 Σ /elcor
(#n(ul v) #(u y)) (dy)
2
Br(wr)
wEl((r)
MBAV (#n(ulv") #(u y)) My (dy)
vEl((r)
1
(
(#n(u 1u" #u (y(yyy
2
B+(0*)
(El(()
c' d'
> 6
(22)
2
2
Pass to limit of Lipschitz functions. + [0, 1].
Now consider the sequence of m-Lipschitz functions fm:: y x
Sm u) = min { (1-11-addee 11-mddyy 5r) - mr + +))+)
sgn ( (#n(u (1) a(u
(((ll'))) v/)) 1e (155)) mo(w)))
where (.)+ = max(-,0) 0) and sgn is the sign function. Note that fm 0 for all & B, (v"). Further, as m - XC,
Sm (y) sgn (LOOo)) (#nlu w²) #(u (l)) ())))))))))))
Then, by the prequel, we have by monotone convergence, +
// fm (v, ("n
v) - (ddu b) (
=I (B()) well
Σ
Smn (#n v) 7( v))
> -m..orr)
Σ sgn (1aces (#n(w)) 11n) a(u 2/)) ny (dsd') (ft(ul v) - -i(u (d)
well
Σ
SplCo (#n(u (oull)) = u)) (dd)
nE(((r)
Σ sgn (1s (#n(u (aalulv") 114) - #(u v/)) (o(c's))) / B.. (#n(u I w) - #(u (db)
(r)((())
> Σ
(An(ul) 1 11)) #(u w)) Myy (du)
vE(((r)
Σ
y) #n y ))))))))))
ug(((r)
Σ
Se.lss (# 11") - #(ul ,d)
uK(((r)
2 -c (B,((B(("")) - c-c'
2
4
2
32

---

Published as a conference paper at ICLR 2024
io.. in n. for the first term by (21) and (22), second by (20) and third by (19), noting that e' Hence, we may choose m such that e.g.
Jf fm (v. ( (du v) - #(du py (dy) > (e - - My (B,()))))
Lower bound. duality, we have
Finally, by noting that n- fm Lip(1) and applying the Kantorovich-Rubinee
// fly, (t v) - #(du (d)
Wilpy @ #n.Hy 2 R)
sup SELip(1)
l1 m Sm u) (du v) - #(du ()
1
m (
1 1
- By (B,
io. in n, and therefore My & n. A By 2 R. But My X was assumed be the limit of My fr. leading to a contradiction. Hence, My-a. -acc convergence must hold.
(ii) Lipschitz extension of lower-level policies. For finite actions, note that P(4) is (Lipschitz) equivalent to a subset of the Hilbert space TR( Therefore, by the Kirszbraun Valentine theorem (see c.g. (Cobzaş al., 2019, Theorem 4.2.3)), we can modify स to be Ln-Lipschitz not only a.e.. but on full V.
(iii) Equality of limits. #) 0 and therefore h By @ #. First, note that by the triangle inequality, we have
We show that for any 6 > O, W1 (h, By #) < c, which implies Wi(h, My ?
W1 Wi My 2 +) S Wi(h, My @ kn) + Wi @ #n. Hy @ #)
and thus by By मn h for sufficiently large n, it suffices show W1 Wi(Ky tia My (+) <E.
By the prequel, we choose a measurable set A C y such that (y(AA) < 2diam and 5n (v) converges uniformly on Y' \ A. Now by uniform convergence, choose sufficiently large such that (*) # (w)) < & on y \ A. By Kantorovich-Ruuinsee. duality, we have
/f flu, u)( v) #(du v))u,(dy)
Wi Wu & An. My 8 +)
sup seLip(1)
/
/ [(y. v) - #(du v)) my (dy( SELip(I)
sup
W, (1t + #(y)))((d)
/ Wi (v). ( (dg) + / W (())) diam + (1 2diam(())
ynA
<e
2 diam((U)
This completes the proof.
L EQUIVALENCE OF DEC-MFC AND DEC-MFC MDP
Proof of Proposition 2. The proof is similar to the proof of Proposition by induction. We begin by showing the first statement. We show oe = 14, at all times &. T, as it then follows that
33

---

Published as a conference paper at ICLR 2024
J(a)= Ei=0 (rrpe))(p)) = Σ ('r(pe) (()) under deterministic á E II. At time t 0, we have by definition Pu = po = iv. Assume Pt = Fe at time t, then at time + 1, we have
/|| P(z, (o) y) PMdy (e)
P1+1 (ier
(23)
-[J] P(a, u, pe) y, P (a) (dz) = pa+1
(24)
by definition of (v) = f [he]. which is the desired statement. An analogous proof for the second statement in the opposite direction completes the proof.
M OPTIMALITY DEC-MFC MDP SOLUTIONS
Proof of Corollary 3. As in the proof of Corollary 2, first show Dec-POMFC optimality
of o (+)). Assume (( (f))) < sup='em J(T'). Then there exists x' E such that
J( (+)) < J(T'). But by Proposition 1, there exists Il such that J(H') = (TT).. Fur-
ther, by Proposition 2, there exists #' E II such that J(#') = j(*). Thus, J(+) = J( -
(+))) < J(x') = J(a') = {(4') which contradicts क E arg max,. J(H'). Therefore,
$ (+)) E arg ma Jxx'). Hence, (+)())) fulfills the conditions of Corollary 1, completing the proof.
N LIPSCHITZ CONTINUITY OF RBF KERNELS
Proof of Proposition 3. First, note that
Vyn((,,8)1 exp (-Imm - w/13)
Kyb
v)
1 20²
20
diam max||yll 20
yEy
for diameter dia < oo by compactness of y, which is equal one for discrete spaces. Further,
x(yv, .) 2 My exp (- diam()))
Σ ME([My]
x(yv,v) =
Σ
20²
WE(My)
and [r[[bb 8)1 S
Hence, the RBF kernel y د k(yb, y)ps exp( for any 6 E [My]] since for any y, V. E V.
1y v/ with parameter a² O is Lipschitz 20
2 (v)m((,,
V yonlyn v) EveLNayl k(yW,)) v) Eve(My) Vyn(v,, ())()
EEeE(Ms) k(Uw,))) w))
1 diam(Y) maxl|y/|M My + My 20 diam()) max [[vl]]
1
<
M3 exp²
diam()))
20
yEy
20
diam max ||v|l
o²My exp²
diam(()) 20
for any b E [My]. Hence, by noting that the following supremum is invariant to addition of constants,
W1 z (y)
Σ
k(y6, u)pr. Z 'v')
Σ
r(y0, vf )p
bE(My]
bE(Mss
34

---

Published as a conference paper at ICLR 2024
/
sup seLip(1)
(v)
Σ
A(yh, v)dpb - (z')
Σ
xc(so, y )dpo
bE bE(N)s]
bE(My)
sup fetip()),Ifis diam(24)
/ (-)
Σ
x()6,)) -20 (x'
E x(0088))
8')
dpp
bE[Myl
DE DE[ANyl
Σ 2 (y) v) - z (- (U'))()))))))) be Myl
/ sdps
sup
SELiD(I)i/II$ diam(?4)
diam (J) max I1 Ily - 1'll diam(U))
S My o2My exp? (- 20 diam()))2)
which is Inn--ipschhtz. if
diam(Y) d() max
< Ln
20 exp² (- 2 d))m(
o² expx (-2de 202
diam())²
1
diam())) diam(()) max ly]l- wEY
Ln
Note that such 2 exists, as 0² exp² (- ff- diam()))?) + +00 as 0² +00.
O POLICY GRADIENT APPROXIMATION
Proof of Theorem 3. Keeping in mind that we have the centralized training system for stationary policy # parametrized by 0,
E # (p.))
# ===A(E)
T ~~PP P" F.. M.)
at te Vi), l441 1 (M Vi [N],
which we obtained by parametrizing the MDP actions via parametrizations & E E, the equivalent Dec-MFC MDP system concomitant with (4) under parametrization A(g) for lower-level policies is
&t R (pe). Åot T(pe Et) = fIf P(x, u(x,u,full (.f)))(())((du)) pdy xx, (d))
(25)
where we now sample & instead of h4. Note that for kernel representations, this new Í indeed Lipschitz, which follows from Lipschitzness of fu, & po A(E:) in (fi,
Lemma 4. Under Assumptions la and 3, the transitions I of the system with parametrized actions are Lipshihhhitz with Lg- = 2Lp + LpLx + 2L'Lpm
Proofs for lemmas are found in their respective following sections.
First, we prove d. - dge in P(P())) by showing at any time t that under # the centralized training system MF fu converges to the limiting Dec-MFC MF fú, in (25). The convergence is in the same sense as in Theorem 1.
Lemma 5. For any equicontinuous family functions F S (R)) under Assumptions la-lb and 3, all times we have
sup JeF
E [f(iN) f(1)]] + 0.
(26)
We also show that (H, (v. E), since we can show the same convergence as in (26) for new conditional systems, where for any μe, E we let jio the initial state distribution of the centralized training system.
(4g and & = & = Eo at time zero, where joo
Lemma 6. Under Assumptions la and 3, as N + oo, we have for any E P(X), & E E that
120 (() - (p, +
35

---

Published as a conference paper at ICLR 2024
Furthermore, Q" (v.E) is also continuous by a similar argument.
Lemma 7. For any equicontinuous family of functions F S PR((),, under Assumptions la and 3, at all times t E T, the conditional expectations version of the MF is continuous in the starting conditions, in the sense that for any (Mm En) (v, E),
sup SEF
[E [f(ee)
Po0 = 50 = Enl (f(( AO u.5o = =E]l - 0.
Lastly, keeping in mind d% = (1- EIET
X C+o (Zt ) we have the desired statement
(1 - (1-7)- E pondy (M) (Q"( log (5 m0) - VoJ(+9)
$ (1 - 7) Bundy (mt
- Q' (4) (Q(vss)) Vo log # t 11)
+ +|[11 7) Eu (m) (M) log (E [J] veJ(a"))
s (1 - Bundy (M))(() [(09(u.)) (4( Vo log 49 10)
E ((P) (2 Vog (E (1t))
log # (E [c]
Σ y (E 1Q log (E ) - Q (pr, () log # [e)]
+
for the first term from (1, - (1, {) uniformly by Lemma 6 and compactness of the domain, for the second by Assumption and la uniformly bounding Vo log x², Q and choosing sufficiently large T, and for the third by repeating the argument for Q: Notice that
7 Eg [2 (x Vo log (E ) (Re, (E pc]]
(E [r(pe) Aa = P.N Eu = E] V log (E pa,))
- E(r((e)) | juo = pr, &o
E V log (E (+))]
7' E E~ (aN)
EE [r(jw) noo = -u,"" Eo- = { V log # 0 (E) (.
Er((fe) poo- = fue Eu f Vo log # pc))]
where the inner expectations are on the conditional system. Letting T' sufficiently large bounds the former term by uniform bounds on the summands from Assumption 3. Then, for the latter term,
apply Lemma 5 at times I' < 7" to the functions f(u)= f IE (r(āv) مم μ,, Eo = <] V log * (E)
4)** (dg p)dg, which are continuous up to any finite time t' by Lemma 7 and Assumption 3. O
P LIPSCHITZ CONTINUITY OF TRANSITIONS UNDER PARAMETRIZED ACTIONS
Proof of Lemma 4. We have by definition
[|ʃ P(z, (L)E)(a) v) P) (dy a. (dd
= -]ll P(a, u, (i) Ede[payl Klyn, y) (d)
Eee[Ma) H(Ub, v) p (dly x. ())((d))
36

---

sup Published as a conference paper at ICLR 2024
Consider any E E. j. E P(X). Then, for readability, write
Pay = fi @ P't),
Payu = Aay @ A((E),
Paynar = Parya 2 P(A),
Kay = fl' & P" (p'), Pays = Paay @ >A(E'), Plaguat == Plago 2 P()'),
AP(: 2, u) == P(. Ir u. P(. 1,,
AA( (1) == Eb k(yb: y) (2o(E)()) (o(E))())
Es (c(6,8))
A | x) = (. x, P. x. ((), Ap == f f
obtain
[J]] P(z, u.) u.
Eo n(yo (E) pa (" x, p))((d)) En x(D,, y)
Wi
JI] ux Es kiybo y) Ao puddy x. fz) (dx))
E(y.. y)
SELip(1) J/ʃl fxx') (Drzynar (dx,, dy, du, dx') Deeyer (dx, dy, du,, dx'))
sup SELip(1)
s sup SELip(1) ((x') xx ux dy, du)
sup fELip(I)
Iz)))(ar' (z,u,
y) U) dy))
feLip(1) [(2))((a') (1), Es koun, y) ())o (d ())(dx)
sup
Es C5K(,,, v)
SELip(1) f())) (x, Eo r(lb y) ())o(" Es k(bb)))
po x. (d)(x)
sup sup 1SCO)AP(a Ix, u)
seTip(I) (x
+ sup sup feLip(1) (x.))xxx)) // f(x') |x, (r,,,A))A(C 2)
/I/ ((x')P(a'') (x, En klyn, v) ") P 2²) Eo ruyb. v)
sup sup feLip(1) ZEX
[(x)) ",,, En k(yb, (ddu))) pe pd Ir, (z(())
sup feLip(1)
Es k( v)
bounded by the same arguments as Theorem 1: For the first term, we have that the function x' + f(x'' is 1-Lipschitz, and therefore
sup sup L1) / ((d') xx)
(, P(dx' [x, (x,,,,1)) s (
by Assumption la.
For the second term, we have Lp-Lipse a u+ [(())))(da) |a, u, R²), since for any ʃ E Lip(1) and (z.z) X x V, we obtain
// f(x') f(x') u.) / f(z')P(dx' x, m.
< Wi(P(a, u, (²), P(x, ui, R1))) < Lpd(u,u')
for any u, u' by Assumption la, and therefore
// f(l)) x, u. fI) Es K(y6. v) (Ao(E) - (d(5) E. k(yb. v)
sup fELip(1)
sup [x,,,eXxx)
37

---

Published as a conference paper at ICLR 2024
Es K(yo D)L,W, (Ao(E)
S LpLid(E,EE
by Assumption 3.
For the third term, we have L' L'-Lipschitz y + JJ f(x') | a, (²) Eo (lgn where Ev (MSMM)
diaun()) diami(4) ma [lyl, since for any f E Lip(1) and x X, we obtain we define L' = Lp 20 expp 207 diam
VI f(x') |x, (()
Eo NL(b. v)Ao(E Eo m(bb yf ((o())(())) Es k(bh.3))
Es K(yb,
Lp. diam(Y) diam(U) max |1 d(y,y') 2 exp² (- 1 2n2 ()))
for any y. v E by Proposition 3 and the prequel, and therefore
/.ʃ ((z))((de x, u. Es <V6,, y) (dd))(dd))) feLip(1).z
(P" x. ()) (dy I 2r, xz)
sup
Eo k(y6: v)
$ I' L1 A')
by Assumption la.
Lastly, for the fourth term, SJʃ [xx) (a')P(da'' u., En Mio-- pd x., Rl)) is Es k(v6.0)
similarly (Lp + L'Lp LLipschiz,, since for any f E Lip(1), we obtain
VII P(da) x, u.) Es kym, y) (ddu(")
(d x²)
-IIl f(z') ,d",,, Eo k(Ub. U)Ao(E (dyy x² x²
Koyo00.8 y)
< [S] ((x') x.u.A) P(dx' (()) Es k< Ao pd (dy Ix, f()
Ee Cybb. v)
+ +|/If f(o') f(o')((de uu,,, Eo k(yb, V) EB k(166. y) $ LLd((Ixx)) + L' L'Lp Lud((e, = (Lp + L' )
(P" x,,(²) pd | x", ,²))
for any x. E X by the prequel, which implies
|SS/ ((t))(dx' (,,,,fi' Eo k(yb, v) (s((") pdd xx, (f(dz) - f (dx))) s (Lp ('..))
sup fELip(1)
E, kyy, y)
Overall, the map í is therefore Lipschitz with constant La 2hp LpLx 2L'
Q (CENTRALIZED) PROPAGATION OF CHAOS
Proof of Lemma 5. The proof is the same as the proof of Theorem The only difference is that for the weak LLN argument, we condition not only on a:V. but also on &., while for the induction assumption, we still apply to equicontinuous functions by Assumption 3 and Lemma 4.
In other words, for the weak LLN we use
E (ds (pit. T (iN.&))]
Σ m
m E
// Sm d (Nitz T (nM, & )
< sup
E
E
I Sm
(nit1= - 7 7"(11) (n:" ....&.) a/ &
m>1
38

---

Published as a conference paper at ICLR 2024
and obtain
[V fm (nitz - 1it (i))
E
2i"" Er
2
E
Σ =EIN
(fm(at+1)=
(1(+1) x."" .616
2/ E.
- N
4
+0,
while for the induction assumption we use the equicontinuous functions μ + S F(T(M (E jO by Assumption 3 and Lemma 4.
R
CONVERGENCE OF VALUE FUNCTION
Proof of Lemma 6. We show the required statement by first showing all times / that
sup SEF
E [((( ((x) Au / Po: Eu Eo + 0.
(27)
This is clear at time 1 = 0 by joo
isn = Eo and the weak L.N argument as in the proof of Lemma 5. At time 1, we analogously have
=
sup fEF
E
[(a))) f(11) i 1 = io, Eo = === [o]
[s(aN) (00T6)"EE)))
nuo = 14, Eo = = =
S sup FEF
sup fEF
E [[CT(C Ev)) -((z))
iAn = === io Eo = =
and
f((()t) f( iuo == ioo. =
sup fEF
E s(NNN)) - (Ei((N,E)))a ii yas
I sup /EF
Ho = =1,=0,50 5, =
| JCTIN (E') a jag' f(1++1)
sup fEF
E.
N0 11 ioo Eo = 0o
for times + 1 1. each with the weak LLN arguments applied the former terms (conditioning not only on 5.". but also Et), and the induction assumption applied the latter terms, using the equicontinuous functions ++ ()), judl by Assumption 3 and Lemma 4.
CONTINUITY OF VALUE FUNCTION
Proof of Lemma 7. For any (Jus En) - (),,), E), we show again by induction over all times that for any equicontinuous family F,
sup E (fte) AÃ = unn Eo En] - E Ao = p. = <] -
(28)
SEF
as N + oo, from which the result follows. At time 0, we have by definition
sup |E (f()) Ao Pns En] (f)) Ao n, Eo [] 0.
feF
Analogously, time 1 have
sup feF
E (f(r) p = En = Enl -E (f(r | A0 = Eu = El1
sup feF
E ((T(Po:Eo)) Ao Pn, Eu En] E
[(TT En))
Ao p. =
39

---

Published as a conference paper at ICLR 2024
fEF suu f(T((a: Sn)) - (((((E))) +
by equicontinuous f and continuous / from Lemma 4.
Now assuming that (28) holds at time t 2 1, then at time t + 1 we have
sup fer
[R (f(tt)) I No = Un: So = En] B (f((+1)) Ao = 50 E]]
| ((T(MNE)))) (e)de' Po :n = En
sup fet
E I ((T((EEE)))E pi)de 140 = 50 =
sup (g( po = Un. = - Ao = 14, = E] - 0
9Eg
by induction assumption on equicontinuous functions by Assumptions 1a and 3, Lemma 4, and equicontinuous f F, as in Theorem 1.
The convergence of |Q" En) (u, + 0 thus follows by Assumption la.
40


Reviews:
[{"review_id": "h8UjTmGWSH", "reviewer": "Reviewer_PQjP", "summary": "This paper studies mean field control problems with decentralized decisions and partial observations. After introducing the problem, the authors prove theoretical results connecting the mean field problem and the N-agent problem. They then propose a policy-gradient based method and provide experimental results on several examples.", "questions": "Is it possible to replace the assumption on policies (which means restricting a priori the set of policies) by an assumption on the model which would ensure that the optimal policy satisfies this Lipschitz property? And would this be sufficient for your purposes", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper seems rigorous and has both theoretical contributions and numerical experiments.", "weaknesses": "Some assumptions seem quite restrictive, such as Assumption 1b which says that the policies should be uniformly Lipschitz (see question below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JdpT3JlMSM", "reviewer": "Reviewer_gUVQ", "summary": "This paper applies MARL to the proposed decentralized partially observable mean-filed control (Dec-POMFC) model, which can be reduced to single-agent MDP. A PPO-based method in the CTDE framework is applied to solve such problems. Theoretical analysis is also given.", "questions": "- What is the particular difficulty of solving the Dec-POMFC system, and how does the proposed method solve such difficulty? It would be easier to follow the paper if these questions were explicitly explained.\n- How much information do you assume each agent can observe? i.e. how severe is the partial observation problem in the proposed model and algorithm?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well written.", "weaknesses": "- It is unclear to me where the particular difficulty is from. If it is from the partial observation, how do you solve the partial observation problem? Note that many Dec-POMDP problems can be grouped into weakly-coupled POMDP [1] where the partial observation can be sufficient to make optimal decisions.\n- The CTDE training is very common in MARL, and the baseline of IPPO is not fair as it cannot use global information while the proposed method uses more information.\n- The benchmark looks simple. It would be good to include more realistic and complex benchmarks to indicate the importance of the studied question and the proposed method.\n\n[1] Witwicki, Stefan, and Edmund Durfee. \"Inﬂuence-Based Policy Abstraction for Weakly-Coupled Dec-POMDPs.\" *Proceedings of the international conference on automated planning and scheduling*. Vol. 20. 2010.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HmJxWuBlnE", "reviewer": "Reviewer_xiBU", "summary": "The manuscript introduces Decentralized Partially Observable Mean Field Control (Dec-POMFC) to extend scalable MFC to a decentralized and partially observable system. The paper includes rigorous theoretical proof. The experiments are performed on representative collective behavior tasks such as adapted Kuramoto and VIcsek swarming models, against the SoAT IPPO methods.", "questions": "1. The manuscript mentioned extending the proposed methods to handle additional practical constraints and sparser interaction. However, the reviewer sees it can be hard given the adopted assumptions in the paper. The reviewer is curious about the feasibility of making this extension.\n\n2. The author claims about the generality of the proposed method. Can the authors elaborate on which design in the algorithm contributes to this generality in addition to using RL itself?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles a challenging problem featuring partial observability, multiple agents, and decentralization. This area is worth exploring.\n\nThe paper is well-written and provides very details algorithm description and theorectical proof. The experiments are carefully carried on.", "weaknesses": "The experiments were carried out on adapted Kuramoto and VIcsek swarming models. The authors report training curves, and the returns as the number of agents increases in the main manuscript. However, there is no direct comparison with the SoAT methods in terms of the second method.  In addition, there is no explicit summary or discussion about significant improvement, and it is hard for the reviewer to judge the contribution of the proposed methods in terms of the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)