Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning
Authors: Barna Pásztor
Andreas Krause
Ilija Bogunovic
Venue: Accepted by TMLR ()

Abstract: Learning in multi-agent systems is highly challenging due to several factors including the non-stationarity introduced by agents' interactions and the combinatorial nature of their state and action spaces.
In particular, we consider the Mean-Field Control (MFC) problem which assumes an asymptotically infinite population of identical agents that aim to collaboratively maximize the collective reward. In many cases, solutions of an MFC problem are good approximations for large systems, hence, efficient learning for MFC is valuable for the analogous discrete agent setting with many agents.
Specifically, we focus on the case of unknown system dynamics where the goal is to simultaneously optimize for the rewards and learn from experience. We propose an efficient model-based reinforcement learning algorithm, $\text{M}^3$--UCRL, that runs in episodes, balances between exploration and exploitation during policy learning, and provably solves this problem. Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC, obtained via a novel mean-field type analysis. To learn the system’s dynamics, $\text{M}^3$--UCRL can be instantiated with various statistical models, e.g., neural networks or Gaussian Processes. Moreover, we provide a practical parametrization of the core optimization problem that facilitates gradient-based optimization techniques when combined with differentiable dynamics approximation methods such as neural networks.

Published in Transactions on Machine Learning Research (05/2023)
Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning
Barna Pásztor
barna.pasztoraaicomm etwaa.cteeh..t
ETH Zürich
Tijja Bogunovic University College London
i. &bogumoniiccucll.cc.ooo uk
Andreas Krause
krausen@ethz.ch
ETH Zürich
Reviewed on OpenReview: https: // optp//veew.r...w....mo. net/ forum? id= gucDSDYUZe
Abstract
Learning in multi-agent systems is highly challenging due to several factors including the non-stationarity introduced by agents' interactions and the combinatorial nature of their state and action spaces. In particular, we consider the Mean-Field Control (MFC) problem which assumes an asymptotically infinite population of identical agents that aim to collaboratively maximize the collective reward. In many cases, solutions of an MFC problem are good approximations for large systems, hence, efficient learning for MFC is valuable for the analogous discrete agent setting with many agents. Specifically, focus the case of unknown system dynamics where the goal is to simultaneously optimize for the rewards and learn from experience. We propose an efficient model-based reinforcement learning algorithm, M UCRL, that runs in episodes, balances between exploration and exploitation during policy learning, and provably solves this problem. Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC, obtained via a novel mean-field type analysis. To learn the system's 's dynamics, M--UCRI be instantiated with various statistical models, e-g., neural networks Gaussian Processes. Moreover, we provide practical parametrization of the core optimization problem that facilitates gradient-bass based optimization techniques when combined with differentiable dynamics approximation methods such as neural networks.
1 Introduction
Multi-Agent Reinforcement Learning (MARL) extends the scope of reinforcement learning (RL) to multiple agents acting in shared system. Tt is receiving considerable attention given a great number of real-world
applications including autonomous driving (Shalev-Shwartz et al. 2016), finance (Lee et al. 2002 Lee et al.
2007 Lehalle Mouzouni] 2019), social science (Castelfranchi 2001 Leibo et al.) 2017 swarm motion
(Almulla al. 2017), e-sports Vinyals et al. 2019 Pachocki et al. 2018), and traffii routing (El (El-Tantawy ct al. 2013)) to name a few. Despite the recent popularity, analyzing agents' performance these systems remains particularly challenging task for several reasons including non-slationarity, scalability, competing learning goals, and varying information availability of agents. In this work, target the challenges of non-stationarity and scalability. From the perspective of an individual agent, the dynamics of the system are non-stationary, since other agents update their behaviour, which alters over time how the environment reacts to certain actions of the agent. Multi-agent systems also suller from a combinalorial nalure due to the exponential growth of the state and action space as the number of agents in the system increases.
To tackle the previous challenges, Mean-Fieer Control (MFC) exploits the insight that many relevant MARL problems involve large number of very similar agents that coordinated via. centralized controller.

---

Published in Transactions on Machine Learning Research (05/2023)
As motivating application, consider ride-hailing service in which central dispatcher coordinates the routes of many drivers around the city. MFC considers the limiting regime of controlling infinilely many identical collaborative agents, and utilizes the notion of mean-field approximation from statistical physics (Weiss] 1907) to model their interactions. Specifically, instead of focusing on the individual agents and their interactions, the slate of the system is described via the agents' stale distribution and the dynamics and rewards are formulated accordingly. Despite the introduced approximations, the solution to the MFC problem often remains adequate for the finite agent equivalent (Lacker) 2017].
The focus of this work is on learning the optimal policies in MFC when the underlying dynamics are wnknown. In complex domains such as flects of autonomous vehicles or robot swarms, this is often the case, and the mod- els consequently need to be learnt from data, giving rise to a well-known exploration exploitation dilemma. In this paper, we propose the Model-Based Multi-Agent Meon-Field Upper- Confidence RL algorithm (M²- UCRL) that is provably efficient and can effectively trade-off between exploration and exploitation during policy learning. Similarly to other Model-Bascd Reinforcement Learning (MBRL) algorithms, M M col- lects data about the unknown dynamics online (i.e., by proposing and executing a policy in the true system) and estimates the possible dynamics the system might follow. To optimize the agent's actions, it simulates the unknown system by using the estimated dynamics. Chua ct al. (2018) has shown that MBRL can solve challenging single-agent RL problems with low sample complexity. We transfer this property to multi-agent problems and show that M²-UCRI can efficiently solve the MFC problem with unknown dynamics.
Related work. Our M UCRL extends the class of Model-Based Reinforcement Learning (MBRL) algo- rithms that rely on the optimism-in tae-n--i--tte------------- face-face-of--meee -aate---f-mcceeaiitt principle. Algorithms based on this approach achieve optimal regret for tabular MDPs (Brafman & Tennenholtz) 2002 Auer et al. 2009; Efroni et al.
2019 Domingues et al. 2020), while in the continuous setting, Abbasi- Yadkori & Szepesvafi (2011) and Jin et al. (0200) show (VVT) regret bounds for LQR and linear MDPs, respectively. The recently designed
episodic model-based algorithms (Chowdhury & Gopalan, 2019 Kakade et al.) 2020 Curi et al. 2020:
Sessa ct al. 2022) achiove similar regret in the kernelized sctting. Scssa ct al. (2022) consider the general multi-agent selting, and prove a regret bound with ((2 dependence on the number N of agents. In comparison, 2 M²-UCRL assumes an asymptotically large population of identical agents. Crucially, as we will show, its performance docs not depend on the number of agents, making it more suitable for large systems.
Mulli-Agent Reinforcement Learning (MARL) has seen tremendous progress in recent. years as many real-world applications involve interactions among a number of agents. Buçoniu et al. (2008) provide an overview of classical and early MARL results. Several other MARL surveys focus on: non-stationarity induced by the agent interactions [Hernandez-Lear et al., 2017]. deep MARL [Hemandez-Leal et al. 2019
Nguyen et al. 2020). theoretical foundations in Markov /stochasticc Setpuehasstc/eettesieeeeeee /extensive--oomm games (Zhang et al.
Recently, [Zhang et al. (2019) and Nguyen et al. (2020) recognize the lack of practical model-based MARL algorithms (with an carly exception
2019) and cooperative MARL [OroojlooyJadii & Hajinezhad
2019
of the R-Max algorithm proposed by Brafman & Tennenholtz
2002) for two-player zero-sum Markov Games). We fill this gap by designing practical model-based algorithm that considers a large population of agents, and that is compatible with deep models.
2000
Modeling interaction via the mean-field approach in MARL has recently gained popularity due to the introduction of Mcan-Field Games (MFG) (Lasry & Lions 2006a b Huang et al. [2006] 2007]. MFG consider the limiting regime of competitive game among identical agents with the objective of reaching a Nash
Equilibrium. Various works (Yin et al. 2010 2013
Cardaliaguet & Hadikhanloo 2017
Guéant et al. 2011
Bensoussan et al. 2013 Carmona & Delarue
2013
Carmona et al. |018 Gomes et al.
2014) mainly consider
solving MFGs in contimouus-time, while some very recent ones (Yang et al. 2017 2018 Fu et al. 2020 Guo
et a.. 2019 2020 Flie et al. 2020 Agarwal et al. 2019 analyze the problem from
the discrete-time RL.
perspective. The progress of the field has been recently summarized by Laurière et al. (002))
At the same time, cooperative discreto-time Mean-Field Control, the main focus of this paper, has received
significantly less attention. Gast et al. (2012): Motte & Pham (019);; Gu et al. (2020 2021 and Bäuerle (2021) consider planning with known dynamics. In the learning setting, Wang et al. (2020): Carmona et al.
(2019b); Gu et al. (2021). Angiuli et al. (2022 2021 extend the Q-Learning algorithm and show convergence for specific variants of MFC focusing on discrete state and action spaces. The mean-field extension of the
2

---

Published in Transactions on Machine Learning Research (05/2023)
Anear-quadrate problem has been studied by Carmona et al. (2019a): Wang al. (2021) Carmona et al. (2021) | with policy optimisation methods. [Sbramanian & Mahajan (209g) also proposes a policy-gradient based algorithm that solves Mean-Field Games and Control problems locally. Additionally, [Chen et al.) (2021);
:: Li et al. (2021): Wang et al. 2020) consider the approximation error between the Mcan-Field and the related Agett- Reinforcement Learning problems. These previous model-free methods are either sample inefficient, assume access to simulator or consider finite state and action spaces. On the contrary, our algorithm model- sample-efficient, works for continuous spaces and learns the policy online by performing exploration on real system.
Main contributions. We design novel Model-bosed Multi-agent Mean-field Upper Confidence RL (M²- UCRL) algorithm for the centralized control problem of a large population of collaborative agents. M UCRL is practical, performs exploration on a real system (requires no simulator and efficient in terms of samples), and is compatible with neural network policy learning. Our main contributions are the general theoretical regret bounds obtained via a novel saan--iild-bbseeee analysis, that we also specialize to Gaussian Process models. In the main step of our theoretical analysis, we bound the distance belween the mean-field distributions under the true and the approximated dynamics via the model's total estimated uncertainty, Finally, we pose the exploration via entropy maximization problem used in recent MFG survey (Laurière et al. 2022) in the continuous state and action domain and demonstrate the performance of our algorithm in it. Our results show that M²-UCRI is capable of finding close-to-optimal policies within a few episodes, in contrast to model-free algorithms that require at least six orders of magnitude more samples.
2 Problem Statement
We consider the episodic Mcan-Field Control (MFC) problem with time-horizon H, compact state space S C IP and compact action space A C IR that are common for all the agents in the system. We index episodes with the variable and use to denote the number of episodes passed. In standard N-agent settings, the system's state in episode at time h E {O...., H - 1} is described with individual agents' =(,))) ) SN which grows exponentially in the number of agents N and represents common issue in scaling MARL algorithms. In MFCs, we assume that all agents are identical and the population is asymptotically infinite, i.e., N + co, therefore, the agents' states can be described by the mean-field distribution:
states (1) ft.h ,1,h
N (s) = lim Σ
1
No000 N
where Mit belongs to the space of probability measures P(S) over S. Due to the assumption of identical agents in MFC, we can focus on representalive agent from the population that interacts with the distribution of agents instead of individual agents and interactions.
System Dynamics. Before every episode t, the representative agent selects a policy profile -1) (TT.......,,,,--)) where the H individual policies Teh S x P(S) + A are chosen from set of admissible policies II (e.g., we parametrize our polices via neural networks; see below). At every time h, the representative agent selocts an action Or = (st, Pr,n). The environment then returns reward r(s,,h oc, He,),, and the agent observes its new state $1,4+1 and the mean-field distribution Mr,+1
In MFCs, the system's dynamies are typically given by a McKean-Vlass type of stochastic processes and depend on the agent's state, action and the mean-field distribution:
8t = [(8t,,t ae,o M1, + Wi,b:
(1)
where Wa,h is an d. additive noise vector. This is in contrast to the standard single-agent model-based RL (cf. [Chowhhury & Gopalan 2 Curi et al. (2020); Kakade et al. (2020)), where the dynamics only depend on agent's action and state. Crucially, since the focus of this work is on learning in MFC, we
assume that the true dynamics are unknown, and the goal is to explore the space and learn about f over a number of episodes.
To do so, the representative agent relies on the collected random observations,
We expect our results to easily extend and account for unknown rewards by using the same modeling assumptions for the reward function (e.g. as in Chowchury & Gopalan (019
3

---

Published in Transactions on Machine Learning Research (05/2023)
D. - (((s,h dt (r,h), (1,++)))k=0 which come from the interaction with the true system (i.e., from policy rollouts) during episode t.
Finally, after every episode, we assumne that the whole system is reset and the agent's 's initial state so is drawn from a known initial distribution Po, i.e. 8u ~ pe, that remains the same in every episode.
Mcan-field Flow. Since, MFC, all agents are identical, interact common environment, and follow the same policy T. (Tr......, mean-field flow property shown by Gu et al. (2020):
-T.H-1),, "o the subsequent mean-field distributions satisfy the following
Lemma 1. For given inilial distribulion pa, dynamics I and policy æ ( Th-1), the mean-field 1H-1 distribution trajectory (pn)h=) follows
Ph+i(ds') = / 1h (ds)))(s))+ E ds'),
(2)
AES
where PA P(S) for all 2 0, on (h)) 8h+1 is given by Eq. and pn(ds) P(sn ds} under π
We provide the proof in Appendix B.2 To shorten the notation, we use P(pr,hs Te,h: to denote the mean-field transition function from Eq. (2), i.e., we have 1r (Mu,, Te, f). The lemma shows that the next mean-field distribution explicitly depends on the unknown f (since St,411 = f at p4,x) + wi,)), policy played by the agents Tr,,, and previous state distribution ji,h-
Performance metric. Given policy profile æ = : the performance of the representative agent measured via the expected cumnlative reward:
11
[(x) = Σ h=0
rsh. an. mn)
(3a)
s.t. ag Th (8h;HH))
(3b)
8h41 = f(h,, Oh [4h) + Wh:
(3c)
Mh+1 = 2 (p), The f):
(3d)
where the expectation is taken over the noise in the transitions and initial distribution sp Mo- By considering the representative agent's perspective, the goal is to discover a socially optimal policy " that maximizes the expected total reward, i.c., T E arg max I(Tr)
(4)
In our theoretical analysis, we make the following assumptions regarding the system's true dynamics, reward function and the set of admissible policies. We use 2, 2 2 = ASxAxP(SS to denote (s,a, and (s',a', respectively, and we make use of the Wasserstein-1 metric: Wi v²) inf, Ss S || - -wlll dv(x, (), where is any probability measure S x S with marginals μ and p' (see Appendix A for useful properties of (1, that are used in our analysis).
Assumption 1. The transilion function J is T, ------chiaa-cooiinnoooe where ((2,2') ;= [| s + ||e a'l12 + Wi(u, n²) and wi, are i.i.d. additive o-su Gaussian noise vectors for all t 2 1 and h E {0..... Assumption 2. The set of admissible policies II consists of La Lipschita-contiimun policies such that for any क E II: [s p) - ((s',pp')) ,2))l S [= - s'Il2 + WW(p. p')), and the reward function LipLishchitz- continuous, i.e,, (r(=) r(=')((+ [T,d(z,=')
i.e., II((=) - s(=)]l2 S L,dd(z, z²)
-8-1.. 1).
Regularity assumptions like these are standard in the single-agent prodel-based reinforcement learning liter- ature (Jaksch et al. 2010: Curi et al. 2020 Chowdhury & Gopalan 2019 Sessa et al.) 2022) and mild since, in practice, the policy and reward function classes are typically chosen to satisfy the previous smosthness assumptions. In our experiments (see Section 5 we parametrize our policies with Neural Networks with Lipschitz continuous activations (c.g., tanh, ReLU and lincar). We note that the Lipschitzness of such poli- cies then follows from that of the activations when the network's weights are bounded (in practice, this can be done by directly bounding them or via regularization).
4

---

Published in Transactions on Machine Learning Research (05/2023)
Remark 1. The episodic Hean-Field Control problem relates closely to the finite-horizon case of the evolutive Menn-Field Game problem described in Laurière el al. (2022). Both setlings consider mean-field distribultons evolving over time depending on the agents' policies, however, the MFC' problem focuses on collaborative agents while agents in the evolutive MFG are competitive. This distinction leads to different solution concepts of the problems; social welfare in MFC and Nush Equilòrrium in MFG.
3 The -U-UCRL Algorithm
In this section, we tackle the Mean-Field Control Problem with unknown dynamics from Eq. (4 by relying on model-based learning scheme. Our Model-based Multi-agent Mean-field Upper Confidence RL MCCU) algorithm uses statistical model to estimate the system's dynamics and to effectively trade-off between exploration and exploitation during policy learning. Before stating our algorithm, we consider the main properties of the considered dynamics models.
Statistical Model. Our representative agent learus about the unknown dynamics from the data collected during the interactions with the environment, i... the policy rollouts. In particular, we take a model-based perspective (sce Algorithm D where the agent models the unknown dynamics and sequentially, after every episode t, updates and improves its model estimates based on the previous transition observations, i.c., Di U....UD where D, = {[(St,h, &e (r,a); (=)++)))==)) is the set of state transition observations in episode t. To reason about plausible models at every episode t, the representative agent can take a frequentist or Bayesian perspective. In the first case, it estimates the mean m, : 2 + S and confidence E, : 2 - R functions. In the second case, the agent estimates the posterior distribution over dynamical models p(f(D,), that leads to m²(2)= Fy>p//i,,) [f])], and (2) Var[f(=)]. We denote (.).) = diag(E,()) and make the following assumptions regarding the considered statistical model irrespective of the taken perspective:
Assumption 3 (Calibrated model). The statistical model is calibrated w.r.t. f, i.e., there is a known non-decreasing sequence of confidence parameters {3e [Se)u2, each Be Rzo and depending on 6, such that with probubility al least. 1 5, it holds joinlly for all and E Z that [f(=) (0,(=)m+==+)))))) S 3,0((2) elementaiise.
Assumption 4. The function (.).) is Lo cdipschitz-contimunn for all t 1.
The calibrated model assumption (or its equivalents) is standard in online model-based learning (Srinivas ct al. 2010: Chowdhury & Gopalan 2017 2019 Curi et al. 2020 Sessa et al. 2022). It states that the agent can build high probability confidence bounds around the unknown dynamics at the beginming of every episode. As more observations become available, we expect the epistemic uncertainty the model (that arises due to the lack of data and is encoded in (-)) to shrink, and consequently allow the representative agent to make better decisions it becomes more confident about the true dynamics. Our theoretical results obtained in Section
E hold for general model classes long as Assumption 3 and Assumption 4 are satisfied. Below and in Section we provide concrete conditions (about f' and models for which this assumption provably holds.
Algorithm. At the beginning of episode t, the representative agent constructs the confidence set of dynamics functions, denoted by Ft-1, satisfying the elementwise confidence interval in Assumption g with m--1(-) and 1()) estimated based on the observations up until the end of the previous episode 1, i.e.,
Fi-1= - - (1-10 elementwise and Vz E 2].
Then, the agent selects the optimistic policy ग, which achieves the highest possible cumulative reward over the set admissible policies, II, and plausible system dynamics Fi-1. In particular, the representative agent solves the following problem:
H-1
n: arg max max E HEll JEF
Σ
r(ae,n Tr,n
(5a)
s.t. āt. = (5 Pr,,))
(5b)
8 = f(Be, Te,n, (.) + Weh
(5c)
PcM+1 = (ir,, Te. Te, f),
(5d)
5

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm 1 Model-based RL for Mean-field Control
Input: Calibrated dynamical model, reward function r(5t,,, 0e, Me,.)) horizon H, initial state 81,0 Moo for t= 1.2,... do
Use - -CURRL to select policy profile π. = (Tr,0r... TtA--1 by using the current dynamics model and reward function, i.c., solve Eq. (5)
for h .....,
do
0t = t (8t.h, (t,h), $r,6+1 = f(st,h, dt, Mt.hi) We,h
M1,++ = P(HL,,, Ft.
end for
Update agent's statistical model with new observations {(SEh, (g.hs/ 81,4 1 Reset the system 1, & no and $14100 ~ Mc+1.
end for
where the expectation in Eq. Gal is taken w.r.t. the initial state distribution Ho and noisc in transitions.
The policy found by solving the previous problem is then used in the true multi-agent system (e-g. used by every driver in a ride-hailing service) for one episode. It induces the observed mean-field flow, and aims to improve the total collective reward. After the episode ends, the representative agent augments its observed data, i.., D1s = D1:-11 UD, and improves its model estimates m.(-) and (-).. The algorithm is summarized in Algorithm 1 Since solving Eq. 5 is a challenging task even for known dynamics and the focus of this work is on the statistical complexity of learning in MFC (similarly to Kakade et al.] (2020); Chowdhury & Gopalan (2019;; Curi et al. (2020); Sessa al. (0022)
assume that can solve Eq.
for given Fi-1 and II. We propose a practical implementation below with details solving Eq.
The algorithu implements the upper-confidence bound principle, since the system's 's true dynamics ʃ belong to the confidence set Fi-1 with high-probability (by Assumption 3). Consequently, the reward achieved by
under the best possible dynamics ff upper bounds the performance of under the true dynamics f.
Practical Implementation. In general, optimizing over the set F is not tractable. However, a practical problem reformulation has been designed (Moldovan et al.) 2015 Curi et al. 2020 for single-agent problems, which defines an auxiliary policy select the dynamics function f E Ft-1- We generalize this approach to the MFC setting implement M UCRL for our experiments (see Appendix D.I for details).
M can be combined with any statistical model that satisfies Assumption 3 For example, under mild assumptions on the unknown dynamics, Gaussian Processes (GP) models (Rasmussen 2003) can be prov- ably calibrated under some regularity assumptions on the dynamics (Srinivas et al. 2010 Yadkori & Szepesvafi, 2011). Neural Network (NN) models (Anthony & Bartlett 2009) such as Probabilistic and De- terministic Ensembles (Lakshminarayaaan et al. [011]][hua Chua et al. 2018 are more scalable, and even though they are not calibrated by default, their empirical recalibration possible (Kuleshov et al.) 2018). Finally, using such differentiable dynamics models and parameterizing TT(-) and the auxiliary policy selecting ʃ via separate neural networks, allows for optimizing Eq. 5 by using gradient-based approaches (sce Appendix D
Remark 2. M UCRL uses an optimistic upper- confr-cco bound strategy similarly to single-agent model- based coppr-coofieence algorithms (Chowdhury & Gopalan) 2019 Kakade et al., 2020, Curi et al. 2020
Sessa el al.) 2022), however, il addresses several importanl differences: (i) dynamics and rewards explicilly depend on the states of all agents through the mean-field distribution; (ūi each agent's 's action depends on the mean-field distribution and the evolution of the mean-field trajectory is induced by the collection of actions taken by the agents; (tii) crucially, -UCRLL employs the principle that each agent in the system executes the same policy selected by the representative agent; Moreover, we note that single-agent MBRL algorithms are inherently unsuitable for solving the Mean Field Control problem.
4 Theoretical Analysis
We analyze our approach using the standard notion of curulaativ regret. measures the difference in the cumulative expected reward between the socially optimal policy x¹ (from Eq. (4) and the individual

---

Published in Transactions on Machine Learning Research (05/2023)
policies selected by M'-UCRL in every episode: Ry -i=(((()) -(m))) We also use r. to denote the regret incurred during episode I, i.e., Tt I(x²") - I(T:). We aim to show that M UC-U achieves sublinear cumulative regret, ie., limr H = 0. This implies that as the number of episodes increases, the performance of the selected policies converges to that of the optimal policy: J(m,) + J(x=). We defer all the proofs from this section to Appendix B and only highlight the main contributions of our work.
In general, the speed of convergence of M UCRL will depend on the difficulty of estimating f. For more complex functions, we expect our models to require more observations to achieve close approximation of the underlying dynamics. Consequently, we use the following model-based complexity measure to quantify this aspect of the learning problem:
T
IT = max
Σ Σ
1=)_1())11
(6)
Di.....rr Dr
=ED,U..UD.
where D. is a set of possible observations in an episode for each t t= ....,, T, and 0--1(-) is the confidence estimate of the selected statistical model computed based on D₁ U.-1. We note that the analogous notion of model complexity has been recently used in single agent model-based RI. (Chowwhury & Gopalan 2019 Curi et al. 2020) and the multi-agent sctting (Sessa et al. 2022). Intuitively, Ir measures the choson model's total confidence in estimating the dynamics over episodes in the worst-case, iee. when the sets of observations Di...., Dr are the least informative about the unknown dynamics f. For statistical models that fail to learn and generalize, Ir may grow lincarly with T. On the other hand, for models that learn from 'small' number of samples, we expect 0--1(-) to shrink fast.
General Model-based Regret Bound. The following theorem bounds the cumulative regret after T episodes in terms of the model complexity mcasure IT defined in Eq. 6
Theorem 1. Under Assumptions to 4 let Lr-1 = + 2(1 + I.m) [L, + -Trl and let ju, E P(S) for all I, h 0. Then for all T and fixee constant H > 0, with probability least 1 6, the regret of -CCL is at most:
Ry - (1=
+ L H-1 WHOTIT)
The obtained regret bound explicitly depends on constant factors that describe the environment such as the episode horizon H and the Lipschitz constants. The dependence on the size of the problem, iee. p and g, is implicit in 3r and Ir and specific to the statistical model used in the algorithm. We provide more explicit dependencies for Gaussian Processes below and in Appendix C.1 We also note that 37 is a function of 8 (see Assumption 3). When polices are selected according to Eq. 5 at every episode, our result implies that the obtained performance I(TT,) eventually converges to the optimal one I(xx") if the joint dependence on the model-dependent quantitics IT and 3r scales as O(VT)
In comparison to other model-based algorithms, both -URRL and H-UCRL (Curi et al. 2020 achieve O O ((HHTII regret while H-MARL [Sessa et al. 2022) achieves O(N H/2 H/NN//2/** /TTTIT). The additional factor of ((N"/2) comes from the fact that H-MARL considers separate agents with individual action spaces while M UC-RL optimises for a representative agent. instead. In terms of tightness of our results, we expect that the dependency on O(VTIT) in the regret bound is unaviidabe. SSarlett et al. (2017) showed that this is lower bound for the kernelized bandit problem for specific kernels and using Gaussian Process. This result relates to the special case of MFC when H = 1 and using Gaussian Process statistical model to estimate the transition dynamics. The dependency on H3, however, could be improved under more restrictive assumptions.
In our theoretical analysis, we address non-trivial challenges that arise from considering an infinite number of agents. In particular, we first translate the Mean Field Control to non-standard MDP with the dis- tributional state space P(S) and action space II of all admissible policies (for details, see Appendix B.2 This enables us to use the upper-confidence bound principle with the assumption of well-calibrated models to show the following bound on the episodic regret:
rt S 2L, (1 + Lm)
Wi Wa Pc,b):
7

---

Published in Transactions on Machine Learning Research (05/2023)
11
1.0
10
10
0.35
9
0.8
0.30
8
8-
7
0.25
0.6
6
0.20
5
0.4
0.15
3
0.10
2
0.2
0.05
1
0
0.0
0
0.00
0
6 7
10
11
3
4
5
10 11
(a)
(b)
Figure 1:
Fig. la shows the initial mean-field distribution M4.0 for cach episode h. Fig. 1b shows the
mean-field distribution at the end of an episode using a policy that selects actions uniformly at random.
where Met and ji.h are the mean-field distributions when the policy n is deployed in (i) the true multi-agent system and (ii) the system with dynamics given by the transition function ft selected by the oracle corre- sponding to Tr. The main thcoretical contribution our work provided in Lemma (see Appendix [B3] which shows the following bound on the Wasserstein-1 distance between the two mean-field distributions:
1-1 h-1 / 10o-1 Tels, Ped), (.d.) (d.)
(P 28
Σ
This shows that the deviation between the used Pe,h and μ4, is bounded by the uncertainty of the statistical model integrated over Mr,.- As the model's uncertainty decreases, the set F. shrinks towards the true dynamics f and the dynamics f selected in Eq. (5) is restricted to be close to f.
Next, we show an example of how the previously obtained regret bounds can be instantiated for particular models. To do 80, we consider the case of Gaussian Process models.
Bounds for Gaussian Process (GP) Models. GP models are frequently used to model unknown dynamics in the model-basod litcrature (see, o.g., Srinivas et al. (2010); Chowdhury & Gopalan (2019]; Curi et al. 20200;; Sessa et al. (202))). ). They can successfully distinguish between model''s and noise uncertainty, while their expressiveness follows from different kernel functions that can be used to model different types of correlation in data.
Under some regularity assumptions on f, i.e., when the true f has a bounded norm in the Reproducing Kernel Hilbert Space (RKHS) induced by the GP kernel function, these models provably satisfy Assump- tion 3 Moreover, we formally show in Appendix [ that for such GP statistical model, we can bound lT by the GP marimum mutual information (MMI). Similarly, conclude that sublinearity of our regret depends on the MMT rates. Srinivas et al. (2010) obtain sublinear (in the number of episodes T) upper bounds for this quantity in case of the most commonly-used kernels, such as the linear, squared-exxooeettial) or Matérn kernels, while Krause & Ong (2011) prove sublinear bounds for certain composite kernels.
can express 3r via. the same quantity, and
Finally, we note that the aforementioned works consider kernels defined on Euclidean spaces while our input space for namely Z = S A x P(S), includes the space of probability densities P(S). Providing bounds the GP maximum mutual information capacity in case of kernels defined on probability spaces is an interesting dircction for future work.

---

Published in Transactions on Machine Learning Research (05/2023)
11
80
10
0.014
75
000012
7-
Reward
70
0.010
40
MB-UCAL
4
0.008
30
BPTT
3
20
0.006
10
0
5
10 15
20
25
30
0
3
4
5
7 8 10
11
Episode Number
(a)
(b)
Figure 2: Convergence of the M UCRL algorithm under system dynamics given by Eq. Fig. 2a shows the rewards defined in Eq. (4) achieved by M²-UCRL over 30 episodes. Confidence bounds show the minimum and maximum over 10 independent experiments. The BPTT line corresponds to the rewards achieved under known dynamics and considered to be optimal in the environment. Fig, 2b shows the close to uniform mean-field distribution at the end of the episode, h 20, for a policy optimised by UCRL.
5 Experiments
In this section, we demonstrate the performance of the M algorithm on the exploration via entropy macimization problem introduced by Geist. et al. (2021 and used as a benchmark problem in recent survey on Mean-Field Games [Laurière et al. 2022). Different from previous works, we formulate the problem in the continuous state and action spaces as follows. The state-space of the model is described by 2-dimensional space in [0, 11]2 which is split into 4 equal-sized rooms separated by unit-sized walls with one corridor connecting neighboring rooms (See Fig. Tab. Agents are free to move around within the area and are stopped by the walls if they try to move through them. Each episode consists of 21 steps starting from = and in each time-step the representative agent chooses its actions from the action space = [0, 1Ja. The dynamics of the system in Eq. (1) are of the following form
f(1 dr (r, = 81,5 + Gt.b:
(7)
and the additive noise is Gaussian with zero mean, variance o² and independent dimensions, i.e., W N(O, o( for all t and where In is the 2 x 2 unit matrix. The individual reward function defined as r(s, a; ()) - log(u(s)) meaning that the representative agent aims avoid crowded areas. On the whole population's level, the average one-step reward is given by Es~zl- [o)([((s)) ( = p(s) (s) log(u(s)), maximizing the average reward is equivalent of maximizing the population's entropy. The population is initially condensed in the bottom-left corner (as shown in Fig. Ia] in unit square, therefore, policy is optimal if it disperses the population quickly to reach uniform distribution.
We parameterize our policy with a Neural Network and use a Deep Ensemble model ([akshminarayyaaa) et al. 2017) for estimating the system dynamics. To represent the mean-field distribution, we discretize the state-space to unit squares and assign each coll the probability of the representative agent is inside of it. We provide further details on the implementation in Appendix D Additionally, we ran experiments with Gaussian Process models for the swarm motion problem to showease the flexibility of design choices in MURLL Details and results described in Appendix E.2
Laurière et al. (2022) shows that exploration is not trivial in the discrete state and action space equivalent of this environment by considering the uniform random policy which fails to reach the uniform distribution within the episode. This result holds in the continuous environment as well as shown by Fig. lb
Results. Due the lack of an analytical solution to the problem, we compare the rewards achieved by MU-CCRL in the policy roll-outs (i.e., the interactions with the environment) to the rewards achieved by

---

Published in Transactions on Machine Learning Research (05/2023)
the policy optimised via the denote this policy by BPTT and find it to be adequate since when it is used, the population's 's entropy quickly reaches the maximum achievable corresponding to a uniform distribution. We provide further analysis of this policy in Appendix D.4
through time (BPTT) algorithm with known dynamics. We
Fig. 2a shows the rewards achieved by M--UCRI, over 30 episodes. The learning can be characterized by two phases; in the first 10 episodes M UCRL learns how to navigate the whole state-space, after which it reduces exploration and focuses more fine-grained improvements. Learning to navigate the environment is challenging due to the precise movements needed to enter the corridors, especially, from angle. Small changes in the action can decide whether the representative agent moves through the corridor to another room or is stopped by wall. As shown by the swift improvement in M UCRL's performance, it successfully accumulates information in the first phase of the learning process in order to have sufficienlly precise policies that can navigate the agent population through the corridors. In the second phase, it gradually refines its policy further to distribute the agents among the rooms quicker and achieve a much closer uniform distribution shown on Fig. 2b The optimised policy after 30 episodes achieves reward of 77.86 compared to 78.26 that of π BBPTT, i.e., less than 1% difference. 2
While MU-UCRL converges within 30 episodes, comparable tabular Q-learning algorithm for the Mcan- Field Control problem fails to achieve similar performance in more than 10 million training episodes in the discrete problem defined in Geist et al. (2021) and Laurière et al. (2022). This is most likely due to the large state space, the limitation of deterministic actions, and challenging exploration in the environment. We provide further details of the comparison and why Q-learning requires long training for convergence in Appendix E.
6 Conclusion
In this work, we propose the first poddll-basod algorithm for the Mcan-Ficld Control problem. The M algorithm runs in episodes, builds optimistic performance estimates, and uses them to efficiently explore the space during policy learning. We proved general regret bounds and showed how they can be specialized to Gaussian Process models. Furthermore, our experiments showcase the practicality of the algorithm and how it can be easily combined with deep Neural Networks. We believe that our approach provides an important step towards making model-based MARL algorithms more principled, scalable, and sample-efficient We see the following topics important for the further progression of the model-based mean-field learning for both the field control and mean-field game problems: 1) Planning with known dynamics to solve Eq. 5 similarly to Gu et al. (2020 Motte & Pham (2019), 2) Eliminating the need of recalibration to have Neural Networks satisfying Assumption
3) Maximum mutual information bounds for kernels defined on probability spaces.
Acknowledgments
This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation program grant agreement No 815943 and ETH Zurich Postdoctoral Fellowship 19-2 FEI-47. This publication was made possible by an ETH AT Center doctoral fellowship to Barna Pasztor.
References
Yasin Abbasi-Yadkori and Csaba Szepesvari. Regret bounds for the adaptive control of linear quadratic systems. Journal of Machine Learning Research, 19:1 26, 2011.
Mridul Agarwal, Vaneet. Aggarwal, Arnob Ghosh, and Nilay Tiwari. Reinforcement learning for mean field 2019.
game. Xi preprint
Noha Almulla, Rita Ferreira, and Diogo Gomes. Two numerical approaches to stationary mean-field games. Dynamic Games and Applications, 7(4):657-682 -82, 2017.
www.ddanncaanianaiinno.ooo aninationnmm.................... mp4) and the MS-UCRL policy (n3 ucrl_ animation_egee ceccioonn_ppddde___eeeeee episode_t.mpd) are included in the supplementary material. Episode 10 illustrate the quick learning during the exploration phase while episode 26 the best-performing one also used for Fig. 26
2 AAimations of episodes with the BPTT policy
10

---

Published in Transactions on Machine Learning Research (05/2023)
Andrea Angiuli, Jean-Pierre Fouque, and Mathieu Lauriere. Reinforcement Learning for Mean Field Games, with Applications to Economics. Xiv preprint or arXiv:2106. 13755, 2021.
Andrea Angiuli, Jean-Pierre Fouque, and Mathieu Laurière. Unified reinforcement q-learning for mean field game and control problems. Mathematics of Control, Signals, and Systems, pp. 1 55, 2022.
Martin Anthony and Peter L Bartlett. Neural network learning: Theoretical foundations. Cambridge Uni- versity Press, 2009.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Tn Advances in Neural Information Processing Systems, 2009.
Nicole Bäuerle. Mean Field Markov Decision Processes. arXiv preprint arXiv:2106.0873 2021.
Alain Bensoussan, Jens Frehse, Phillip Yam, et al. Mean field gumes and mean field type control theory, volume 101. Springer, 2013.
Ronen I Brafman and Moshe Tennenholtz. A near-optimal polynomial time algorithm for learning in certain classes of stochastic games. Artificial Intelligence, 121(1-2):31-47, 2000.
Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for ncar-optimal reinforcement learning. Journal of Machine Learning Research, 3:213 231, 2002.
Lucian Bugoniu, Robert Babuška, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews, 38(2): 156-172 172, 2008.
Pierre Cardaliaguet and Saeed Hadikhanloo. Learning in mean field games: the fictitions play. ESAIM: Control, Optimisation and Calculus of Variations, (3(2):569 591, 2017.
René Carmona and François Delaruc. Probabilistic analysis of mean-field games. SIAM Journal Control and Optimization, 51(4):2705 2734, 2013.
René Carmona, François Delarue, et al. Probubilistic Theory of Mean Field Games with Applicalions I-II. Springer, 2018.
René Carmona, Mathieu Laurière, and Zongjun Tan. Linear-quadradt mean-field reinforcement learning: convergence of policy gradient methods. arXiv preprint arXiv:1910 04295, 2019a.
René Carmona, Mathieu Laurière, and Zongjun Tan. Model-free mean-field reinforcement learning: mean- field mdp and mean-field qlearning. orXiv preprint arXiv:1910. 12802, 2019b.
René Carmona, Kenza Hamidouche, Mathieu LauriÉre, and Zongjun Tan. Lincar-quadratic zero-sum mean- field type games: Optimality conditions and policy optimization. Journal of Dynamics and Games. 2021, Volume 8, Pages 403-443, 8(4):403, 2021. ISSN 21646074.
Cristiano Castelfranchi. The theory of social functions: challenges for computational social science and multi-agent learning. Cognitive Systems Research, pp. 5-38, 2001.
Minshuo Chen, Yan Li, Ethan Wang. Zhuoran Yang, Zhaoran Wang, and Tuo Zhao. Pessimism Moets Invari- ance: Provably Efficient Offline Mean-Field Multi-Agent RL. Advances in Neural Information Processing Systems, 34:17913 94:17913-17926, 12 2021.
Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning, 2017.
Sayak Ray Chowdhury and Aditya Gopalan. Online learning in kernelized Markov decision processes. Proceedings of Machine Learning Research, volume 89, pp. 3197-3206, 2019.
11

---

Published in Transactions on Machine Learning Research (05/2023)
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Newral Information Processing Systems, pp. 4754 4763, 2018.
Scbastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient model-basod reinforcement learning through optimistic policy search and planning. In Advances in Neural Information Processing Systems, 2020.
Omar Darwiche Domingues, Pierre Ménard, Malteo Pirolta, Emilie Kaulmann, and Michal Valko. Regret bounds for pernel-based reinforcement learning. arXiv preprint arXiv:2004.05599 2020.
Audrey Durand, Odalric-Ambrym Maillard, and Joelle Pineau. Streaming kernel regression with provably adaptive mean, variance, and regularization. The Journal of Machine Learning Research, 2018.
Yonathan Efroni, Nadav Merlis, Mohammad Ghavamzadch, and Shie Mannor. Tight regret bounds for model-based reinforcement learning with greedy policies. In Advances in Newral Information Processing Systems, pp. -22244, 2019.
S. El-Tantawy, B. Abdulhai, and H. Abdelgawad. Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): Methodology and large-scale application on downtown toronto. IEEE Transactions on Intelligent Trunsportation Systems, 14(3):1140 1150, 2013.
Romuald Elie, Julien Pérolat, Mathieu Laurière, Matthieu Geist, and Olivier Pietquin. On the convergence of model free learning in mean field games. AAAI Conference on Artificial Intelligence, 34(05):7143 (14-)):714377100 2020.
Zuyue Fu, Zhuoran Yang, Yongxin Chen, and Zhaoran Wang. Actor-critic provably finds nash equilibria of quadar-q mean-field games. In International Conference on Learning Representations. OpenRe- view.net, 2020.
Nicolas Gast, Bruno Gaujal, and Jean- Le Boudec. Mean field for Markov decision processes: from discrete to continuous optimization. IEEE Transactions on Automatic Control, pp. 2266-2288 2012.
Matthieu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Olivier Bachem, Rémi Munos, and Olivier Pietquin. Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint. Pro- ceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 1:489-497, 6 2021. ISSN 15582914
Diogo Gomes et al. Mean field games models a brief survey. Dynamic Gomes and Applications, 4(2): 110- 154, 2014.
Haotian Gu, Xin Guo, Xiaoli Wei, and Renyuan Xu. Dynamic programming principles for mean-field controls with learning, arXiv preprint arXiv:1911.07311 2020.
Haotian Gu, Xin Guo, Xiaoli Wei, and Renyuan Xu. Mean-field controls with q- gelennning for cooperative mark: Convergence and complexity analysis. SIAM Journal on Mathemalics of Data Science, 3(4):1168 1196, 2021.
Olivier Guéant, Jean-Michel Lasry, and Pierre-Louis Lions. Mean field games and applications. In Paris- Princeton lectures on mathematical finance 2010, pp. 205 266. Springer, 2011.
Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning mean-field games. Advances in Neural Information Processing Systems, volume 32, pp. 4966 4976, 2019.
Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. A general framework for learning mcan-field games. arXiv preprinl arXio:2003.061 06069, 2020.
Pablo Hernandez- Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. arXic preprint 100.0000000 2017.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Mulli-Agent Systems, 33(6):750- -97, 2019.
Minyi Huang, Roland P Malhamné, Peter E Caines, et al. Lange population stochastic dynamic games: closed- loop mckean-vlasov systems and the nash certainty equivalence principle. Communications in Information & Systems, 6(3):221-252, 2006.
Minyi Huang, Peter Caincs, and Roland P Malhamé. Large-population cost-coupled lag problems with nonuniform agents: individual-mass behavior and decentralized e-equilibria. IEEE Trunsactions on Aulo- matic Control, 52(9):1500-1577 2007.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 2010.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with lincar function approximation. In Conference on Learning Theory, pp. 2137-2143, 2020.
Sham Kakado, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Information theo- retic regret bounds for online nonlinear control. arXiv preprint wrXx:2200.12246, 2020.
Leonid Kantorovich and Gennady S. Rubinstein. On space of totally additive functions. Vestnik Levingrad. Univ, 13:52-59. 1958.
Andreas Krause and Cheng Soon Ong- Contextual Gaussian process bandit. optimization. Advances in Newrul Information Processing Systems, 2011.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, pp. 7806, 2018.
Daniel Lacker. Limit theory for controlled mekean-vlasov dynamics. SIAM Journal on Control and Opti- mization, 55(3):1641 1672, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- lainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.
Jean-Michel Lasry and Pierre-Louis Lions. Jeux à champ moyen. i le cas stationnaire. Comples Rendus Mathématique, 343(9):619-66) 625, 2006a.
Jean-Michel Lasry and Pierre- Louis Lions. Jeux à champ moyen. ii horizon fini et contrôle optimal. Comptes Rendus Mathématique, 343(10):679 684, 2006b.
Mathieu Laurière, Sarah Perrin, Matthieu Geist, and Olivier Pietquin. Learning Mean Field Games: Survey. arXiv preprint 22Xi::205..1944// 2022.
J. W. Lee, J. Park, J. O, J. Lee, and E. Hong. A multiagent approach to q-learning for daily stock trading, IEEE Transactions Systems, Man, and Cybernetics Part A: Systems and Humans, pp. 864 877, 2007.
Jae Won Lee, Byoung- Tak Zhang, et al. Stock trading system using reinforcement learning with cooperative agents. In International Conference on Machine Learning, pp. 451-458, 2002.
Charles-Albert Lehalle and Charafeddine Mouzouni. A mean field game of portfolio trading and its conse- quences on perceived correlations. Xrxiv preprint 10:i::990000000 2019.
Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent rein- forcement learning in sequential social dilemmas. In Conference on Autonomous Agents and MultiAgent Systems, pp. 464-473, 2017.
Yan Li, Lingxiao Wang, Jiachen Yang, Ethan Wang, Zhaoran Wang, Tuo Zhao, and Hongyuan Zha. Permu- tation Invariant Policy Optimization for Mean-Fied Field Multi-Agent Reinforcement Learning: A Principled A Approach. Xixiv preprint ar 200:0000000,,,8 2021.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Teodor Mihai Moldovan, Sergey Levine, Michael Jordan, and Picter Abbeel. Optimism-driivn exploration for nonlinear systems. In JEEF International Conference on Robolics and Aulomation, 2015.
Médéric Motte and Huyên Pham. Mcan-field Markov decision processes with common noise and open-loop controls. Xiv preprind or urXi::112..0791 078883, 2019.
Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybermetics, 50(9): 3826-3839, 2020.
Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. arXiv preprint arXiv:1908 wrXi::1908.03633, 2019.
Jakub Pachocki, Greg Brockman, Jonathan Raiman, Susan Zhang, Henrique Pondé, Jie Tang, Filip Wolski, Christy Dennison, Rafal Jozefowicz, Przemyslaw Debiak, al. Openai five. URL https:/ /blog./logg.. openai. com/openai-five, 2018.
Victor M. Panarctos and Yoav Zemel. Statistical Aspects of Wasserstein Distances. Annual Review of Statistics and Its Application, 6(1):405-431. 2019. ISSN 2326-8298.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning, 2003.
Jonathan Scarlett, Iija Bogunovic, and Volkan Cevher. Lower bounds on regret for noisy Gaussian process bandit optimization. In Salyen Kale and Ohad Shamir (eds.), Proceedinggs of the 2017 Conference on Learning Theory, volume 65 of Proceedings of Machine Learning Research, pp. 1723 1742. PMLR, 07-10 Jul 2017.
Pier Giuseppe Sessa, Maryam Kamgarpour, and Andreas Krause. Efficient Model-based Multi-agent Rein- forcement Learning via Optimistic Equilibrium Computation, 2022. ISSN 2640-3498.
Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint orXiv:1610.0022 032295, 2016.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference Machine Learning, 2010.
pp.
Ingo Steinwart and Christmann Andreas. Support vector machines. In Climate Change 2013 The Physical Science Basis. Springer, 2008. ISBN 978-0-387-77241-7.
Jayakumar Subramanian and Aditya Mahajan. Reinforcement learning in stationary mean-field games. In International Conference on Aulonomous Agents and MulliAgend Systems, pp. 251-259, 2019.
Oriol Vinyals, Igor Babuschkin, Wojeicch M Czarnocki, Michaöl Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-33) 354, 2019.
Lingxiao Wang, Zhnoran Yang, and Zhaoran Wang. Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning. In International Conference on Machine Learning, pp. 10092-10103, 2020.
Weichen Wang, Jiequn Han, Zhuoran Yang, and Zhaoran Wang. Global Convergence of Policy Gradient for Quaera-Qe Mean-Field Control/Gama in Continuous Time, 2021. ISSN 2640-3498.
Pierre Weiss. L'hypothèsc du champ moléculaire la propriété ferromagnétique. J. Phys. Theor. Appl., 6 (1):661 690, 1907.
Jiachen Yang, Xiaojing Ye, Rakshit Trivedi, Huan Xu, and Hongyuan Zha. Learning deep mean field games for modeling large population behavior. International Conference on Learning Representations, pp. 1-15, 2017.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning, pp- 3571-5580, 2018.
Huibing Yin, Prashant G Mehta, Sean P Meyn, and Uday V Shanbhag. Learning in mean-field oscillator games. In 49th TEEE Conference Decision and Control (CDC), pp- 3125 3125-3132, 2010.
Huibing Yin, Prashant G Mehta, Sean P Meyn, and Uday V Shanbhag. Learning in mean-field games. IREE Transactions on Automatic Control, 59(3):629 644, 2013.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. aixiv preprint 10066, 2019.
15

---

Published in Transactions on Machine Learning Research (05/2023)
A Preliminaries on the Wasserstein Distance
In this section, we provide brief overview on the Wasserstein distance and its properties used in the proof of our general regret bound in Theorem [
Definition (Wasserstein Distance). Let and be two probability distributions on R then for e (1, the Wasserstein dislance of order k. is defined by
Jaxn f | ull dr(2,, 1/2
[inf
Walu, v)
where the infimum is taken over all joint distributions y with marginals p.v. v. For k 1, a stondard dual formulation derived by Kantorowich & Rubinstein (1588) is given by:
Wi(u,)) = sup IEF {
f(x)d(u- v)(x))
(8)
where F= : such that If(x) f(y)1 S | Vx,y E R Both formulations above gen- eralises to laws defined on more general space, e.g., complete and seporable metric spaces and infinile- dimensional function spaces such as 10]
The Wasserstein distance is usually interpreted the minimum transportation distance between the distri- butions and For random variables X μ and Y v, use the notations W&(X, Y) and (a v) interchangeably. Assuming that (x Y) is finite, i.e., |]]|X]|] + E < oo, the distance obeys the following properties Panarctos & Zemel 2019):
For m S n, by Jensen's Incquality, we have:
W. (x,(X & (XX,Y
(9)
For a E R, it holds
Wa(oX,aY) (x.W((X.))
(10)
For æ e R2, it holds
W& + v, Y x) Wx(,YY)
(11)
For E RP, have that
- E W
(12)
Corollary 1. Let X,Y be independent identically distributed random variables on R with B|||X|| and v. y E IRP. Then,
<
00
Wi(X + x, + -
Proof.
W + +) = Y.
+
By Eq. 11
=|x - -1r-y+E[X]] E W)(X,
By Eq. (22
= | - =
X and Y are i.i.d.
Therefore,
WXX +x, 9) < W2(X + + + yll2
By Eg. 99
16

---

Published in Transactions on Machine Learning Research (05/2023)
B Regret Bound Proof
In the following subsections, present the theoretical analysis that leads to the results of Theorem |
B.1 Measuring performance under arbitrary transition function
Before we start proving the main theorem, introduce the following notation of J(TT, to measure the performance of policy under the transition function f:
H Σ
[( 1] E
r(Sh Th Tine))
(13a)
8.1.
0n Th (5n: (m)))
(13b)
h = f(s ar. pbn) + W/
(13c)
Aa-1 ( The f),
(13d)
where the expectation in Eq. (3aa] is taken with respect to initial state distribution μi and noise in transitions. We note that rewrite the optimization problems in Eq. (4) and Eq. (5) such that n arg men [( arg max m(T and π. arg max maxjer.e I[T, f), respectively.
B.2 Measure Formulation
First, we provide the proof of Lemma Then, we reformulate the optimization problems in Eq. and Eq. (5a) as Markov Decision Process probability measure space with transition dynamics defined in Lemma The following proof follows Lemma 2.2 in Gu et al. (2020). We are stating it here for the sake of completeness.
Proof of Lemma 4 Fix and let be bounded measurable function on S. Recall that sy denotes random variable of the representative agent's state at time and, by the definition of phs 8h Ph-
First note that
E [+((4+)()))
(ds SSee 1h+ 1 (p(s))
On the other hand, by the law of iterated conditional expectation,
Eeaci [(((#++1] Eco...eh
Eenzi (o( snl]
Lt p(s')IP(s+t E do' ds'lso....,. sn]
Egg
J'ES
=Eoo...en I ((s'))P( Tnlons (nnhnoon))saa un). + wn ds'
-Io Jees (ls')Enommnn (1 wn ds']
I WES v((0) / un (dd (s, wm ds'].
In the third equality, used the dynamics of the system Eq. I which states that 8h41 is determined by Sh, Hh as 8441 f(n an pn) + wa. The last equality follows from (n(ss) P(s E ds). Therefore,
f Mh+ (ds')
sees (0(5) / (s, ma) wh. E ds'),
17

---

Published in Transactions on Machine Learning Research (05/2023)
which implies the desired results
(ds') Nh (a. + tw ds'
JEES
Remark 2. The relationship described in Lemma is the discrele-time equivalent of the Fokker- Pokker-Planck (Kolmogorov forward) equation for the McKean- Vlasov Process. While we provided a proof for the dynamics in Eq. 1. it holds for any other functions ft E F. for t 1,2,.....
Now, we consider the objective under the true dynamics, Eq. (4. The following lemma is based on Lemma 2.1. in Gu al. (2020).
Lemma 2. For ony policy क (Too,..., TH-1),
H-1 f(mn, mn);
J(x) = Σ h=
where f is the inlegraled reward function defoned
f(u, π) - ((,,))
(14)
and ph is the mean-field distribution induced by the policy π following the dynamics defined in Eq. Proof.
2
J J(T) E Es po.
Σ r(sn TT (n (n))
Eso- rshs Tn (n(s pa), pn)
Plsn mns),
(
H
f(un,,
The expectation in the first line is taken over the random variables so, .., WH-1 that induce the state trajectory 80, 81, 8m. In particular, for fixed policy the continuous random variable 8h. is func- tion of so, (h, In the second line, we use the linearity of expectation and 8y 's independence of Wh...., The third line follows from the definition of expectation.
Note that the fixed and known initial distribution μo: the true transition function f, and the fixed policy π define the mean-field distributions ph, for all 0 < h H via Eq. (2) The fourth equality follows from Lemma i.c., (ds) = P(sn E ds].
Corollary 2. The arguments in Lemma g apply in the case when the transition function of the system is unknown but approzimated via f. We can rewrite I(T, defined in Eq. (1al as
H-1 Σ
M(. f)
f(jeer TN).
where Pe, is the mean-field distribution induced by the policy under the estimated transition function f.
18

---

holding true episode i, the following holds for Published in Transactions on Machine Learning Research (05/2023)
Corollary 3. By using the objective formulation from Lemma can restate the optimization problem
Eq. 4
max J(x+) Σ
f(uns Th)
(15)
4=0
s.t. Ph+1 D(p, Th. f).
(16)
where 4 is defined in Eq. (2 The some holds for the oplimization under any other transition function f in Eq.
Remark 3. Corollary 3 turns the optimization problem Eq. (4) into Markow Decision Process where the state space is the space of probability measures P(S) and the action space is the space of functions II = { S x P(S) + A}. This reformulation comes with a cerlain trade-off: P(S) and II are more complex than I2 and C R' but, in contrust to Eq. (4) the transition function in Eq. (16) is deterministic.
B.3 Proof of Theorem
As described in Section 3 at the beginning of episode t, the representative agent optimizes over both the set of admissible policies, II, and plausible transition functions, Ft-1, satisfying Assumptions to 3 In the following, we denote the transition function corresponding to the optimal policy by ft-1 E Ft i.c.
Tt Jt-1 = arg max (TT,))
Lemma 3. Conditioning on the event in Assumplion episodic regret:
Tt = J( (TT,) s (T., -((T))
(17)
Proof. Recall that x' denotes the socially optimal policy from Eq. (4) and let f = arg max Jef, I( , f).
The well-calibrated property from Assumption 3 implies that the true transition function is in the set of functions F,-1 meaning that the true dynamics of the system f has been considered in Eq. ( at the beginning of episode t. Then, since we provide additional flexibility to the optimization, we have
i(T',, fim1) 2 J(TT'
Using this inequality we can bound the episodic regret as follows:
Tt = J(T") J(T,)) S J(TT" fi J(TT:)
< [T f(-1) - I(T):
where the last incquality comes from the fact that - selects the policy æ, according to (Te, fi-1) arg max n. [[T,,
Lemma Under Assumplions and conditioning on that the evend defined Assumption true, it follows for all t 1:
holds
H-1 Wa (Pe.h,P
ri S ((TT,, I(TT)) S (1, + I.J
Σ
(18)
where 1t,h is the mean-field distribution observed in the true unknown environment with dynamics f as in Eq. when agents follow ne and Urt is the mean-field distribution under the approzimated dynamics fo in Eq. 13a..
Proof. By Lemma 3 we have that
rt S (1))) I(TT.)-
19

---

Published in Transactions on Machine Learning Research (05/2023)
By using Lemma Corollary and the triangle inequality get
H
H-1
[i(T,, ft-1) - J(TT))
=
f Te,b)
Σ
F(u Teh
h=0
Consider a fixed h E {0, ..., 11 13, then
[M Te,)) - fMMe,,
r(8, (e,,Ten((s PeNN De I r(s, Te,
From Eq.
14
Is r(s,, (en, Pea Pen / r(s, (ls
/ r(s, (1857 - / r(s, Te pe.)), Pe. mas
Ir(s, (LNs r(s, (an (an
(19)
+ I M (asd))))
(20)
where the last inequality follows from the triangle inequality and the inequality for the absolute value of definite integrals.
We start by considering the integrand of the term in Eq. 19
(r,8, (s, Panh) - r(s, Te, pe,h), (4,11)
I Tlrs all2 [| (e,)) - (8. + Wi (De, pt,)))
< (Ln (l|s s|l2 Wi Mc,)) + Wa
< (, + L (P.
(21)
where the inequalities follow from Assumption 2 Next, consider the function g(s) (1 (e,ns,
1
10(x) s(v)1=
|r(x; Te, (2, r(u. Te, (v, He.h).
L, Lx)
L,
$ L, + Lx)
(110 vll2 + [T, Te, (v. + Wi WH Hc,))
L,
(11:0 - vll2 + L ull + Wi(He, (e,))
[1 + Txz)
|| - vll2.
where Assumption 2 has been applied in the second and third lines. Therefore, g is Lipschitz-1 function.
20

---

Published in Transactions on Machine Learning Research (05/2023)
We bound the second term in Eq. 20 by using the previously defined function g
[s r(s, Te, He,)), Me, (Pe, He, (ds))
(1 + 1(z) / (1+1+ Lx) Te, Me Me (4,h
S L + W ( pr,)).
(22)
The last incquality comes from the Kantorovich-Rubinstcii dual formulation of the Wasscrstein-1 distance in Eq. [8) Using the inequalities in Eq. and Eq. 22 for Eq. 119 and Eq. (20). respectively, we get that
[ Te,A) f mrAA S 21, (e.h)).
Summing it up from h to h H 1, we obtain the desired result:
[i,,, - J(T.)) 2L, L.)
Σ Waljie
We note that h = 0 is removed from the sum since the initial distribution is fixed, Pú,0 Pr.0 Pos therefore Wi (P( Pr,0) = 0.
]
Lemma 5. Under Assumptions 7 to and assuming that event in Assumption 3 holds true, and for h E (I,..., H} and t 2 and fized policy We, it holds:
Wi(pe,hs 23 1
10. (s,)) Healll (ds),
(23)
bb where T--1 = b, 13
Proof. To simplify the notation when it comes to both f and fi-1, we omit the action variable as it is deter- mined via π, la and the current state, i.e., (8, = [ss n. (8,8, 1), and similarly for fo-1- Furthermore, we fix t 2 1.
First, we establish a relationship between W (fr, 11,++1) and Wi (Mr, Mr,n), i.c., the change in the Wasserstein distance between the state distributions under the true dynamics and the approximated dynam- ics fi-1. Then, will use this relationship and the fact that the initial distribution is fixed, i.c., Pe.0 f4,0: to bound the distance at time h.
First, we rewrite Wi Wi p(,+1) in terms of Te, Rr.b and Pr use the triangle inequality for the Wasserstein-1 distance, and use the Kantorovich and Rubinstein formulation from Eq. 8
Wi (7(AA11))MA41)
(Le (TT f)) < (W fc-1), (TTeee ft-1)) ( Pr,, fe-1), (Tr, f))
/ u(>') (f P()r-((s,Pe))) Ten E ds' ds') (Pr,, - (en(ds))
sup v.Lip(e)S1 rES
(24)
u(5') (1 (FIf Pe.h) + ain ds' - (s. + wen E ds') ds'))
sup Li
(25)
Next, we consider terms from Eq. 24 and Eq. 25
separately, and obtain upper- upper-bounds bounds for both of them
in terms of Wi (D (4,a) and a: Tehh
21

---

Published in Transactions on Machine Learning Research (05/2023)
Step 1: Bounding Eq. (25) First, fix such that Lip(t)= - 1. Then, consider Eq. 25
Joe olto (Tes (P[fe- (e,)) + wish ds' dsd] [P]]-P([(s,)))) (s, + we, ds je (dss))))
- -Isss JUES 2(5') ((6))(P(Fe 1 + Senh E ds' Plf(s, + wes E ds'] pe, (ds( (ds)
I
Fubini's thm.
[
$ JSeS (1.. + we fs, (r, + (c,)))
Eq.
8
| (1, [(8, (...))))))) (ds)
Corollary T
(26)
Note that the state space compact hence the first moment of PLA+1 and Mr,41 exist and ", < oo, therefore, the integration above is finite and Fubini's theorem applicable. Next, consider the following two claims.
Claim 5.1. Under Assumplion
3 and assuming that the event defined in it holds true, we have that
1 M.,)) - [(8, (e, S 28
||o--1 1 (e,a n,,,), 1e,2)l2-
Proof.
F Jt-1 (s, pe,n)) - [8, (r,n1)12
| (8, me. m. m:- (.) + mm (8., [88 (r ll2
(s, (e, m. m (8, + 11 (8, (r,h) f( (8, 11,,
16, - m. (Lnn)
m m - f(s, 11,
< 2
s.n (0, Pe,)), 11?
1=1
2,,-1 ||o--1 (8, Te, (8, 1)), (12)
where H- denotes the i-th coordinate. The first inequality follows from the triangle inequality, after which, we used the event defined in Assumption 3 and the fact that f- is chosen to satisfy the same condition. j
Claim 5.2. Under Assumption and Assumption we have that
If (v.)) - f(, Hc,)ll S Ly(11 W
Proof. From Assumption 1 and Assumption 2 it follows that
(I f(8 p1,))l2 s (l (e,r)) - Te, Heaill2 + W1 (PeNMe)))
< L1 Wi (Pe.NMe).))
Now, we can combine Clain 5.1 and Claim 5.2 to get
fl (s, ) f(s.
(27)
< f (s, (..,))
f(s, Rr + (ff(s,eeh)) pe,,le
< 28,-1 11, Pe,)), (e.) + Ly Men))
(28)
22

---

Published in Transactions on Machine Learning Research (05/2023)
Substituting back Eq.
(88 into Eq. 26 at the beginning of Step 1, we obtain:
J.es 0(6') [ 18, (s,n) Winh E ds' (P + wen E dod) pea(ds)
SES
$ S (8, - [8, (1,) (en(dss)
s L Ln)Wi((eha Me, + 2Bc-1 f
100-- Te, Pen)) (d,)
(29)
J
Therefore, we obtain an upper bound for Eq. (25
Step 2: Bounding Eq. (24
Next, we consider the other part, Eq. 24
Again let be fixed function such that Lip(v) S 1. We have:
I I o(d))((i-1 Wan E ds' (Pen(d)) - pea(dss)
JOes
=I [ v(6') (P we ds' ([S(s, + was del) (jr,)(ds) (e,((ds))
(30)
JOESS
/ u(s)) Pe, + wan (is, (da)).
(31)
SES SEES
We use Fubini's theorem again to change the order of integration, and we consider the inner integration from Eq.
30
S u(s') (P) + Wen E ds'l - (c, + wen E ds'))
< Wi Wi (1, (e.n) w f(s, (e) + w)
Eq. 8
< f (l. f(s,(( Pe.
Corollary 1
< 23 1 | (s, ne, Pe,t)ll2
Claim 5.1
Let g(8)
|| 18, n (1.h (e,)) then note that
(1+1,
1
1ø(x) g(v)l
Pe.)) |0, Te, Re,, (e,n)l2 men P..)) PeN)ll
Lo (1 + Ln)
1
10-- (z. m (z. p. Pe,)) 0--1 Te, P..))
Lo (1 + Lx)
(||= 2l1 + [|, (.. -00--1(100 T 11.
Assumption
1 + L
11 yll
Assumption
where the first incquality follows from the reverse triangle incquality. Therefore, we can bound Eq. 30
f/ / u(s')
1s (e + wi E ds' (P pt, + wan do') (Pen(ds)) - Me. (ds))
SES VES
2 / ||0r-108, me, (8. p..)): p2, (Tr, - 14,n (ds))
sES
/ JEES
-23,-1 Lo (lo(11 + (1+ L.x)
((sn(()) (ds))
2 W Pt,k)
(32)
The last inequality comes from the dual formulation of the Wasserstein-1 distance Eq. (8) and the previonsly shown fact that g is 1-Lipschitz.
23

---

Published in Transactions on Machine Learning Research (05/2023)
Next, consider the remaining term in Eq. 31 namely,
lES J.ss v(s))[[((s PLN) wan E ds' ( - pen(
Let g(s) n,t+ta Lees (( (e,)) was E ds',, then
(q(x) - -(()l
Wes o(s') (P(((,))))))))))) + wes E ds')
L Lxn
- Plf(y.PuN) wan E ds'))
1
Wi (2, + wa, Pa,)) wend
Eq. (8)
L, + L,)
1
[f(2, S(,,
Corollary [
( + lmm)
1
|| 8l1e + [T, (e,1) men pe.))12)
Assumption
+ Lm)
|| 1o--yl12 y112-
Assumplion 2
It follows that is 1-Lipschitz, and therefore
JeS / u(s))P(f(ss + wan (s')()e pe,n
WES
-T,(1 + 1ix)
(()((un(ds) - ( SES
L L Li
(33)
The first equality follows directly from the definition of the function q, while the last inequality comes from the Wasserstein Dual formulation W1 in Eq. (
Combining Step 1 and Step 2: By combining Eq.
29
Eq. 32
and Eq. 33 get
(1 1.)11 $ (1 + Lx) + 2B + W (Fe,n)e),))
+ 2 SEs
11, Tenns, (r,),, De, ( Pe, (ds).
Also, note that
||o (s, (r.))
1|001-1ss, (s, (s
0.-1 me, pe, 112
< (s) (t,,) (s, W (i ((,)) [|o,-1(s, Te,n Me,,, Me,t) < (1 + T. (1 ne,h (,, (4,)) (4.)))) 1|
where the first inequality comes from Assumption and the second from Assumption Therefore, we have the following upper-bound on Wi (0..4411
W (4,41) 2(1 + L [L
W1 (i(peM
+ 2,
(8, MEA), Men) JES
(ds).
The above inequality establishes the relationship between (P Me,)+1 and W1 (Pe, H.,.). We use the fact. that W, (74,0, Pt.o) 0 and apply this relationship repeatedly h - times to derive the following upper
24

---

Published in Transactions on Machine Learning Research (05/2023)
bound for Wi Wi
WilpenhPee, Pt,))
h-1
$ -
2 Lx)+ 28,-1Lo11
(01-1(s,T,,(s,, pe,); He,e)l2 (e,((ds)
1=0
3ES
h-1
/ JES
< 23,-1
Σ
+ 2(1 + Ln, f + 23,
Lo
(s, He, Mo,ill (e,
1m0
=:h--1
h-1
S 28
1--1
o 10-11 (1,8, pe))) 24)))) (ds).s))
t=
WES
Remark 4. Consider the setting of Lemma 5 Note that {Mr,s y is the trajectory of state distributions i.e. P(st. E A] Mx)(A) for all A C S as shown in Lemaal Therefore,
h-1
S
h-1
Σ
10t-1 (e, mea (dd)
Σ
E
.ES
i=0
h 10.- (0
=E
where žt,i (st.s, Te,((St. (e..(( (t,),,,,)) and the erpectation is taken over the initial distribution and the tran- sition stochasticity. The first equality comes from the aforementioned fact, P( E A] Me,;(A), while the second equality comes from the linearity of expectation. Therefore,
Wi Wi Pc,))) 28, 4-
E
Σ 1o
(34)
Lemma 6. Under the selling of Lemma 5 and assuming that the event in Assumption 3 holds true,
Ø-1
ri S 48 L.
HE
Σ
Proof. First, note that under the setting of Lemma 6
Lemmas
to
also hold.
r S [[(TT,, 1(-1) - I(TT)))
Eq. 17
H-1
2L, Ln) Σ
Wi WPe Penh)
Eq. 18
h=1
H
< 44, Lr + La)
Li
E
o (zw)l2
Eq. 34
h=1
H
A 44, Lx)
HE
Σ
(2
1=0
In the last inequality, used the fact that Tu hence H-1 - Furthermore, || 2 0 for any input combination, hence Ed=1 ( En- 12,,11
Now, we ready to prove Theorem 1
25

---

Published in Transactions on Machine Learning Research (05/2023)
Proof of Theorem Cauchy-Schwaad ineq.
R2
r1
1=1
T
r
1=1
T
16p (1 + L L H
2 2
oo (zre)l2
Lemma
$ 116pI L + -2) 21 2 H
Σ
|0 1 (2wx)12
11
$ 116pT Lx) LT - 2 H
| (2,,)11)
Jenson's inoq.
< 116pT + + > 2H H IT.
The expectations in the expressions above are taken over the initial state and transition noises. For the second inoquality, we used that $2-1 non-dccreasing in by Assumption 3 while the last inoquality follows
from Σ E [ < Ir for all 24, E Z for t = 1,..., T and O...., H 1. Taking square root of both sides yields the desired result.
C Gaussian Processes
In this section, we specialize our main theorem, Theorem to Gaussian Process models.
C.1 Gaussian Process
The main step of relating Theorem 1 to the Gaussian Process models to use a Gaussian Process to approximale the unknown dynamics function f. In particular, we select GP with prior GP(O, k) where k is positive semi-definite kernel on [p] where [p] {4..... PP} and e [p] specifies the index of the output dimension. The posterior of this Gaussian Process is calculated under the assumption that the observed noise, ÊiA. $1,4+1 f(2,,,, is drawn independently from AI,,AI,) for all and h. Here À is free-paramete that does not necessarily depend on the distribution of wi,h for any t and h. Then, at the beginning of episode t, the posterior mean and variance functions conditioned on Dj U... U D obtain a closed-form solution (Rasmussen. 2003):
m ki
k. k2-1 A4-1)) U1-1, o2-1(2,5) ((, (z,j)) k. k,-1 (K,1 + A1-11) -M1-))-1s-116-p)) 1=,1(,,)),
(35a)
(35b)
where I,- E ((=-)))p(tt-)) is the identity matrix and
1.H.p y1-1 = [[8 1, E ((t-1)Hp
(x) = (FaA.)))) (t
K,- [M((E,,n))) 1), (Zv , 1- Rdt ((--1))x((-1111
where [s denotes the 1-th clement of the vector 8i We use the following notation for the mean and variance vectors m2)1(2) = [m--1(2; (2, 1)..... m......m-12pppp m=-1(z,p) and of = [o (2, 1).... oi-(2, p)] for all z in Z. In particular, choose À pH.
26

---

Published in Transactions on Machine Learning Research (05/2023)
C.2 Assumptions and Relevant Results
We make the following assumptions on the semi-definite kernel l and the true dynamics f of the system. Assumption 5. The unknown function f belongs to the Reproducing Kernel Hilbert Space (RKHS) of positive semi-definite kernel k. (Z pl) (Z x [pl) + I, and has bounded RKHS norm, i.e., llfll VJ, < By where (: is the inner product of the RKHS and By is fixed known posilice constont. Furthermore, assume that Wt.h is r-sub-Gaussiia for all and h.
This assumption is standard when Gaussian Process models are used (e.g., Srinivas al. & Gopalan 2019) Curi et al. (220
2010 Chowdhury
Assumption 6. The posilive semiidefinite kernel k is symmeetiic, continuously differentiable with bounded
derivative, and k(2, 2) S 1, E Z. We define the kernel metric as 2z(2,22) [k(2,2) k(z', 2') ) (2(2,- 2²) and assume that it is Lipschitz-continnet Wi(..' and La is some positive constant.
i.e., (z 22) S Lad(z, z²) where d(z, == ||s 2'112+11 ||a a 11
We note that the previous assumption is very mild since it holds for the most commonly used kernel functions. Assumption 5 and Assumption also imply Assumption [ which states that the unknown function f is Lipschitz-contimous,, i.c., [f(z) f(2')ll2 I L,d(2, z') where Ly is a positive constant (soc Corollary 4.36
in Steinwart & Audreas 2008
First, we show that under Assumption 5 the Gaussian Process conditioned on D1 UU..UU, is calibrated for all t 1. Our argument follows from the following lemma:
Lemma (Concentration of an RKHS member, Lemma in Chowdhury Gopalan (2019) ). Let k X' X' - IR be symmetric, positive semi-definile kernel and : + R. be member of the RKHS Hx(())
of real-valued functions X with kernel k. Let (21)121 and {e1121 is conditionally R-sub- Gaau-GGassinnn for positive constant R, i.c., VI M Fr-1] S Sxxp
A² 2
where Ft-1 is the a-algebra generated by (Iis es) ( and 0t. Let {yee h221 be a sequence of noisy observations the query points {2r)21, where y/+ = f(xt) + Et. For > and X, let
(4-1(x) == (1-1(1) + 21) YY-1,
0 (x) := k(1, - 1- (x)
where Y (yi....,yh 1 denotes the vector of observations at (tt1...,.t-)} Then, for any 0 < 8 < 1 with probability at least - 6, uniformly over t 2 1, E X,
If(z) - [1fll
R V
2 log(1/6) + Σ log(1 + + (z)))( ol-((z).
We note that similar results have been shown independently in Durand et al. 2018
Before showing that the above specified Gaussian Process calibrated, we define the mazimum mudual information, Tt, to measure the maximum information gain the representative agent could possibly obtain by observing any set A C Z [p] of size t.
Definition 2 (Maximum Mutual Information). Let. f x - R be a real-valued function defined on a domain X, and a posilive integer. For each subset A C X, let yA denole noisy version of fA with PfyA)fA). The Maximum Mutual Information after noisy observations is defined as
x)f :=
max I(fA:HA), ACX:A|=
where I(fr: VA) denoles the mulual information between fA and ya
27

---

Published in Transactions on Machine Learning Research (05/2023)
Using the Maximum Mutual Information, the following corollary shows that the Gaussian Process satisfies Assumption 3 under Assumption 5
and Assumption G
Corollary 4. Under Assumption f and Assumption with probubility 1 ā for all t 0 and E 2
- m2) s & (
(37)
where
B. By 2(log(1/0) + Tpens (k; 2 x (p))).
Proof. This corollary follows from Lemma 10 in Chowdhury & Gopalan 2019
Additionally, the GP also satisfies Assumption under Assumption 5 and Assumption 6
Lemma follows
from Lemma 12 from Curi et al. (2020)
Lemma 8. For all and 2t in Z and all 2 0, we have
[Ix(2) - oz('2l1 S du (2,
where de is the kernel metric defined in Assumption 0
We note that Lemma 3 and Assumption G constant La for all 2 0.
immodiately imply that or satisfies Assumption 4 with Lipschitz
C.3 Regret Bound
Now, we are ready to show the regret bound under the choice of a Gaussian Process model.
First, we recall a partial result in Eq. (39) from Lemma 11 Chowdhury Gopalan (0099) stating that under Assumption and Assumption
Σ
Σ
(o:- s 2ep (k. 2 x [p]
(38)
where Dr is an arbitrary observation sets during episode t for t 1,2,3.....
Thcorem 2. Let = 2(1 L.) + 1bl and let Pr, E P(S) for all t, > 0. Then, under
Assumption 2 Assumption 5 and Assumption 0 and using Gaussian Process as described in Appendic C.1 for all T 2 1 and fized constant H 0, with probability at least 1 - 6, the regret of M -URL is at most:
RI s o BT + H-1 112 (TTHT( 1f x IPl)
(39)
Proof. As noted before, Assumption and Assumption i imply Assumption tion hence, by Thcorem D with probability least 8
Assumption and Assump-
Ry = 0 0 B1 Lz) H1 H3TI
T
= | Lx L
H3T max D
Σ Σ
||o-
where L,- 1+ 2(1 L,) [Lg + Lo). Then we can substitute Eq.
to reach the desired result.
28

---

Published in Transactions on Machine Learning Research (05/2023)
Theorem 2 shows that the sublinearity of our regret depends on the Maximum Mutual Information rates, in particular, for sublinear MMT rate the regret is sublinear. Srinivas et al. (20T0) consider the most commonly used kernels, and obtain the following sublinear rates for D C R2. Linear ker- nels m(f. = O(dlogt), Squared Exponential kernels "(f.D) = ((loggt)++1) and Matérn kernels
Te,, D) = o(deltt+//((aatdttt))) (log() with a > 1. Furthermore, Krause Ong (2011) show the sub- lincar property of specific composite kernels. We note that this line of work provides large freedom for kernel design on Euclidean spaces, however, the state space in our problem includes the mean-field distribution space P(S), therefore, the results not readily applicable. Proving similar bounds probability measure spaces is promising direction for future works.
D Experiment Implementation
D.1 Hallucinated Control Implementation
In Section we introduced - -UCRL, an algorithm that optimizes over the set of plausible dynamics Fr-1
and admissible policies II (see Eq. 5 However, as noted in the paper, optimizing over Fi-1 is intractable in most of the cases. Tn this sections, we describe a practical and equivalent reformulation of Eq. (5) that helps parametrizing the problem and enables using gradient based optimization to find æ, at every episode t. First, we introduce an auxiliary function 1) Z + [-1, 1]" (as in Moldovan et al. (2015): Curi et al. (2020)
where p is the dimensionality of the state space S and define the following hypothetical dynamics model:
fz)1(2) 1 (z) 1e-1Ex-
(40)
Conditioned on the event from Assumption holding true, fi-1 for any n is calibrated, i.c., fi-1 E Fi-1- Furthermore, by the confidence interval in Assumption every function in Fi-1 can be written in the form of Eq. 40 i.e.,
E Bn + 1-, such that f--(2)) mm,-mm) 1
V2 e Z.
Therefore, we can reformulate Eq. function n: 2 + [-1, 1|2
5 as an optimization over the set of admissible policies II and auxiliary
H
T. = arg max
max
E
Σ 0=
r(rr Ce,h. me
(41a)
7:2 (1111
s.t. āτ = Tt (Be, (,,,)),
Lτ, = (8c,)),
(41b)
(z,,() -m-1 (Zt.n)) + + E,
(41c)
3t,h+1 = fc-1(Er,)) We,hs
(4ld)
Mr,t1 D( Re, f(-1).
(4le)
We note that for a fixed policy म. the auxiliary function 77-) can be rewritten as 77s,, a, p)) = 17s8, (s., u))
=
1188,1 11). In essence, this turns the function v7()) into an additional policy that exerts "hallucinated" control over the set of plausible models F (Curi et al. 2020)
Taking expectation in Eq. 41a still hinders the practicality of the algorithm, however, we note that Corol- lary 8 applies to Eq. 41 and the following optimization problem is equivalent,
H-1
n. arg max
max
Σ
(inn Te,h)
(42a)
HEll
mZ 1-2-(-1.1)) 1111p
s.t. f = m4 m4-1 (Zt,h) + 33-1 1
(42b)
Pu,+1 = (ii,,,,, Teh f(-1)
(42c)
We remark that the introduction of n(-) policy a simple trick used before Moldovan 2015 that suitable practical implementation of masel-based rein forcemettntt learning algortihms.
Curi al.
2020
29

---

Published in Transactions on Machine Learning Research (05/2023)
where @ is defined in Eq. 2 and 8 is defined in Eq.
14
/ (dds)r(s,
f(u, x)
Reformulating the optimization problem as in Eq. 42 and choosing parametrizable functions, e.g., Neural Networks, for æ,, 77. and the statistical estimators m and Σ,- allow using gradient ascent to find an optima for Eq. 42a). Even though running gradient ascent does not guarantee the discovery of global optimum, found in our experiments (see in Section 5) that it still provides close approximation. We provide further details on the implementation of the transition function & in Appendix D.2
D.2 Transition Function Implementation
The main challenges of implementing the mean-field distribution function & from Eq. 2 are representing the mean-field distributions and taking the integral over the state space S. For our experiments, we discretize the state-space S with 104 unit-sized cells. We denote these cells by Ciy for i,j E (o....,,0f 10} where Oip denotes the cell [i, + x li.i + 1). Note that cells corresponding to walls (see Fig. lal. e.g., Cs,0, Ca,1+ or Cas, are not part the state-space. For each episode and time-step h, the i.j entry of Pa,a represents the probability that the representative agent lies that cell, i.e., [,n,)] = P(st, E i + 1) x b + 1)]. Furthermore, we denote the middle of each cell C.j by Cij, i.e, GJj
0.5, 0.5).
The transition function & is then implemented follows, for every i,j {O..... .... 10) (for which Ciy does not correspond to a wall), every episode i = 1,2,.... and every time-step h = O..... 19
(Pentileg = Σ
+ wie E Caglluanka
We assume that the noise term Winh is Gaussian with mean and o² variance and the two dimensions are independent, therefore,
P P[f(ck, (en, P(.)): PaN) + win Capll
[[s(( Pen) PLA) + 1W,,11 E [ 1)] PE)) + we, bi.j 1)) = (éli (eeu,, He,) (.,)) - (4,))2 x (Øli - ((k,,, Te, (,N),, - Φ( + (C
where @ is the cumulative distribution function of NO, o²).
In the implementation, we also redistribute some of the probability mass during transition based on the walls location. there is a wall between C,,, and f(u,,, (e,, (r,n,, Mr,n) + We,ss we set [[f(Ckk (ck, Pt,)): pe,)) + Webb E Ctyl O. Similarly, for Ct, adjacent to wall or border we in- crease P[f(ck,, Te, (CH, He,,), pr + Wa, C.,l by the amount that is zero due the adjacent wall.
D.3 Transition Model estimation
To estimate the unknown transition function f in Section 5 we use deep ensemble models (Lakshminarayanani et al. 2017). In particular, use ensemble of 10 forwadd Neural Networks (NNs) with one hidden layer of size 32 and leaky ReLu action functions. We use two output layers joined to the hidden middle layer. The first one uses lincar activation and returns the mean of the function while the second returns the estimated variance using softplus activation. We follow the optinisation procedure described in (Laksh- minarayanan et al. 2017) that minimizes the negative lig-liiel for each NN under the assumption of heteroscodastic Gaussian noise. We included the adversarial training procodure as well for robustness and smoothing. In prediction time, the ensemble estimates the mean and variance of the unknown function via the empirical mean and variance of the Neural Networks' mean outputs.
30

---

Published in Transactions on Machine Learning Research (05/2023)
11
0.10
10
0.06
10
9
0.08
8
0.05
8
7
0.04
0.06
6
5
0.03
0.04
0.02
3
2
0.02
0.01
1
1
0
0.00
0.00
0
5
6
7
8
10
11
3
4
5
10 11
(a) Mean-field distribution under the BPTT optimised pol- (b) Mean-field distribution under the BPTT optimised pol- icy at time-step
icy at time-step h = 3.
11
0.04
11
10
10
0.0110
9
8
0.03
0.010
7
0.010(
0.02
0.009!
4
3
0.01
0.009(
2
2
1
0.008
0.00
0
6 7 8 10
3 4 5
10 11
(c) Mean-field distribution under the BPTT optimised pol- (d) Mean-field distribution under the BPTT optimised pol- icy at time-step = 20.
icy at time-step I = 8.
Figure 3: Mean-field distribution following the policy optimised via BPTT at four time-steps (h = 3, 5, 20) in single policy roll-out. Arrows depict the actions taken by the agents the corresponding squares. Note that the coloring range changes with the time-steps.
D.4 Policy optimisation with known environment
We consider the policy optimised via. back-propagation through time, π BPTT reasonable point-of- reference for our convergence analysis as it successfully disperses the population in the state space quickly without any notable improvements in its strategy.
Fig. 3 shows the mean-field distribution over one episode following म BPTT 1 As shown on Fig. 3a in the first few steps π BPTT spreads the distribution evenly in the bottom-left corner, however, in the next steps shown Fig. 3bb it concentrates larger mass into the corridors to the neighbouring rooms. It is optimal choice of action because the corridors act as bottlenecks hence the more it push through once the easier it will be to achieve a uniform distribution in the latter time-steps. In the next 3 steps until h = 8, n BPTT further propagates a larger portion of the population mass to the empty top-right room. In the meanwhile, it starts to distribute the population in the top-left and bottom-right rooms as well. In the following time-steps, BPTT focuses on achieving an even distribution within the rooms without pushing
"The supplementary material includes animation of the whole episode named known knwws_dynanniceeeeccccatnnoeaa animation.npt
31

---

Published in Transactions on Machine Learning Research (05/2023)
4
Population entropy
Maximum entropy
3
Entropy
2
1
0
2
5
8
11
12
13
14
16
17
19
20
Time-step h
Figure 4: Reward and population action size per time-step in one episode. The policy followed by the agents optimised via. BPTT.
Training
U2-MF-QL-FH Stardard Evaluation
QL Extended Training
65
U2 MF-MLFFH Extended Evaluation
MD-UCRL
Reward (entipy))
55
BPTT
a5
10
9M 10M
Figure 5: Rewards achieved by 12-MF-QL-FF over million training episodes. Training Rewards show the estimated reward by U2-MF- U22MP-QLI-FH while Evaluation Rewards show the performance of the greedy policy in roll-out. The policy is evaluated after every million training episodes. The hyperparameters of the training were w = 0.7. w 0.05 and 0.15.
significant population mass through the corridors. Finally, Fig, 3d shows the mean-field distribution at the end of the episode, h = 20, when the population is distributed uniformly and maximized its entropy. We provide a further animation with every time-steps as part of the supplementary material.
Fig. shows the mean-fiell distribution entropy over the time-steps the same episode as in Fig. π BPTT
successfully increases the entropy until time-step 14 where reaches the maximum achievable by the uniform distribution and then maintains this level. The small stagnation in entropy from h h due the bottleneck of the corridors shown on Fig. 3b
32

---

Published in Transactions on Machine Learning Research (05/2023)
E Additional Experiments
E.1
Comparison to model-free reinforcement learning
M UCRL is a unique algorithm in the Mean-Field Control literature because it considers continuous state and action spaces and finite time-horizon. As described in Section 2 the goal in this setting is to find an optimal policy that considers the evolution of the mean-field distribution over one episode for a fixed initial distribution. In comparison, the majority of the works consider finite spaces and infinite time-horizon in which the goal is to find the optimal stationary pair of policy and mean-field distribution. We note that this is significantly different solution concept to ours, hence, proposed algorithms are not suitable to solve the exploration via entropy maximization problem described in Section 5 To the best of our knowledge, the only algorithm that considers a finite time-horizon problem is the U2-MF-QL-FH proposed by bynniiulli et al.] (2021p. This algorithm is a Q-Learning variant that optimises the Q values via a two timescale approach. Even though, the problem formulation for U2-MF-QLE MF-QQLFFH is similar to ours described in Section 2 there are notable differences. First, U2- AL-F is proposed for finite state and action spaces, uses a tabular representation for the Q values and select deterministic actions. Second, it is model-free and assumes access to simulator of one-step transitions. On the other hand, M MCUUCRL does not rely on tabular representation, learns in continuous spaces, and have limited interactions with the enviromment to reduce costs associated with these interactions.
Due to the formulation of 12-MF-QL-FEE we ran experiments in the finite state and action space variant of the Exploration via entropy maximization problem as described in Geist et al. (2021): Laurière et al. (2022). As we demonstrate below, the deterministic policy used by U2- QL-FH 11-MM-QLL-F is a significant limitation,
therefore, we also alter its action space to be set of distributions over the 5 potential actions from which 12-MP-QL-FL chooses We call the former Standard formulation and the latter Extended.
As shown on Fig. 5 U2-MF-QL-FH converges slowly with high variance in rewards for both formulations, however, the Extended version outperforms the Standard version for both training and evaluation perfor- mance as expected. One of the main factors for slow learning is that the Q values are hard to estimate due to the large number of stale-action-time pairs and infrequent visits of many states. In particular, the Q value table has [S] |A| H 104 x 5 x 20 10, 400 entries (18, 720 for the Extended version), and states far from the bottom left corner are rarely visited because they can be reached only in the later steps of an episode. While extending the action-space to non-delerministie actions clearly improves performance, it increases significantly the tabular representation leading to even longer training required. Even after 10 million episodes, we observed large changes in the Q value table indicating that the algorithm would need further iterations, but we had to terminate it due to computational limits. These results further highlights the contribution of -CRR to the literature.
E.2 Swarm Motion Experiments
To complement experiments in Section we also implemented M with Gaussian Process models for the swarm motion problem. This sctting considers an asymptotically large (N - 2o) population of agents moving around in a space, maximizing a location-dependent reward function, and avoiding congested areas (Almulla et al. 2017).
We choose the state space S as the [0, interval with periodic boundary conditions, i.c., the unit torus, and the action space A as the compact interval [_7,7]. Each episode partitions the unit time length into H = 200 equal steps of length Ah = 1h=1/200. The unknown dynamics of the system is given by
f(s,a)= = aahh
(43)
and Wi.h ~ (0, Ah) for all t and h. We note that, for now, the dynamics do not depend on the mean-field distribution, but later on in Eq. 45). we also consider such a case. The reward function is of the form
Thhe set of distribution cousists of the following: 5 deterministic actions (4 directious plus staying idle), 4 50% 50% split between adjacent moves (up-right, right-down, down-left, left-up) and 1 distribution that assigns 25% to all 4 directions. Note that the action space the Standord formulation is subset of this action space, therefore, it is expected that the Extended version achieves better performance.
33

---

Published in Transactions on Machine Learning Research (05/2023)
Episode Rewards
-200
Episode Rewards
-50
-400-
Pewardd
-10
Reward
Nº UCRL Median
M UCAL Madian
UCRL Max interval
2000
MP UCAL Min Max interval
MC
RC
-3000
Dynamics
-3000
Dynamics
2 3
15 16
20
10 11 12
14
15 16 17 18
Number opisodas
Number of afeppoddes
(a) Rewards achieved by M
UCRL over 20 episodes.
Actions
Actions
M UCRL
Mº UCRL
5.0
2.5
Known Dynamics
2.5
Known Dynamics
-2.5
-2.5
0.0
0.4
0.6
0.8
1.
State:
State:
(b) Actions in the ergodic optimal state.
Mean-Field Distributions
Mean-Field Distributions
3-
Ma UCRL
M -UCRL
uE
Dynamics
Known Dynemics
1-
0.4
0.0
0.8
1.0
State:
(c) Mean-field distribution in the ergodic optimal state.
Figure 6: Results for dynamics model: (Left column) f(s, a) = s and (Right column) f(s, a, nt) (4 Ap. (F)) represent analytic solution to the continuous time swarm motion problem, and thus, it is not discrete-time optimal. The main discrete-time benchmark is Known Dynamics that unlike our M knows the true dynamics. In both cases (left and right), M converges in small number of episodes and discovers near-optimal policy.
r(s, a, (ss) - (nnl-in(((s)),,) where the second term penalizes large actions, the third term introduces the aversion to crowded regions and the first term defines the reward received at the position 8, namely, (ss) = 2 sin (2Ts) cos(2ns)) + + sin(sss).
The advantage of this task is that one can analytically obtain the following optimal solution in the continuous- time setting when Ah and the time horizon is infinite, i.e., H Almulla et. al. 2001
pc(s, = 2n (2Ts))
exp(2 sin (2
(44)
re(s) = Texp(2 (2 ds
where ze and pe: form an ergodic solution, i.c., He = o( Tē. f). We use the continuous time optimal solutions from Eq. (44) (where the subseript C' stands for continuous-time solutions) as a benchmark for comparison, however, we note that our time discretization introduces certain deviations. Consequently, a might no longer be optimal, and so we also compare our solution to the one obtained under the same sctup of discrete time steps and optimization procedure when the true dynamics are known (Known Dynamics). This corresponds to our main benchmark.
34

---

Published in Transactions on Machine Learning Research (05/2023)
Mean-Field Distributions with N/0.5. 0.04)
Mean-Field Distributions with Uo UÑO,
h m 16
MC
h 16
Known Dynamics
Known Dyna
h = 12
uls)
2-
h= 12
His)
h m-3
h n
h= 8
D
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
State:s
State: s
Figure 7: Mean-FFess -eend distributions over one episode with different initial distributions and dynamics given
by Eq. [43 after training has converged. (Left side) Ao ~ (N(.5, 0.04) (Right side) μa ~ ((0 1). Already
at = 16, the distribution induced by M UCRL policy nearly-matches the one when dynamics are known.
Finally, we also consider the modified swarm motion problem with dynamics given by
f(s, a)) = + a(4 A4)
(45)
The additional term that depends on the mean-field distribution models the effect of congestion on the transition. Specifically, in comparison to Eq. 443). the more congested an area is, the larger the action an agent must exert to achieve the same movement.
While swarm motion resembles the exploration via entropy maximization problem in Section 5 there are meaningful differences besides the lower dimensional state and action spaces. Namely, certain areas in the state space rewarded differently and large actions are explicitly penalized, therefore, agents not only have to explore the state space but also learn the tradc-off between areas and actions. Eq. (45) further extends the swarm motion problem with a dependency on the mean-field distribution that increases the complexity of the agent's learning problem.
Fig. G depicts the results when the true dynamics are given by Eq. 143 (Left column) and Eq. (45) (Right column). In Fig. Ba| (Left), we show the rewards achieved over 20 episodes. In Fig. 6b (Left) and Fig. 6o (Left), we show the ergodic solution of M and compare it with (#), and the solution found when dynamics are known. As expected, the introduced time-discretization results in some deviations from Eq. (44) which allow the policy found when the dynamics are known to exploit them and achieve higher reward. On the other hand, M converges a small number of episodes and drives the environment into an ergodic state with policy and mcan-ficld distribution almost identical to the solution under known dynamics (see Fig. 6b (Left) and Fig. 6c (Left)) We note that, while minimal fluctuations in the episodic rewards are present, the M - is robust in finding close to optimal solutions.
In Fig. 6 (Right column), we show the results obtained for the dynamics from Eq. 45 Even though n* and n² in Eq. 444) are no longer applicable for this problem, we include them in the figures to show the effect of the introduced congestion term. Similarly to our previous experiments, -CRL successfully converges to the solution obtained under known dynamics. Fig. 6b (Right) shows that the two policies are not completely aligned after the same number of episodes as before, which is due to the harder estimation problem and larger confidence estimates.
In the previous experiments, we selected μ² as the initial distribution to focus on finding an optimal policy function π. To evaluate the robustness of 2 -CCCCL and its ability to drive an arbitrary initial distribution towards the optimal solution, we perform additional experiments with uniform and normal initial distri- butions. Fig. shows the subsequent mean-field distributions in one episode after the algorithm reached a stable policy. We observe that M - is robust the initial distribution and swiftly drives the meel- distribution towards a steady state that maximizes the rewards over time.
35


Reviews:
[{"review_id": "I8zk3UE0BX", "reviewer": "Reviewer_SDgn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem studied in this work is of interest to the community. The manuscript is well written and the ideas introduced, while simple, appear to be novel and work sufficiently well. The supporting theory introduced is treated with rigor and its analysis appears to be correct.\n\nOverall, the biggest weakness of the manuscript is in its Experiments section. The problem considered is quite simple, even though it has been used previously as a comparison benchmark in Laurière et al., 2022. However, the case considered previously is for discrete state-action spaces, while the authors consider the continuous case, making performance comparisons difficult. The manuscript would benefit from either more complex simulations and/or the application of the authors method to the discrete case to better compare it with the previous state of the art.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": " The authors study multi-agent reinforcement learning, wherein they tackle the mean-field control problem. This formulation assumes an asymptotically infinite population of agents interacting with a common environment and aiming to collaboratively maximize a collective reward. A model-based algorithm is proposed, for which regret bounds are provided. The algorithm is tested in numerical simulations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A better experimental section would improve the manuscript. Nonetheless, I found this work well-written and deserving of publication.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yC6HPY1l2n", "reviewer": "Reviewer_6s4o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe work proposes a novel MORL approach for MFC, based on Hallucinated-Upper-Confidence Reinforcement Learning (H-UCRL), together with a general theoretical analysis on the regret bound. A practical implementation of M^3– UCRL is provided and evaluated in the exploration via entropy maximization problem.\n\n**Weaknesses:**\n\nThe main body of the paper does not currently encompass all the necessary details for a clear and reproducible work, thus some restructuring is in order. \nThere are a few aspects and statements that require clarification.\nThe theoretical and empirical evaluation are not in full alignment.\n\n(more details below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a novel model-based reinforcement learning (MORL) approach for mean-field control (MFC) entitled Model-based Multi-agent Mean-field Upper Confidence RL (M$^3$– UCRL). It provides the first general regret bounds for model-based reinforcement learning for MFC, along with a practical parametrisation  that allows the use of gradient-optimisation based techniques to solve the problem.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As far as I can tell, currently, the work follows the structure of the H-UCRL contribution, but I think it could benefit for a stronger positioning in the MFC literature. Following for example the survey of Laurière et al. (2022), you can describe/highlight in your setting details such as evolutive setting, cooperative games, social welfare optimisation. \n\nTo improve the clarity of the work, there are a couple of statements that should be strengthen throughout the work:\n- \"a novel mean-field type analysis\" - concretely specify what is the the novelty of the analysis\n- the sample-efficiency of M^3– UCRL. This property is also present in the title, however I do not find it is concretely and sufficiently supported in the work currently\n- related to the previous point, statements such as \"Our results show that M3–UCRL is capable of finding close-to-optimal policies within a few episodes, in contrast to model-free algorithms that require at least six orders of magnitude more samples.\" should also be supported by empirical evidence present in the main body of the paper. To this end, I suggest to include results from Appendix E4 in the main paper.\n\nThe theoretical analysis also includes bounds for GP models, however, as far as I can tell, there is no empirical evaluation that employs GPs. Would it be possible to extend the results with experiments that include the use of a GP model, perhaps on a different environment? (e.g., a congestion game). \n\nAdditionally, for the deep ensemble model, can you present more implementation details on the model calibration (or how Assumptions 3 and 4 are met), and how the posterior distribution over dynamical models is derived.\n\nFinally, in my opinion, appendix A should also be part of the main paper. \n", "broader_impact_concerns": "No concerns at this moment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HvV13Pevgg", "reviewer": "Reviewer_yvms", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- (Analysis) The paper provides a regret analysis for a typical optimistic RL procedure adapted to cooperative mean-field control, which is claimed to be novel in the paper (I am not familiar with the mean-field games literature, and unaware of previous comparable results);\n- (Minor. Suggested application) The paper numerically validate the proposed algorithm in a maximum state entropy exploration setting, which suggests an interesting application domain for mean-field control, although this was considered in the literature (e.g., Geist et al., 2021);\n- (Minor. Clarity) The paper is well-written and easy to follow.\n\nWeaknesses\n\n- (Interpretation of the regret) The paper does not compare the regret bound of the proposed algorithm neither to previous MFGs results nor to statistical limits of the settings, which makes hard to fully understand the value of the analysis;\n- (Discussion of the model assumptions) The regret analysis is provided under two model assumptions (smoothness of the dynamics and calibration) that looks relatively strong. Especially, it is not clear from the paper how much expressive the setting is under those assumptions;\n- (Minor. Numerical validation in toy domain) The paper stresses the practical upside of the procedure, but only provides a numerical validation in a simple domain, without comparing against any alternative approach.\n\nThis paper tackles an interesting multi-agent learning setting addressing sample efficiency and scalability through a mean-field game formulation. I have limited knowledge on the related literature, but I was not aware of a previous analysis of UCRL-based approaches in this setting. However, I think the paper is somewhat weak in terms of interpretation of the value of the provided analysis and discussion of the modeling assumptions (see comments below). If the paper could improve in those aspects (see requested changes), I believe it might be a valuable contribution within the TMLR journal.\n\nDetailed Comments\n\n(C1. Notation) The notation looks to be closer to previous works in mean-field games rather than RL, e.g., denoting the transition dynamics through a McKean-Vlasov stochastic process instead of MDP-like notation. This makes the paper slightly harder to process for an RL audience, and I would consider making a clearer connection between the proposed notation and usual MDP components.\n\n(C2. Strength of the model assumptions) In the proposed setting, the transition dynamics is modeled through a Lipschitz function with additive noise. How expressive is the setting under this assumption? What kind of applications can actually model? Especially, it does not seem to include tabular domains in general, which might limit its applicability.\n\n(C3. Model calibration) The calibration of the statistical model requires the existence of element-wise confidence bounds over then transition function. This seems to be relatively strong, especially if the environment admits states-distribution pairs that cannot be frequently reached with any policy.\n\n(C4. Regret bound interpretation) The regret upper bound provided in the paper does not include any explicit dependence with the size of the problem (from the appendix, it appears to be linear in the state dimensions p, but the action dimensions q does not appear). Moreover, the paper does not compare the obtained result with previous works. Even if this might be the first paper tackling this specific setting, can the authors at least roughly compare their results with previous analyses in finite domains?\n\n(C5. Regret lower bound) Without a lower bound setting a statistical barrier for this specific setting, it is hard to understand how far is the proposed algorithm from minimax optimality. Even without a lower bound, can the authors discuss the reported dependencies and whether they think the bound can be improved?\n\n(C6. Experiments) The paper stresses the fact that the proposed approach has practical upside, especially in combination with GPs or neural networks. However, it only reports a numerical validation in a toy domain, and does not compare against alternative approaches for mean-field games or state entropy maximization (e.g., Hazan et al., Provably efficient maximum entropy exploration, 2019).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of learning an optimal control policy in cooperative mean-field games with unknown transition dynamics. It proposes a model-based reinforcement learning approach that is an adaptation of the UCRL algorithm to this specific setting. Especially, it provides an analysis of the proposed algorithm, which results in a $O( \\sqrt{H^3 T})$ regret upper bound under smoothness and calibration assumptions over the transition dynamics. The paper further explains how the proposed algorithm can be implemented with GPs or neural networks retaining some of the theoretical guarantees. Finally, the algorithm is numerically validated in a continuous gridworld in which the rewards are the entropy of the mean-field distribution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe that the current manuscript would need the following changes in order to reach the bar for acceptance:\n1) An interpretation of the regret results in comparison to other (possibly model-free) approaches for mean-field games or against a lower bound;\n2) A thorough discussion of the model assumptions, how much is the resulting setting expressive and what kind of applications can be modeled under these assumptions.\n\nOther changes that can add value to this work are:\n- A more substantial experimental analysis of the proposed algorithm in challenging domains and against some relevant baseline;\n- A revised notation that draws a better link between the considered setting and the usual MDP setting.", "broader_impact_concerns": "This paper provides an essentially theoretical contribution. While the presented approaches might be employed in sensible applications in the future, I do not think that an impact statement is needed at this stage of development.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers find the work novel and interesting, while offering a number of detailed suggestions to improve the paper. Most of them are about improving clarity of the positioning, limitations of assumptions, technical novelty, among others. The authors provided a revised version that addresses these questions. We are happy to recommend acceptance.
Decision: Accept (poster)