Population-size-Aware Policy Optimization for Mean-Field Games
Authors: Pengdeng Li
Xinrun Wang
Shuxin Li
Hau Chan
Bo An
Venue: ICLR 2023 poster (2023)

Abstract: In this work, we attempt to bridge the two fields of finite-agent and infinite-agent games, by studying how the optimal policies of agents evolve with the number of agents (population size) in mean-field games, an agent-centric perspective in contrast to the existing works focusing typically on the convergence of the empirical distribution of the population. To this end, the premise is to obtain the optimal policies of a set of finite-agent games with different population sizes. However, either deriving the closed-form solution for each game is theoretically intractable, training a distinct policy for each game is computationally intensive, or directly applying the policy trained in a game to other games is sub-optimal. We address these challenges through the \textbf{P}opulation-size-\textbf{A}ware \textbf{P}olicy \textbf{O}ptimization (PAPO). Our contributions are three-fold. First, to efficiently generate efficient policies for games with different population sizes, we propose PAPO, which unifies two natural options (augmentation and hypernetwork) and achieves significantly better performance. PAPO consists of three components: i) the population-size encoding which transforms the original value of population size to an equivalent encoding to avoid training collapse, ii) a hypernetwork to generate a distinct policy for each game conditioned on the population size, and iii) the population size as an additional input to the generated policy. Next, we construct a multi-task-based training procedure to efficiently train the neural networks of PAPO by sampling data from multiple games with different population sizes. Finally, extensive experiments on multiple environments show the significant superiority of PAPO over baselines, and the analysis of the evolution of the generated policies further deepens our understanding of the two fields of finite-agent and infinite-agent games. 

Published as a conference paper at ICLR 2023
POPULATION-SIZE-AWANI POLICY OPTIMIZATION FOR MEAN-FIELD GAMES
Pengdeng Li' Xinrun Wang Shuxin Li' NNayyang Technological University, Singapore (pengdeng. 1i, xinrun. wang, shuxin. 1i, 1i, boan) @ntu. sg, hchan3 edu
Hau Chan
Bo An'
University of Nebraska, Lincoln, USA
ABSTRACT
In this work, we attempt to bridge the two fields of finite-agent and infinite-agent ageet games, by studying how the optimal policies of agents evolve with the number of agents (population size) in mean-field games, agent-centric perspective in con- trast to the existing works focusing typically on the convergence of the empirical distribution of the population. To this end, the premise is obtain the optimal policies of set of finite-agent games with different population sizes. However, either deriving the closed-form solution for each game is theoretically intractable, training a distinct policy for each game is computationally intensive, or directly applying the policy trained in a game other games sub-optimal. We address these challenges through the Population-size-4 ware Policy Optimization (PAPO). Our contributions are three-fold. First, efficiently generate efficient policies for games with different population sizes, we propose PAPO, which unifies two nat- ural options (augmentation and hypernetwork) and achieves significantly better performance. PAPO consists of three components: i) the population-size encoding which transforms the original value population size to an equivalent encoding to avoid training collapse, ii) hypernetwork generate a distinct policy for each game conditioned on the population size, and iii) the population size as an addi- tional input the generated policy. Next, construct multi-task task-based training procedure to efficiently train the neural networks of PAPO by sampling data from multiple games with different population sizes. Finally, extensive experiments on multiple environments show the significant superiority of PAPO over baselines, and the analysis of the evolution of the generated policies further deepens our understanding of the two fields of finite-agent and infinite-agent games.
INTRODUCTION
Games involving a finite number of agents have been ex- tensively investigated, ranging from board games such Go (Silver et al. 2016 2018), Poker (Brown& & Sandholm Moravčík al. 2017). and Chess (Campbell 2002) to real-time strategy games such as StarCraft et al. Viiyaals al. 2019) and Dota 2 (Berner et al. 2019).
PPO-Naive
PPO
30
PAPO (Ours)
2018 2019:
Approx NashConv
20
However, existing works are typically limited to a handful of agents, which hinders them from broader applications.
10
To break the curse of many agents (Wang et al.) 2020).
mean-field game (MFG) (Huang et 2006 Lasry & Li- ons 2007) introduced study the games that involve an infinite number of agents. Recently benefiting from reinforcement learning (RL) (Sutton & Barto 2018) and deep RL (Lillicrap et al. 2016 Mnih et al., (015). MFG provides versatile framework model games with large population of agents (Cui & Koeppl 2022 Fu et al. 2019
10
15
20
25
30
35
40
45
50
Population Size
Figure 1: Experiments on Taxi Match- ing environment show the failure of two naive methods and the success of PAPO. means the lower the better performance. See Sec. 5.] for details.
Guo et al. 2019 Laurière al.] 2022 Perolat al.
2021
Perrin et al. 2022 [2020] Yang et al. 2018).
"Corresponding author.

---

Published as a conference paper at ICLR 2023
Despite the successes of finite-agent games and infinite-agent games, the two fields are largely evolv- ing independently. Establishing the connection between an MFG and the corresponding finite-agent Markov (stochastic) game has been a research hotspot and it is done typically by the convergence of the empirical distribution of the population to the mean-field (Saldi et al.] 2018: Cui & Koeppl, 2022 Cui et a 2022 Fabian al. 2022]. However, few results have been achieved from an agent- centric perspective. Specifically, a fundamental question is: how do the optimal policies of agents evolve with the population size? As the population size increases, the finite-agent games approx- imate, though never equal to, their infinite-agens counterparts (Cui & Koeppl 202T] Mguni et al., 2018]. Therefore, the solutions returned by methods in agget- games should be consistent with that returned by methods in infinite-agent games. As we can never generate finite-agent games with infinite number of agents, we need to investigate the evolution of the optimal policies of agents, i.e., scaling lawe to check the consistency of the methods. However, theoretically investigating the scaling laws is infeasible, as obtaining the closed-form solutions of a set of finite-agent games is typically intractable except for some special cases (Guo & Xu 2019]. Hence, another natural ques- tion is: how to efficiently generate efficient policies for a set of finite-agent games with different population sizes? Most methods in finite-agent games can only return the solution of the game with a given number of agents (Bai & Jin| [2200] Jia et al. 2019, Littman et al. (200]. Unfortunately, the number of agents varies dramatically and rapidly in many real-world scenarios. For example, in Taxi Matching environment (Nguyen et al.) 2018 Alonso-Mora et al. 2017). the number of taxis could be several hundred in rush hour while it could be a handful at midnight. Fig. demonsttatess the failure of two naive options in this environment: i) directly apply the policy traieed in a given population size to other population sizes Paive,aive), and ii) train a policy by using the data sampled from multiple games with different population sizes (PPO). Furthermore, computing the optimal policies for games with different population sizes is computationally intensive.
In this work, we propose novel approach to efficiently generate efficient policies for games with different population sizes, and then investigate the scaling laws of the generated policies. Our main contributions are three-fold. First, we propose PAPO, which unifies two natural methods: augmenta- tion and hypernetwork, and thus, achieves better performance. Specifically, PAPO consists of three components: i) the population-size encoding which transforms the original value of population size to an equivalent encoding to avoid training collapse, ii) a hypernetwork to generate a distinct pol- icy for each game conditioned on the population size, and iii) the population size as an additional input to the generated policy. Next, to efficiently train the neural networks of PAPO, we construct a multi-task-basee training procedure where the networks are trained by using the data sampled from games with different population sizes. Finally, extensive experiments on multiple widely used game environments demonstrate the superiority of PAPO over several naive and strong baselines. Fur- thermore, with a proper similarity measure (centered kernel alignment (Kornblith et al. (211)), we show the scaling laws of the policies generated by PAPO, which deepens our understanding of the two fields of finite-agent and infinite-agent games. By establishing the state-of-the-art for bridging the two research fields, we believe that this work contributes to accelerating the research in both fields from a new and unified perspective.
2 RELATED WORKS
Our work lies in the intersection of the two research fields: learning in Markov games (MGs) and learning in mean-field games (MFGs). Numerous works have tried to study the connection between an MFG and the corresponding finite-agent MGs from a theoretical or computational viewpoint such as (Saldi al. 2018 Doncel et al. 2019 Cabannes al. 2021), name a few. The general result achieved is either that the empirical distribution of the population converges to the mean-field as the number of players goes to infinity or that the Nash equilibrium (NE) of an MFG is an approximate NE of the finite-agent game for a sufficiently large number of players, under different conditions such as the Lipschitz continuity of reward/cost and transition functions (Saldi et al.) 2018 Cui & Koeppi| [021 2022 Cui et al. 2022) or/and the convergence of the sequence of step-graphons (Cui & Koeppl 2022 Cui et al. 2022). Though the advancements in these works provide a theoretical or
IIn this work, we focus on the finite-agent Markov games sharing similar structure with the MFG, see Sec. 33 and Appendix [A. more details and discussion.
We use the term "scaling laws" to refer to the evolution of agents' optimal policies with the population size, which is different from that of (Kaplan et al. 2020). See Appendix Aomendix[A.4]foa for more detailed discussion.
2

---

Published as a conference paper at ICLR 2023
computational understanding of the connection between the two fields, very few results have been achieved from an agent-centric perspective. More precisely, we aim to answer a fundamental ques- tion: how do the optimal policies of agents evolve with the population size? Toward this direction, the most related work is (Guo & Xu 2019), which proves the convergence of the NE of the finite- agent game to that of the MFG by directly comparing the NEs. However, requires the closed-form solutions of both the finite-agent game and the MFG to be computable, where extra conditions such as convex and symmetric instantaneous cost function and cădlăg (bang-bang) controls are needed. In a more general sense, deep neural networks have been widely adopted to represent the policies of agents due to their powerful expressiveness (Laurière et al. 2022] Perolat al.. 2021| Perrin et al. 2022 Yang et al., 2018). In this sense, to our best knowledge, our work is the first attempt to bridge the two research fields of finite-agent MGs and MFGs. Specifically, we identify the critical challenges (theoretical intractabilityy computational difficulty, and sub-optimality of direct policy transfer) and propose novel algorithms and training regimes to efficiently generate efficient policies for finite-agent games with different population sizes and then investigate the scaling laws of the policies. For more discussion and related works on the two fields, see Appendices A.2 and A.3
Our work is also related to Hypernetwork (Ha et al., 2016). A Hypernetwork is neural network designed to produce the parameters of another network. It has gained popularity recently in a variety of fields such computer vision (Brock et al. [0188] Jia et al. 2016:] Littwin & Wolf 2019 Potapov jet al. 2018) and RL (Huang et al. [b22bb] Rashid et al. 2018). In (Sarafian et al.] 2021). Hypernet- work was used to map the state or task context the Q policy network parameters. Differently, in
addition to generating a distinct efficient policy for each population size by using Hypernetwork, we also investigate the scaling laws of the generated policies, which has not been done in the literature.
PRELIMINARIES
In this section, we present the game models (finite-agent Markov game and infinite-agent mean-field game) and problem statement of this work.
Finite-agent Markov Game. A Markov Game (MG) with N < oo homogeneous agents is denoted as G(N) = A.
N = (1,... NN is the set of agents. and A are respectively the shared state and action spaces. At each time step L E T T= {0, 1.... ,T), an agent i in state gf takes an action at_ Let NN denote the empirical distribution of the agent states 8; = (5),
E 1[s s), for all S. Then, = E (SS), the probability distribution over S. For agent i, given the state si, action of. and state distribution =!Y. its dynamical behavior is described by the state transition function p: S x x ASS) + A(S) and it receives reward of
with (s) 3/4
r(s ). Let n² S x T + A(A) be the policy of agent Accordingly, ा = (=*) the joint policy of all agents and π ; (TT))EN the joint policy of all agents except i. Given the S N is given by
initial joint state 8 and joint policy л, the value function for agent 1 sis
T
Vi( ; M2, ग = E Σ
r of /06 5 S ~ p.ai ~
(1)
A joint policy πt" = (T"") JEN is called a Nash policy if an agent has no incentive to unilaterally deviate from others: for any i E N. v'(s", T²) 2 max," Vi(s", " , the NASHCONV(TT) = ΣN max;; Vi(8", 43, क - Vi(s T) measures the "distance" of ι to the Nash policy. That is, π is Nash policy if NASHCONV(T))
i.*). Given a joint policy π,
O.
Mean-Field Game. A Mean-Field Game (MFG) consists of the same elements as the finite-agent MG except that N = oc, which is denoted as G(oo) = (S, A. p. r, T). In this case, instead of modeling N separate agents, it models a single representative agent and collapses all other (infinite number of) agents into the mean-field denoted by μn E A(S). As the agents are homogeneous, the index i is suppressed. Consider a policy π as before. Given some fixed mean-field (Nt)eT of the population and initial state ~ μi of the representative agent, the value function for the agent is
V(s, v, (UA)ETT) = E
r(sr, ur, me) s0 8141 P,
O
In experiments, we follow Laurrèèr et al. the state with time.
2022 to make the policy dependent on time by concatenating
3

---

Published as a conference paper at ICLR 2023
r" is called Nash policy if the representative agent has incentive to unilaterally deviate from the population (Perolat et al. 2021 Perrin et al.) 2020): V(s, (4f )ee)) 2 max; V(s, it (14) JeT), where (44 HeT is the mean mean-held of the population following x². Given policy x, the NashConv is defined as NASHCONV(n (T) = max+ ((s. st, ((T)tTT) V(s, n,, ((4))TT) with (ET)ET being the mean-field of the population following æ. Then, π is Nash policy if NASHCONV(T)) 0.
Problem Statement. Though numerous advancements have been achieved for the finite-agent MGs and infinite-agent MFGs, the two fields are largely evolving independently (we provide more de- tailed discussions on the connection between MGs and MFGs in Appendix (A.)). Bridging the two fields can contribute to accelerating the research in both fields. There are two closely coupled ques- tions: how do the optimal policies of agents evolve with the population size, i.e., scaling laws, and how to efficiently generate efficient policies for finite-agent MGs with different population sizes? Formally, let G = {G(N)... G(N). G(N) denote set of MGs, where N and N denote the minimum and maximum number of agents. Let T'v denote the Nash policy of a game G(N). Let p(N) = p( x N++) denote some measure capturing the difference between the Nash policy of G(N) and that of G(N + 1) (see Appendix AA. for more discussion on p). Then, one question is how p(N) changes with N. However, directly investigating the evolution of p(NV is infeasible, as we need to obtain the Nash policy V for each game ((N). In addition, directly applying the Nash policy x to game G(N') with Nº * N could result in worse (or arbitrarily worse) performance, i.e., large (or arbitrarily large) NASHCONY(TĀ) for G(N'), as shown in Fig. I Thus, another question is how to efficiently generate policy TN that works well for each game G(N) E G, i.e., small NASHCONY(TNN). though it may not be the Nash policy TN A z... Unfortunately, most existing methods can only return the Nash policy for the game with a given N. which hinders them from many real-world applications where the number of agents varies dramatically and rapidly, e.g., the number of taxis could be several hundred in rush hour while it could be handful at mid- night (Alonso-Mora et al. 2017). Furthermore, learning the Nash policies of all the games in G is computationally intensive. Therefore, our objective is to develop novel methods that can efficiently generate efficient policies" for the games in G and enable us to investigate the scaling laws.
4 POPULATIION-SIZE-AWW POLICY OPTIMIZATION
We start by introducing the basic structure of the policy network of a representative agent. Let me denote the agent's policy parameterized by E O. Note that can be any type of parameteriza- tion such direct policy parameterizations (Leonaroos et al. 2022) and neural networks (Agarwal cet al. 2021). In this work, we use a neural network with three layers to represent the policy, i.c., O ( 02 0z) with 0, (w,, bt. gi) denoting the vectors of weights (wi). biases (b,). and scal- ing factors (gr) of the layer < 7 < 3. In practice, O is typically trained by using RL algorithms (see S.3.F4.3for practical implementation). However, mentioned in the problem statement, either train- ing a distinct policy for each game is computationally intractable or naively applying policy trained under given N to other games is sub-optimal (as shown in Fig. D. To address these challenges, we propose new policy optimization approach, the Population- Aize---ize-aaaaee Policy Optimization (PAPO), which is shown in Fig.
4.1 POPULATION- SIZE ENCODING
As PAPO is population-siee dependent, one may choose to directly take N the input (we call the raw encoding (RE, for short)). However, this might work poorly in practice. In many real-world applications, the population size can be large and vary dramatically and rapidly, e.g., the total number of taxis in Manhattan is around 13,000 and in one day it can fluctuate from a handful at midnight to several hundred in rush hour (Alonso-Mora et al. (017). When directly taking population sizes as inputs, bigger Nss may have higher contribution to the network's 's output error and dominate the update of the network. This could degrade the performance or severely, collapse the training (as observed in experiments where the training of PAPO with RE is collapsed).
To address the aforementioned issue, we propose the population-sie size encoding to pre- pre-process N. We use binary encoding (BE, for short) to equivalently transform the original decimal representation of N into a binary vector with size k > O. Formally, given N, we obtain a vector e(N) = [e1, ... ex]
WWe call a policy an "efficient policy" if has small NashConv in the corresponding game.
4

---

Published as a conference paper at ICLR 2023
encode
N
hypernetwork
head
head
head
0
83
concut
0
02
O3
policy network
Figure 2: The neural network architecture of PAPO.
such that N - Σ 2e where e, {0, After that, the encoding further mapped to the embedding (through an embedding layer parameterized with 7)) which will be fed to the policy network and hypernetwork as shown in Fig. 2 With a slight abuse of notation, throughout this work, simply use N to represent the embedding of e(N) if it is clear from the context.
4.2 POPULATION-SIZE: SWANN-ARRREAAEREEEEEEE ARCHITECTURE
The critical insight of our approach is to leverage the population-size information to generate effi cient policies for the games in G. In view of the fact that none of the existing works has addressed similar problem as ours, we first propose two natural methods: augmentation and hypernetwork.
Augmentation. That is, we concatenate N with 9; and then pass the resulting input to the policy, A similar idea has been adopted in the literature to address the problem of policy generalization. For example, in (Perrin et al., 2022). the state is concatenated with the initial mean-field to endow the policy with the capability of generalizing across the mean-field space. In our context, by augmenting the state with N, the policy can work well across games with different N's.
Hypernetwork. That is, we generate a distinct policy for each game by mapping to the policy network parameters through hypernetwork (Ha et al., 2016) Sarafian et al. 2021). This choice is completely different from the augmentation as the game-level information (population size in our context) is fully disentangled from the policy input. More specifically, let h3 denote a hypernetwork parameterized with B. h/3 takes N as input and outputs a set parameters (weights, biases, and scaling factors) which will be reshaped to form the structure of the policy network. Formally, have 0 = h (N). In the hypernetwork, three independent heads are used to generate the parameters of the three layers of the policy network (more details of the network architecture such as the number of neurons of each layer can be found in Appendix B.I).
Although the above mative answer to the question of which one is better as they achieve the goal from different angles: augmentation uses fixed set of policy network parameters and takes elements from the cartesian product of two domains (S and 12,3,... }) as inputs while the hypernetwork generates a distinct set of policy network parameters for each game. In this work, instead of trying to answer this question, we propose a unified framework, PAPO, which encompasses the two options as two special cases. Specifically, as shown in Fig. 2 PAPO uses hypernetwork to generate the parameters of the policy network conditioned on N and also takes N as an additional input to the generated policy.
methods obtain efficient policies for the games in G, there is affir-
Intuitively, PAPO preserves the merits of the two special cases and possibly achieves better perfor- mance. On one hand, it induces a hierarchical information structure. To a certain extent, it decouples the game-level information from the policy by introducing a hypernetwork to encode the varying el- ements across different games. In the context of our work, the only difference between the games in G is the population size while other game elements such as the state and action spaces are identical. Thus, PAPO can generate distinct policy (or in other words, specialized policy) for each game conditioned on the population size. On the other hand, once the policy is generated, PAPO becomes the augmentation, which, as mentioned before, work well in the corresponding game.
Furthermore, we can investigate the scaling laws of the generated policies by studying the difference between them. Formally, let (N= and (N+1 be two policies generated by PAPO for the two games with N and N agents, respectively. Then, the difference, denoted p(N)
5

---

Published as a conference paper at ICLR 2023
P (To (N))) hN+hh()+ as a function of N, characterizes the evolution of the policies with the population size. In practice, can choose any type of p(N). In experiments, we will identify proper measure (a similarity measure) that can be used to achieve this goal.
4.3 PRACTICAL IMPLEMENTATION
Notice that PAPO is general approach. In this section, we specify the practical implementations of the modules in PAPO and construct a training procedure to efficiently train the neural networks.
Algorithmic Framework. We implement PAPO following the framework of PPO (Schulman al, 2017) as it is one of the most popular algorithms for solving decision-making problems. That is, in addition to the policy network 0 (i.e., actor), value network (i.e., critic) is used estimate the value of a state and also generated by hypernetwork. Then, the networks (hypernetworks and embedding layers) are trained with the following loss function (see Appendix B.2 for details):
T, (((8),,0)) = E. [ = h3 (N)) - e1 (sb = hag (N)) + e2 ( (T)=))
(3)
As the agents learn their policies independently, the policy learning in this work falls in the category of using independent RL to find Nash equilibrium (Ozdaglar et al. (021). Recent works (Ding et al,] 2022 Fox et al., 2021 Leonardos et al. 2022) have shown that independent policy gradient can converge to the Nash equilibrium under some conditions, which, to some extent, provides support to our work. More discussion can be found in Appendix
Training Procedure. To make PAPO capable of generating efficient policies for the games in the set G. a trivial idea is to train the networks sequentially over G, i.c., train the networks until they converge in one game and then proceed to the next. Unfortunately, such a method is computationally intensive and could suffer catastrophic forgetting. i.e., the networks focus on the learning in the current game. A more reasonable approach is to train the networks by using the data sampled from all the games (a way similar to multi-task learning (Zhang& & Yang 202T). ). Inspired by this idea, we construct multi-task-based training process to efficiently train the networks. At the beginning of an episode, a game ( is uniformly sampled from G. Then, PAPO takes N as input and generates policy which is used by the agents to interact with the environment. Finally, the experience tuples are used to train the networks. More details can be found in Appendix B.3
EXPERIMENTS
We first discuss the environments, baselines, metric used to assess the quality of learning. similarity measure used to investigate the scaling laws, and then present the results and ablation studies.
Environments. We consider the following environments which have been widely used in previous works: Exploration (Laurière et al.) 2022). Taxi Matching (Nguyen et al. 2018). and Crowd in
Circle (Perrin et al., 2020). Details of these environments can be found in Appendix C.I
Baselines. (1) PPO: the standard deep RL method. (2) AugPPO: augments the input of PPO with the population size. (3) HyperPPO: without the additional input to the generated policy. (4) PPO-Large: the number of neurons of each layer is increased such that the total number of learnable parameters similar PAPO. (5) AugPPO-Large: similar PPO-Large. The last two baselines are necessary to show that performance gain is not due to the additional number of Icarnable parameters, but due to the architecture of PAPO. addition, all the baselines are trained with the same training procedure as PAPO, which ensures fair comparison between them and PAPO. More details on baselines and respectively.
hyperparameters are given Appendix C.2 and Appendix C.3
Approximate NashConv. Computing the exact NashConv is typically intractable in complex games as it requires obtaining the exact best response (BR) policy of the representative agent. Instead, we obtain an approximate BR by training it for large enough number of episodes (1c5) and thus, obtain an approximate NashConv, which is the common choice in the literature (Laurière et al, 2022 Perrin al. 2020). More details are given in Appendix C.4 Furthermore, in Appendix D.2 we show the BR training curves to demonstrate that the BR has approximately converged, ensuring that the approximate NashCony is reasonable metric to assess the quality of learning.
Similarity Measure. Intuitively, two policies are similar means that their output features (represen- tations) of the corresponding layers (not their parameter vectors) are similar, given the same input

---

Published as a conference paper at ICLR 2023
Exploration
Taxi Matching
Crowd Circle
(A)
PPO
LargL
AugF20
AngFPOCLLrgg
HypesPPO
PAPO (Cun)
20.0
Approx. NathCres
15.0
20
7.5
10
2.5
Populiatio
Popsiation
(B)
Ingul. Raw
Hidden, Raw
Outpur, Xaw
Ingur,
Hiddon, 2it
Ourgut. Fit
1.0
1.0
1.0
0188
0.8
6.90
0.7
0.7
3cea
0.145 (0000000)
1per
(000010)))
1ges
3.00
(633
8.74 (.3
0.6
0.6
0.6
Diroo4
1.01
(000
Dapec
0:31 (100a-00
0.415 10.20 (5)
Dape:
3.009 11.66 00)
8.332 (1/40
0.5
100
200
Population
Popuiation Size
Population Size
(C)
N
X-10
Ni15
N-20
N-190
N-195
N-200
Game State Uppor-right
Bottom-left
Center
Figure 3: Experimental results. (A) Approximate NashConv versus population size. (B) CKA value versus population size. (C) UMAP embeddings some numan-understandable game states: agent on the upper-right/ootoo-lee of the map and agent on the center of the circle.
data. Particularly, the outputs of the final layer are closely related to an agent's decision-making (be- havior). Therefore, we use the centered kernel alignment (CKA) (Kornblith al. 2019) measure the similarity between two policies. More details can be found in Appendix C.5
5.1 RESULTS
In Fig. I we show the results obtained in a small-scale setting (2 < < 50) in Taxi Matching environment to illustrate our motivation. There are two most straightforward options to obtain poli- cies for different games: i) directly apply the policy trained in a given game (N = 20, marked by star) to other games, i.e., PPO-Naive, and ii) train a single policy by using the training procedure in Sec. 4.3 and then apply it to the evaluated games, i.c., PPO. At first glance, might be natural to expect PPO work well across different games when it trained by using the data sampled from

---

Published as a conference paper at ICLR 2023
different games. However, shown in the results, though it outperforms PPO-Naive, it still has large approximate NashConv when evaluating in different games. In striking contrast, PAPO works well across different games. In addition, given the same training budget, PAPO still preserves similar performance in the game (N = 20) where PPO-Naive was trained.
In Fig. we show the results obtained in a larger-scale setting (2 N < 200) in different environ- ments. From Panel A, we can draw the following conclusions. (1) PAPO significantly outperforms other naive and strong baselines in terms of approximate NashConv across different games, which demonstrates that PAPO has achieved one of our goals efficiently generating efficient policies for a set of finite- finite-agent games. (2) As two special cases of PAPO, AugPPO and HyperPPO can work well across different games. However, they are competitive. which verifies that there is no affir- mative answer to the question of which one is better (See. 4.2 and Appendix [.I].. (3) Given the same training budget, simply increasing the number of learnable parameters of the policy (PPO- Large and AugPPO-Large) or only relying on the hypernetwork to generate the policies (Hyper- PPO), though could outperform PPO, is still struggling to achieve better performance. This shows the superiority of PAPO as a unified framework which inherits the merits of the two special cases and HyperPPO) and thus, achieves better performance.
(AugPPO/AugPPO-Larr))
In Panel B, we show how the policies generated by PAPO change with the population size. From the results
see that the similarity between two policies increases with the increasing population size. In other words, with the increase in population size, the policies in the games with different population sizes tend to be similar each other, which shows convergent evolution of the policies in terms of population size. Though it is impractical to experimentally set N = oc, from the results we hypothesize that the similarity measure increases I when N + co (the finite-agent game would become an infinite-agent MFG when N = =oo). To quantitatively describe the scaling laws, we show the curves fitting the similarity, which obtained by employing the tools for curve fitting in Numpy (Harris et al. 22020). We consider a function with the form of p(N) = 1 - N S. 1. which maps N to the similarity measure. The fitted curves again verify the aforementioned conclusions. Furthermore, we compute the p-value of each parameter of the function (a and b), which shows that the curves fit well (p- < 0.05) the original CKA values. Notice that there could be different forms of p(N). For example, more general case is p(N) = 1 .v S 1. However, by computing the p-values, we found that some parameters (c and d) are less significant and hence, can be removed from p(N). We give a more detailed analysis in Appendix D.3
In addition, we observed that the variance of the similarity of the input layer is small while that of the output layer is large (especially when N is small). This coincides with the well-known conclusion in deep learning (Alzubaidi et al. 2021 LeCun al. 2015): the first layer of a neural network extracts the lowest-level features while the last layer extracts the highest-level features. In our context, the only difference between two games is the number of agents while the underlying environmental setup (e.g.. grid size) is the same. Hence, the low-level features extracted by the first layers of the policies are similar. In contrast, as the output layers capture the features of agents' decision makings which are largely impacted by the presence of other agents (more precisely, the number of agents), the extracted high-level features could be very different different games (especially for small s's).
To gain more intuition on how the generated policies are different from each other, in Panel C, we perform an analysis of the policies' output representations similar to (Jaderberg et al., 2019). We manually construct some high-level game states such as "agent on the hwwprrriggggbbbtoommmmmm
of the map" and "agent on the center of the circle" and obtain the output representations of the policy of each game G(N) by using UMAP (McInnes et al. 2018). We colored the representations of some policies: N E (5, 10, 15, 20} and N E {185, 190, 195, 200) to represent small-scale and large-scale settings, respectively. From the results, we can that when N is large, the policies of different games tend to encode the same game states in similar ways (i.c., the UMAP embeddings of the same game state of different policies tend to be centered), which is aligned with the findings revealed by the CKA values. We give more analysis in Appendix
5.2 ABLATIONS
To gain a deeper understanding of the proposed approach and further support the results obtained in the previous section, we provide more discussion in this section and Appendix [}]
8

---

Published as a conference paper at ICLR 2023
Effect of Population-size Encoding. In Fig- 4 show the training curves of PAPO when using RE (i.e.. directly using N as the input). is observed that using RE can result in training collapse. In Taxi Matching environment, though PAPO with RE still gets positive rewards, it is lower than PAPO with BE. The results demonstrate the importance of the population-size encoding for training the neural networks of PAPO. More detailed analysis can be found in Appedix D.3
Exploration
Taxi Matching
Crowd in Circle
PAPO
PAPO (Ours)
200
10
100
10000
20000 Epridd
10040
40200
100000
20003 Eptsede
10000
-0000
10000
20000
50000
40000
K30
Epssdd
Figure 4: Collapsed training of PAPO when using RE.
Generalization to Unseen Games In Fig. 5 we show the performance of applying PAPO and other baselines to unseen games. We found that, though the population-size aware methods (PAPO, HyperPPO, AugPPO, and AugPPO-Large could perform better than the population-size unaware methods (PPO and PPO-Large) in some of the unseen games, their performance fluctuates and could be much worse. Therefore, important future direction is to improve the generalizability of PAPO (as well as other population cogulatiin-size-ree size-aware methods), where more advanced techniques such as adver- sarial training (Ganin et al. 2016) are needed.
Exploration
Taxi Matching
Crowd Circle
.80
'0
Poti-ange
240
NathCoyv
AugPRO
320
200
-30
HyperPPO
50
-0
PAPO (Cund)
60
40
10
225
250
275
500
125
350
375
400
221
250
275
500
125
115
4m
250
100
150
175
Population Size
Population Size
Population Size
Figure 5: Performance of PAPO and baselines on unseen games.
6 CONCLUSIONS AND FUTURE WORKS
In this work, we attempt to bridge the two research fields of finite-agent and infinite-agent games from an agent-centric perspective with three main contributions: i) novel policy optimization ap- proach, PAPO, to efficiently generate efficient policies for a set of games with different population sizes, ii) multi-task-bare training procedure to efficiently train the neural networks of PAPO, and úii) extensive experiments to demonstrate the significant superiority of our approach over several naive and strong baselines, and the analysis of the scaling laws of the policies further deepen our understanding of the two research fields of finite-agent and infinite-age -agett games.
There are several future directions. (1) Beyond the single type of agents, in real- world scenarios. the agents can be divided into multiple types (Ganapathi Subramamian et al., 20200 Yang et al.) (2020a). e.g., in Taxi Matching environment, different vehicles have different capacities (Alonso-Mon 2017]. Investigating the scaling laws of each type requires new methods and training regimes. (2) Beyond the population size (Gatchel] 202T), in future works, other game elements such as state and action spaces or even the rules of the games can be different between games. PAPO demonstrates
al.)
the possibility of learning a universal policy for completely different games. (3) Beyond the target set of games, generalization to unseen games requires more advanced techniques such as adversarial
training (Ganin et al. 2016). (4) Our approach may have implications for Meta-RL (Fakoor et al. Rakelly et al. 2019 Sohn al. 2019) as well.
2019 Finn al. 2017

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Singapore under its Industry Align- ment Fund rre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. Hau Chan is supported by the National Institute of General Medical Sciences of the National Institutes of Health [P20GM130461] and the Rural Drug Addiction Research Center at the University Nebraska- Linnoll,,,,n The content solely the respon- sibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the University of Nebraska.
REFERENCES
Yves Achdou and Mathieu Laurière. Mean field games and applications: Numerical aspects. Mean Field Games, pp. 249-307, 2020.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy policy optimization. In ICML, pp. 151-160, 2019.
Javier Alonso-Mora, Samitha Samaranayake, Alex Wallar, Emilio Frazzoli, and Daniela Rus. On- demand high-capacity ride-sharing via dynamic trip-vehicle assignment. PNAS, 114(3)462-4677
2017.
Laith Alzubaidi, Jinglan Zhang, Amjad Humaidi, Ayad Al-Dujaili, Ye Duan, Omran Al-Shamma, Santamaría, Mohammed Fadhel, Muthana Al-Amidie, and Laith Farhan. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. Journal of Big Data, 8(1):1-74, 2021.
Yu Bai and Chi Jin. Provable self-play algorithms for competitive reinforcement learning. In ICML, pp. 551-560, 2020.
Yu Bai, Chi Jin, and Tiancheng Yu. Near-optimal reinforcement learning with self-play. NeurIPS, pp. 2159-2170, 2020.
Yu Bai, Chi Jin, Huan Wang, and Caiming Xiong. Sample-efficiene learning of Stackelberg equilibria in general-sum games. NeurIPS, pp. 15799-25811 2021.
Albert-Lász16 Barabási and Réka Albert. Emergence of scaling in random networks. Science, 286 (5439):509-57) 1999.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1rXiv:1912.06680) 2019.
Andrew Brock, Theo Lim, JM Ritchie, and Nick Weston. SMASH: One-shot model architecture search through hypernetworks. In ICLR, 2018.
Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418 -2444488444411 2018.
Noam Brown and Tuomas Sandholm. Superhuman Al for multiplayer poker. Science, 365(6456):
885-890, 2019.
Theophile Cabannes, Mathieu Lauriere, Julien Perolat, Raphael Marinier, Sertan Girgin, Sarah Perrin, Olivier Pietquin, Alexandre M Bayen, Eric Goubault, and Romuald Elie. Solving N-player dynamic routing games with congestion: a mean field approach. arXiv preprint 10.. 1944, 2021.
10

---

Published as a conference paper at ICLR 2023
Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep blue. Artificial Intelligence, 134 (1-2):57-83, 2002.
2002.
Zixiang Chen, Dongruo Zhou, and Quanquan Gu. Almost optimal algorithms for two-player Markov games with linear function approximation. arXiv preprint arXiv:2102.07404, 2021.
Kai Cui and Heinz Koeppl. Approximately solving mean field games via entropy-regularized deep reinforcement learning. In AISTATS, pp. 1909-1917, 2021.
Kai Cui and Heinz Koeppl. Learning graphon mean field games and approximate Nash equilibria. In ICLR, 2022.
Kai Cui, Wasiur KhudaBukhsh, and Heinz Koeppl. Hypergraphon mean field games. arXiv preprint arXiv:2203. /6223, 2022.
Dongsheng Ding, Chen-Yr Yu Wei, Kaiqing Zhang, and Mihailo R Jovanović. Independent policy gra- dient for large-scale Markov potential games: Sharper rates, function approximation, and game- agnostic convergence. arXiv preprint arXiv:2202.041/ 2022.
Josu Doncel, Nicolas Gast, and Bruno Gaujal. Discrete field games: Existence of equilibria and convergence. Journal of Dynamics and Games, 6(3):1-19, 2019.
Christian Fabian, Kai Cui, and Heinz Koeppl. Learning sparse graphon mean field games. arXiv preprint arXiv:2209.0388 00.800 2022.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J Smola. Meta Meta- Q-laaning.. In ICLR, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaminn for fast adaptation of deep networks. In ICML, pp. 1126-1135, 2017.
Roy Fox, Stephen McAleer, Will Overman, and Ioannis Panageas. Independent natural policy gra- dient always converges in Markov potential games. arXiv preprint 10.1v:2110.10614 2021.
Zuyue Fu, Zhuoran Yang, Yongxin Chen, and Zhaoran Wang. Actor-critic provably finds Nash equilibria of linear-quadratic mean-field games. ICLR, 2019.
Sriram Ganapathi Subramanian, Pascal Poupart, Matthew E Taylor, and Nidhi Hegde. Multi type mean field reinforcement learning. In AAMAS, pp. 411-419, 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adverss training of neural net- works. Journal of Machine Learning Research, 17(1):299--0300, 2016.
Madelyn Gatchel. Variable-player learning for simulation- based games. Master's thesis, Davidson College, 2021.
Diogo A Gomes et al. Mean field games models brief survey. Dynamic Games and Applications, 4(2):110-154, 2014.
Ramakrishna Gummadi, Peter Key, and Alexandre Proutiere. Repeated auctions under budget con- straints: Optimal bidding strategies and equilibria. In the Eighth Ad Auction Workshop, volume 4, 2012.
Xin Guo and Renyuan Xu. Stochastic games for fuel followers problem: N versus MFG. SIAM Journal on Control and Optimization, 57(1):659 692, 2019.
Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning mean-field games. NeurIPS, pp- 4966-4976, 2019,
David Ha, Andrew Dai, and Quoc V Le. Hypemetworkss. arXiv preprint 1rXiv:1609.09100 2016.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, pp. 1861-1870, 2018.
11

---

Published as a conference paper at ICLR 2023
Juho Hamari, Mimmi Sjöklint, and Antti Ukkonen. The sharing economy: Why people participate in collaborative consumption. Journal of the Association for Information Science and Technology, 67(9):2047-2059, 2016.
Thomas Dueholm Hansen, Peter Bro Miltersen, and Uri Zwick. Strategy iteration is strongly poly- nomial for 2-player turn-based stochastic games with a constant discount factor. Journal of the ACM, 60(1):1-16, 2013.
Charles R Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Coumnapeau, Eric Wieser, Julian Taylor. Sebastian Berg, Nathaniel Smith, et al. Array program- ming with NumPy. Nature, 585(7825):357-266 2020.
Junling Hu and Michael P Wellman. Nash Q-leannigg for general-sum stochastic games. Journal of Machine Learning Research, 4(Nov):1039-1106) 2003.
Baihe Huang, Jason D Lee, Zhaoran Wang, and Zhuoran Yang. Towards general function approxi- mation zero-sum Markov games. arXiv preprint arXiv:2107. 14702, 2021a.
Minyi Huang, Roland P Malhamé, and Peter E Caines. Large population stochastic dynamic games: closed-loop Moceen-Vlasov systems and the Nash certainty equivalence principle. Communica- tions in Information & Systems, (13))221-5522 2006.
Yizhou Huang, Kevin Xie, Homanga Bharadhwaj. and Florian Shkurti. Continual model-based reinforcement learning with hypernetworks. In ICRA, pp. 199-805, 2021b.
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in 3D multiplayer games with population- based reinforcement learning. Science, 364(6443)::59) 865, 2019.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. NeurIPS, pp. 667-675, 2016.
Zeyu Jia, Lin F Yang, and Mengdi Wang. Feature-based Q-learning for two-player stochastic games. arXiv preprint wxXiv:996.0044: 00423, 2019.
Chi Jin, Qinghua Liu, and Tiancheng Yu. The power of exploiter: Provable multi-agent RI. in large state spaces. arXiv preprint arXiv:2106.0335) 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836 2020.
Christopher T Kello, Gordon DA Brown, Ramon Ferrer-i Cancho, John G Holden, Klaus Linkenkaer-Hansen, Theo Rhodes, and Guy C Van Orden. Scaling laws in cognitive sciences. Trends in Cognitive Sciences, 14(5):223-233 2010.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML, pp. 3519-3529, 2019.
Jean-Michel Lasry and Pierre-Louis Lions. Mean field games. Japanese Journal of Mathematics, (1):229-260, 2007.
Mathieu Lauriere. Numerical methods for mean field games and mean field type control. Mean Field Games, 78:221, 2021.
Mathieu Laurière, Sarah Perrin, Sertan Girgin, Paul Muller, Ayush Jain, Theophile Cabannes, Geor- gios Piliouras, Julien Pérolat, Romuald Élie, Olivier Pietquin, et al. Scalable deep reinforcement learning algorithms for mean field games. In ICMI., pp. 12078-12093 2022.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
12

---

Published as a conference paper at ICLR 2023
Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. Global convergence of multi-agent policy gradient in Markov potential games. In ICLR, 2022.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
Kaixiang Lin, Renyu Zhao, Zhe Xu, and Jiayu Zhou. Efficient large-scale fleet management via multi-agent deep reinforcement learning. In KDD, pp. 1774-1783, 2018.
Michael L Littman. Markov games as framework for multi-agent reinforcment learning. In ICML, pp. 157-163, 1994.
Michael I. Littman et al. Friend-or-foe Q-learning in general-sum games. In ICMI,, pp- 322-328, 2001.
Gidi Littwin and Lior Wolf. Deep meta functionals for shape representation. In ICCV, pp. 1824 1833, 2019.
Qinghua Liu, Tiancheng Yu, Yu Bai, and Chi Jin. A sharp analysis of model-based reinforcement learning with self-play. In JCML, pp. 7001-7010, 2021.
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry P Vetrov. On power laws in deep ensembles. NewrlPS, pp. 2375-2385, 2020.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 1802.03426 2018.
David Mguni, Joel Jennings, and Enrique Munoz de Cote. Decentralised learning in systems with many, many strategic agents. In AAAI, pp. 4686-4693, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level 2015.
control through deep reinforcement learning. Nature, 518(7540):529
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý. Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337 2017.
Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Credit assignment for collective multi- agent RL with global rewards. NeurIPS, pp. 8102-81111 2018.
Asuman Ozdaglar, Muhammed O Sayin, and Kaiqing Zhang. Independent learning in stochastic games. arXiv preprint 21Xiv:2111.117433 2021.
Julien Perolat, Sarah Perrin, Romuald Elie, Mathieu Laurière, Georgios Piliouras, Matthieu Geist, Karl Tuyls, and Olivier Pietquin. Scaling up mean field games with online mirror descent, arXiv 2021.
preprint ar
Sarah Perrin, Julien Pérolat, Mathieu Laurière, Matthieu Geist, Romuald Elie, and Olivier Pietquin. Fictitious play for mean field games: Continuous time analysis and applications. arXiv preprint arXiv:2007.03458, 2020.
Sarah Perrin, Mathieu Laurière, Julien Pérolat, Romuald Élie, Matthieu Geist, and Olivier Pietquin. Generalization mean field games by learning master policies. In AAAI, pp. 7143-7150, 2022.
Alexey Potapov, Oleg Shcherbakov, Innokentii Zhdanov, Sergey Rodionov, and Nikolai Skorobo- gatko. Hypernets and their application to learning spatial transformations. ICANN, pp. 476- 486. 2018
Kate Rakelly, Aurick Zhou, Chelsea Finn. Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In ICML, pp. 5331-5340, 2019.
13

---

Published as a conference paper at ICLR 2023
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent rein- forcement learning. In ICML, pp. 4295-4304, 2018.
Lars Ruthotto, Stanley Osher, Wuchen Li, Levon Nurbekyan, and Samy Wu Fung. machine learning framework for solving high-dimensional mean field game and mean field control prob- lems. PNAS, 117(17):9133-9192 2020.
Diego Rybski, Sergey V Buldyrev, Shlomo Havlin, Fredrik Liljeros, and Hermán A Makse. Scaling laws of human interaction activity. PNAS, 10663)):2660-12245, 2009.
Naci Saldi, Tamer Basar, and Maxim Raginsky. Markov-Nash equilibria in mean-field games with discounted cost. SIAM Journal on Control and Optimizatton, 56(6):4256 4287, 2018.
Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In ICML, pp. 9301-9312, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Lloyd S Shapley. Stochastic games. PNAS, 39(10):0995-1100 1953.
Aaron Sidford, Mengdi Wang, Lin Yang, and Yinyu Ye. Solving discounted stochastic two-player games with near-optimal time and sample complexity. In AISTATS, pp. 2992-3002, 2020.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game Go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362( 140- 1144, 2018.
Sungryull Sohn, Hyunjae Woo, Jongwook Choi, and Honglak Lee. Meta reinforcement learning with autonomous inference of subtask dependencies. In ICLR, 2019.
Jayakumar Subramanian and Aditya Mahajan. Reinforcement learning in stationary mean-field games. In AAMAS, pp. 251-259 2019.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czareekk,, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782)3350335555 2019.
Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Breaking the curse of many agents: Provable mean embedding Q-iteration for mean-field reinforcement learning. In ICML, pp. 10092-10103, 2020.
Chen-Yu Yu Wei, Yi-Te Hong, and Chi-Jen Lu. Online reinforcement learning in stochastic games. NeurlPS, pp. 4987-4997 2017.
Chen- Yun-Yu Wei, Chung Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. arXiv preprint arXiv:2006.0933 09510 2020.
Chen-Yu Yu Wei, Chung- Wei Lee, Mengxiao Zhang, and Haipeng Luo. Last-iterate convergence of decentralized optimistic gradient descent/ascent in infinitehhorzon competitive Markov games. In COLT, pp. 4259-4299 2021.
Qiaomin Xie. Yudong Chen, Zhaoran Wang, and Zhuoran Yang. Learning zero-sum simultancous- move Markov games using function approximation and correlated equilibrium. In COLT, pp- 3674-3682, 2020.
14

---

Published as a conference paper at ICLR 2023
Fan Yang, Alina Vereshchaka, Changyou Chen, and Wen Dong. Bayesian multi-type mean field multi-agent imitation learning. NewrIPS, pp. 1469-24788 2020a.
Jiachen Yang, Xiaojing Ye, Rakshit Trivedi, Huan Xu, and Hongyuan Zha. Learning deep mean field games for modeling large population behavior. In ICLR, 2018.
Shanwen Yang. Tianrui Li, Xun Gong, Bo Peng, and Jie Hu. A review on crowd simulation and modeling. Graphical Models, 111:101081, 2020b.
Kaiqing Zhang, Sham Kakade, Tamer Basar, and Lin Yang. Model-based multi-agent RL in zero- sum Markov games with near-optimal sample complexity. NeurIPS, pp. 1166-1178, 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
15

---

Published as a conference paper at ICLR 2023
A LEARNING IN MARKOV GAMES
In this section, we provide more discussion on learning in Markov games. First, we present some discussion and analysis independent learning for finding Nash equilibria of Markov games. Then, we provide more discussion on the connection between MGs and MFGs presented this work to further facilitate the understanding of proposal.
A.1 CONVERGENCE ANALYSIS OF INDEPENDENT POLICY GRADIENT
In given game, as all the agents learn their policies independently, the policy learning in this work falls in the category of using independent RL to find Nash equilibrium (Ozdaglar al., (202T) Re- cent works (Ding al., 2022 Fox et al. 2021 Leonardos et al., 2022) have shown that independent policy gradient can always converge to the Nash equilibrium under some conditions. In this section, we provide more analysis on the convergence of independent policy gradient in MGs, which, to some extent, provides support to our approach.
A.1.1 MARKOV POTENTIAL GAMES
In the standard definition of the Markov game (Littman 1994 Shapley 1953 Ozdaglar al. 2021
Ding et al. 2022 Fox et al., 2021: Leonardos et al.) 2022), agents are typically assumed to have access to the global state. In the context of our work, the global state is the joint state of all agents
8, = (s] sf E S.N. In this sense, a Markov game G(N) is called Markov potential game (MPG) there exists a (global) state-dependent potential function 4, II + such that
( π ")- D π ') = Vi n', ") V' "),
(4)
for all agents N, states & S and policies n², A² E IT' - IT- IT' is the policy space of agent i and IT = II (IT = x (T²) is the space of joint policy of all agents (except i).
As one of the most important and widely studied classes of games, MPGs have gained much research attention recently due to their power in modeling mixed cooperative/comptttive environments and the convergence guarantee of independent policy gradient in these games (Ding et al. 2022 Fox
et al. 2021 Leonardos et al. 2022).
A.1.2 INDEPENDENT POLICY GRADIENT
Assume that all agents update their policies according to the gradient ascent (GA) on their policies independently, i.e., only local information such each agent's rewards, actions, and observa- tions (the agent's 's local state E S) is used during the learning process (Leonardos et al. 2022). The GA is given by
(5) where i indicates the k-th iteration of the policy. Note that the policy wit+1 after k-thh update is always bounded on its policy space -i.++1 E II', i.e., *+*+i(-|ss) A(A) for all E S. If not so, a projection operation can be applied to ensure this condition. We omit such an operation for simplicity. Furthermore, we assume that all agents A use the direct policy parameterization with --greddy exploration as follows:
ri++1 : - + &V (
E
nt (a(ass) - -1-a) +
[A]
where Ti,s,a 0 for all states E S, actions E A and E 1 for all S. Essentially, n (s) ( (als))a) is mixed strategy in state 8. is the exploration parameter.
In practice, the exact gradient v,, (xrf..)) is typically unavailable, agents use stochastic gradient ascent (SGA) update their policies. Specifically, the exact gradient V V.. (rr*kk is replaced by an estimator, denoted as V4. which is typically derived from batch of observations collected through interactions with the game environment by employing the policy h k-th iteration. The commonly used (REINFORCE) gradient estimator is as follows:
R
V log n (arlst))
(7)
n
16

---

Published as a conference paper at ICLR 2023
where R' Et= " is the sum of rewards of agent along the trajectory collected by using the policy wish at k--th iteration. Now the SGA is given by
f Th + + ViEW.
(8)
1.1.3 CONVERGENCE ANALYSIS
In this section, we briefly discuss the convergence of independent policy gradient in MGs, following the results obtained in (Leonardos et al. 2022).
Proposition A.1.1. If the reward function is a global signal, i.e., r(s', a², 2nx) r(z vi N, Vs E S, and Va' E A, the finite-agent MG G(N) is an MPG.
Proof The result can be derived by showing that the conditions in Proposition 3.2 in Leonardos
et al. 2022) are satisfied. Specifically, as the agents share global reward r(=)", ), G(N) is potential game at each (global) state 8 E SN i.e., the potential function is the global reward function de (a(a)
r ((=N) )- Therefore, each agent i's instantaneous reward is decomposed as r(s", , = (a(a) +
u
where the dummy uzg (a ") = 0 trivially satisfies the condition:
T
V T: E
E
(a
= (c.1)
(9)
where and a' are respectively joint actions of all agents and of all agents other than i, ca E R, 1 E RIAI, r is the trajectory when all agents follow the policy π.
Note that here we only give simple discussion on the connection between our work and the recent advances in the convergence of independent policy gradient, which is not the focus of this work. Indeed, a global reward can ensure that G(N) is an MPG, but we do not impose such condition in the definition of the game model in Sec. In different benchmark environments, receive a global reward, a local reward (Lauriere et al. 2022), or both (Nguyen et al. 2018).
agent can
Proposition A.1.2. (Convergence of Independent Policy Gradient, Theorem 1.2 in LLonnadoss et al. 2022)) Consider the MG G(N) satisfying the condition in Proposition 1. If each agent runs SGA direct policy parameterization (Eq. (6) on their policies and the updates are simultane- ous, then the learning converges to an approximate Nash policy.
Note that the above convergence is established under the direct policy parameterization of Te. In practice, such a choice typically less expressive than using a neural network to represent the policy. However, establishing the convergence guarantee for neural network-type parameterization is extremely hard, if not impossible, because neural networks are typically highly non-lincar and non-convex/concave. Nonetheless, empirical verification can be conducted by computing the (ap- proximate) NashConv of trained policy Te. i.e., the (approximate) NashConv is O, then the policy To is Nash policy, regardless of the type of parameterization.
In our experiments, we use neural networks to represent the policies of agents. this context, as the (approximate) NashConv measures the distance of policy to the Nash policy, we train/generate a policy that has a lower (approximate) NashConv, which shows the potential (though not exact) convergence of our approach.
A.2 CONNECTION BETWEEN MG AND MFG
To facilitate the understanding of our proposal, we give some remarks on the connection between the MG and MFG described in Sec. 3
We attempt to bridge the two research fields (finite-agent MG and infinite-agent MFG) from a finite- to-infinite perspective. That is, we try to envision how will the policy of an MFG behave by in- vestigating series of finite-agent MGs. In this sense, the finite-agent MGs we considered should share a similar structure with the MFG, i.e., all agents are homogeneous. Though such game-level connection between MG and MFG is somewhat implicit. MFG provides us the guidance generate the "correct" finite-agent MGs, ie.. the games that are different only in the number of agents while other elements such state and action spaces and reward functions kept identical. Notice that
17

---

Published as a conference paper at ICLR 2023
directly generating game with an infinite number of agents is impractical as it requires infinite computing resourees for simulating the behaviors of an infinite number of agents.
On the policy-level connection, we note that it is not directly comparable between proposed PAPO and the methods for MFGs. In view of the mnite-to-infinit perspective, to implement a fair comparison, naturally we first need to generate game with an infinite number of agents and then train the networks of PAPO by using the data sampled from this game. However, we can neither generate such a game it requires infinite computing resources for simulating the behaviors of an infinite number of agents nor train the networks as PAPO will need to take N = oo as input. In this sense, our experiments do not (or more precisely, cannot) implement such comparison but provide insights into the relationship between the policies generated by PAPO and the policy of the MFG.
On the other hand, much also can be done from an infinite-to- tinte perspective that is opposite to ours. This is grounded on the well-knows fact that policy that gives mean field equilibrium also gives an ((N) equilibrium for the corresponding game with N agents, where ((N) goes to as N goes to infinity (Saldi et al. [2018). In this sense, one can first learn policy in the MFG (under the assumption that the algorithm has access to a simulator which takes the current state, action, and mean-field as inputs and outputs the reward, next state, and next mean-field (Guo et al.] 2019)) and then apply it to the finite-agent games with different N's. Essentially, this is analog to the PPO- Naive mentioned in the Introduction. Differently, here the policy is trained in the MFG while in PPO-Naive it is trained in a finite-agent MG. As a consequence, the (approximate) NashConv could be arbitrarily large when N decreases from large small (ideally, from N = oc to N = 2). In this sense, it is worth interest to propose novel methods to mitigate the loss of applying the policy learned in the MFG to finite-agent MGs, which we leave for future works.
A.3 MORE RELATED WORKS
Learning Markov Games (MGs). Among finite-agent games, Markov game (Littman 1994, Shapley 1953) is a widely used framework for characterizing games with sequential decision- making, e.g. in multi-agent reinforcement learning. There is a long line of work in developing efficient algorithms for finding Nash equilibria of MGs under various assumptions such full en- vironmental knowledge (Hansen et al. [2013] Hu & Wellman 2003] Littman et al. [001]] |Wii et al.
2020), access to simulators (Jia al. 2019] (Sidford et al. [2020] Wei et al. [017]] (021), 1), and special structures like zero-sum games (Bai & Jin 2020 Bai et al. 2020; Chen et al.) Huang et al.
2021a Jin et al. 2021 Xie al. [2200] Zhang et al. (020) and potential games (Ding et al. 2022 2022), or built on techniques for learning single-agent Markov
Fox et al. 2021
Leonardos et al.
[021 Liu al.] 202T). Our work adds to the vast body of existing works on learning in MGs. Instead of concentrating on the policy learning in a given MG, we aim to investigate how the optimal policies of agents evolve with the population size, i.e., scaling laws (Kaplan et al.] 2020; Kello et al., 2010 Lobacheva et al. 2020). and propose novel methods to efficiently generate policies that work well across games with different population sizes, given that the agents are homogeneous which is common phenomenon in many real-worle domains such as crowd modeling (Yang et al.) 20206), Ad auction (Gummadi et al. 2012), fleet management (Lin et al. 2018), and sharing economy (Hamari al. 2016).
decision process (Bai et al.
Learning in Mean-Field Games (MFGs). In contrast to finite-agent MGs, MFGs consider the case with an infinite number of agents. Since introduced in (Huang et al.) 2006) and (Lasry & Lions,
2007]. MFG has gained intensive research attention due to its power in modeling games involving a
large population of agents (Achdou & Laurière| 2020 [omes et al. 2014)) 4, Lauriere| 2021 Ruthotto
ct al. 2020). Recently, the capability of MFG is further revealed by benefiting from RL (Fu ct al.
2019 Guo et al.) 2019 Perolat et al.) [022 Perrin et al. (2020) and deep RL (Laurière et al. 2022 Perrin et al. 2022] Subramanian & Mahajan 2019 Yang et al. 2018). Despite the progress, the evolution of agent games to infnntt-agent MFGs is poorly understood and unexplored when the policies of agents are represented by deep neural networks. In this sense, our work makes the first attempt to bridge the two research fields by establishing a novel tool which we call PAPO.
A.4 MORE DISCUSSION ON THE TERM "SCALING LAWS"
The terminology "Scaling Law" has been widely used in different areas including biology, physics, social science, and computer science. It typically describes the functional relationship between two
18

---

Published as a conference paper at ICLR 2023
quantities. In this sense, we note that the two quantities are typically croblem-dependent, e.g.. the performance of a model and the model size (Kaplan et al., 2020), the fluctuations in the number of messages sent by members in communication network and their level of activity (Rybski et al.) 2009). the probability that a vertex in a social network interacts with other vertices and the & other vertices (Barabási & Albert 1999), to name few.
In this work, the two quantities are: (1) the behavior of the policy network, and (2) the number of agents. In fact, this is conceptually similar to the scaling laws in social networks which investigate how some quantity (e.g., the property of a social network such as the connectivity) changes with the number of vertices (typically. vertex stands for an individual, i.e., an agent) (Barabási & Albert 1999). Our work follows similar idea but focuses on investigating how the behavior of the policy network change with the number of agents. Therefore, the term "Scaling Law" is suited to this work as, in a more general sense, it can be used to describe the relationship between any two quantities which are determined by the specific problem at hand, not only the size of the model or training set or the amount of compute used for training (Kaplan et al., (020). Furthermore, in contrast to the areas such as natural language processing (NLP), computer vision (CV), and single-agent RL which typically consider single model, it is natural to consider the scaling laws of the policy with the number of agents in multi-agent systems (MAS).
A.5 MORE DISCUSSION ON SIMILARITY MEASURE
In this section, we give more discussion on the similarity measure employed in this work. First, we answer an important question as follows:
What is the most appropriate measure for studying the scaling laws in the context of this work?
In this work, we aim to investigate how the Nash policy Thy changes with the population size N. However, it non-trivial to determine an appropriate measure to capture the evolution of x with N. There are two intuitive choices:
(1) The performance of y. which is similar to (Kaplan al. 2020). However, we note that taking the performance as the measure is suitable only when the underlying task is identical, as in (Kaplan et al. 2020). In our work, the games with different N's are essentially different tasks, which means that the performance of (approximate NashConv) is not an appropriate measure. This is also the difference between our work and some machine learning (ML) works such as (Kaplan et al. 2020).
(2) The difference between T and fixed reference policy â. Such measure may unable to show how "i changes with N, because all Ft's could have the same difference. For example, suppose that n and स have two parameters (2 dimensions). Then, when all x 's form circle and the centroid is the fixed reference policy ft, all T 's have the same distance to ât, but they are different themselves.
Thus, considering an "absolute" quantity (performance of n. or difference between v and fixed reference policy カ) could be struggling in studying the scaling laws of the Nash policies x- Instead, we consider the "relative" change between the Nash policies, i.e., the difference p(N) between two Nash policies. Intuitively, such measure could have more implications and does not cause incon- sistency with our objective (i.e., how Th itself changes with N) as one could recover an "absolute" quantity from the "relative" changes. For example, let the reference policy be म = T T p(N) be the difference between " and 1 p(N + 1) be the difference between N and f. Then, it could be possible to derive the difference between n and Ā by combining p(N) and p(N + 1).
In addition, when considering the existence of multiple equilibria, rigorously defining the similarity measure p(NV could be more involved and outside the scope of this work, it could be closely related to closed-form solutions which are typically hard to obtain in complex multi-player games, or needs more elaborate discussion. Nevertheless, in this work, as the policies deep neural networks (DNNs), we use CKA to measure the similarity between two (approximate) Nash policies (the more similar the smaller difference between them), which is well-defined because: (1) CKA is one of the commonly used measures to characterize the similarity between two DNNs, as given in Appendix C.3) and (Kornblith et al., 2019), and (2) As mentioned in Sec. [5] such a measure is capable of characterizing the intuition: two policies are similar means that their output features (representations) of the corresponding layers (not their parameter vectors) are similar, given the same input data.
represented by
19

---

Published as a conference paper at ICLR 2023
B MORE DETAILS ON PAPO
In this section, we provide more details on PAPO and practical implementations.
B. NETWORK ARCHITECTURE
The embedding layer is linear layer with 128 neurons. The policy network consists of three fully- connected (FC) layers, each with 128 neurons. The activation between layers ReLU.
The hypernetwork starts with two FC layers, each with 128 neurons. The activation between layers ReLU. After that, three independent heads are used to generate the parameters the three layers of the policy network. In each head, three linear layers used to map the output of the two FC layers to the weights, biases, and scaling factors. Such an architecture similar to (Sarafian et al, 2021 Littwin & Wolf 2019). Fig. [ shows the architecture of the deseribed hypemetwork. The generated parameters are reshaped to form the structure the policy network. Take the hidden layer = of the policy network
example. Suppose that the output from the input layer is x and the weights, biases, and scaling factors generated by the head corresponding to the hidden layer are (2)2), (2)),, and g2(2.. respectively. Then, the output to the next layer x' ReLU(rw2222 @ (1 + 92 (z)) + b2(2)) To be more specific, in Fig. 7 we present the code snippet in the forward process of PAPO. The "emb" denotes the embedding of the population size and the batch size the number of experience tuples (transitions) obtained in every E episodes as the common practice in PPO (see Algorithm for details). The final logits will be used compute the loss during training or infer the action after passing through a softmax activation during execution.
hypernttwok
head
head
head
Bg
02 (w ax)
Og = (wr, bu
(wy, d.)
Figure 6: The architecture of hypernetwork.
state:
(batch [batch_aise, sise, atate state_dim] [batch ai2a, dim)
w1:
[batch_afae 128, state_dir + din
[batch a52e, leatch size, 128, 1] fbatch aize, leatch size, 128, 1] fbatch
gl:
128)
02:
128, leatch size, action dim, 128) [betch_eiae size, actioo din, leatch size, action din, 2
w3:
03:
g3:
out F. (toorch (w1, terch.cat ambi, unsqueezee))) (1 gi) bI)
w.reluftorch.bor (w2, x g2) b2) out torch.ema (w3, out)
g3) b3
logits torch, squeeze (out, dim-2)
logits: (batch size, action din)
Figure 7: The code snippet in the forward process of PAPO.
B.2 LOss FUNCTION
Let тe and V6 respectively denote representative agent's actor and critic (Tew. and Var respec- tively denote the old version of actor and critic which are periodically updated by copying from and Φ). Similar PPO (Schulman et al. 2017), the loss function for PAPO is:
L,(3A, ,88 = (, = (N)) e, (g
har (N)) + o2 H ((Te (10)
20

---

Published as a conference paper at ICLR 2023
where hag and hig respectively the hypernetworks for generating the actor and critic networks. For notation simplicity, here we use v and ΦN to represent hag(N) and Φ h (N),
respectively. Then, the three terms in the above equation are:
Tex (aclst)
TeN (ccls)) Aa (at(s:) 1 e, 1 + e)Ai)
)
( hg)) = I2 min
Ac, clip
( 11ge (N)) Voxx(())
Σ r(s,,, 0e = )
H( ANa( (-1s,) -BuNTTy log Tax (arlse),
where A the truncated version of generalized advantage estimation:
= = 6, + (rd)6r+1 + + (7^)T~187 with 6, = тr + VVox (5++1) (sc),
where = r(s;, at, ) and ó is the TD error (Sutoon & Barto 2018). The expectation is taken on a finite batch experiences sampled by interacting with the environment.
B.3 TRAINING PROCEDURE
We construct training procedure where the networks of PAPO trained by using the data sampled from all the games in the target set G. Specifically, at the beginning of each episode, game G(N) is uniformly sampled from the target set G. Then, PAPO takes N input and generates policy which used by the agents to interact with the environment for T steps. Next, the experience tuples of the representative agent are stored in the episode buffer D. Finally, every episodes, optimize the surrogate loss function Le(,,, ,88) with respect to B and 3 (as well as the embedding layers 27*h and s7") for K epochs with the collected ET experience tuples. As the agents are homogeneous, without loss of generality, we choose = 1 as the representative agent. A more detailed description of the training procedure shown Algorithm
Algorithm 1: Training Procedure
Input: Hyperparameters
Output: Trained PAPO
D <- 0:
for episode = 2, ... do
Sample game G(N) Uniform G]];
$6 < So, Vi E NN;
Generate policy T ag)
0 T do
Sample action of ~ (-(-s))) Vi EN;
Execute of in G(N), Vi N:
Receive reward r, = r(s), al, 2 vi N;; Observe state $/ vi E N;
Store data D + DU ((s), at ri, s1+));;
end 12
13
episode% then Update 3r and 8e (and 1 and n7 by optimizing L, for epochs; D + 0. Gold - 0, Dold + Φ;
14
15
16
end
17 end
21

---

Published as a conference paper at ICLR 2023
C EXPERIMENTAL DETAILS
C.I DETAILS ON GAME ENVIRONMENTS
Exploration (Laurière et al. 2022
Consider a grid environment with size M x M. The state of an agent is his coordinate &t (z. v) and available actions are: left, right, up, down, and stay. The agent cannot walk through the walls the boundaries. Given S, (xt, y), at, and μt, the reward for the agent is r88e, Gr. (r) = (o)():(s.)). In experiments, we set M = 10.
Taxi Matching (Nguyen al. 2018). Consider grid world environment with size M M. A set of drivers aim to maximize their own revenue by picking orders distributed over the map. As the demand in different zones is varied (e.g., the demand in downtown higher than that in residential areas), each driver needs to find an optimal roaming policy while taking the competition with other drivers into consideration. The state of a driver is his coordinate 8 (x, y) and actions are: left, right, up, down, and stay. Given 8t = (z, y), ag. and M4g the reward for a driver is r(st, Qt: (4) = -O,, log( (s))), where Os, denotes the total order price in state 8t. In experiments, we set M = 10 and consider that there are 100 orders each with reward of 1 and distributed over the map according to the Gaussian distribution with the mean at the center of the map.
Crowd in Circle (Perrin et al. 2020). Consider 1D environment with M = 20 states denoted as S= =1, 21.2..... |S] 20 and form a circle. The available actions for an agent are: left, right, and stay. At each time step t, there is a point of interest (Pol, for short) located at 5.. Given St, ag and (4,, the reward for the agent is r(st Oz, (4x) = log (()()))))) + 5 x { = } i.e., the agent gets a reward of when he is located the Pol while still favoring social distancing. In experiments,
consider two Pols during an episode: for t < z 8, = 5 while for t 1 S, =15.
C.2 ALGORITIM FRAMEWORKS AND REMARKS ON BASELINES
In this section, we first introduce the baselines and then present the details of the algorithm frame- works PAPO and different baselines.
To facilitate the understanding of the experimental design, we introduce the baseline and give some remarks on them. As the network architecture of PAPO is large (has more learnable parameters), in addition to the standard baselines (PPO and AugPPO), we consider three more baselines: PPO- Large, AugPPO-Large, and HyperPPO (note that HyperPPO has a larger number of learnable pa- rameters than PPO and AugPPO and hence, we regard it as a stronger baseline), which have similar numbers of learnable parameters as PAPO by increasing the number of neurons of the hidden lay- of the policy network (PPO-Large and AugPPO-Large) critical to ensure a fair comparison and demonstrate the effectiveness of our approach. In Table give the numbers of learnable parameters of different methods in different environments. As the dimensions of the input and output of different environments are different, the numbers are different (notice that for the same environment, the numbers for different methods should be kept similar). In Table 2 we give the numbers of neurons used in different methods to obtain a similar number of learnable parameters. Note that for HyperPPO and PAPO, we change the two FC layers of the hypernetwork as the policy network structure is fixed.
hypernetwork (HyperPPO). This is
Although the methods using RE (c.g., AugPPO w/ RE, AugPPO-Large w/ RE, HyperPPO w/ RE) can be considered as the baselines, we note that they could be weaker than those using BE (AugPPO. AugPPO-Large HyperPPO) due to the possible training collapse when using RE, as shown in Fig. and Appendix [.5] Thus, when presenting the main results obtained in experiments in Fig. 3 we focus on the baselines which use BE, which also ensures a fair comparison between them and PAPO (as well as the fair comparison between these baselines). We study the effect of RE on the perfor- mance in the ablation study (Fig, and Appendix D.5).
In Fig. we present the algorithm frameworks of PAPO and different baselines. "emb" denotes the embedding of the population size (see Sec. and Appendix B.I.
In PPO/PPO-Large, the actor (critic) takes the agent's 's state and current time inputs and outputs the policy (value). PPO-Large operates in the same manner as PPO except that the number of learnable parameters is similar to PAPO.
22

---

Published as a conference paper at ICLR 2023
In Aug/PO/AugPPO-Larg the actor (critic) takes the agent's state, current time, and the embedding of population size as inputs and outputs the policy (value). AugPPO-Large operates in the same manner AugPPO except that the number of learnable parameters is similar to PAPO.
HyperPPO, we first use hypernetwork to generate the actor (critic) by taking the embedding of population size input, then the generated actor (critic) takes the agent's state and current time as inputs and outputs the policy (value).
PAPO, we first use the hypernetwork to generate the actor (critic) by taking the em- bedding of population size as input, then the generated actor (critic) takes the agent's state, current time, and the embedding of population size as inputs and outputs the policy (value).
%o/V
/T/V
Actoy/rrttc
hypernetork
AdtoCCaticc
emb
s
PPO/PPO-Large
HyperPPO
/T/V
M
Acton/Critic
hyperetwork
ActorCriticc
emb
s
emb
Aug/PO/AugPPO-Lang
PAPO
Figure 8: The algorithm frameworks of PAPO and different baselines.
Table 1: The numbers of learnable parameters of different methods in different environments.
Env./Method
PPO
AugPPO
PPO-Large
AugPPO-Large
HyperPPO
PAPO
Exploration
45K
81K
10,159K 5,936K
10,457K 5,918K
10,166K
10,176K
Taxi Matching
44K
80K
5,883K
5,893K
Crowd in Circle
44K
80K
10,146K
10,444K
9,995K
10,076K
Table 2: The numbers of neurons used in different methods.
Env./Method
PPO-Large
AugPPO-Large
HyperPPO
PAPO
Exploration Taxi Matching
(2230, 2230, 2230)
(2200, 2200, 2200)
(128, 220)
(128, 128)
(1700, 1700, 1700)
(1635, 1635, 1635)
(128, 128)
(128, 74)
Crowd in Circle
(2230, 2230, 2230)
(2200, 2200, 2200)
(128, 220)
(128, 128)
C.3 HYPERPARAMETERS
Table lists the parameters used in our experiments. Unless otherwise specified, the parameters listed in T are the same in different game environments. The parameters related to the network architectures can be found in Sec. Sec. The parameters related to the environments the previous section. Without loss of generality, we evaluate the performance of different methods in subset: G' (10, 20. ... ,200.. As for generalization to unseen games, evaluate the methods
be found in
110,00.-
23

---

Published as a conference paper at ICLR 2023
in set of unseen games: G {220, {220,240, 240. .00).. The approximate BR policy is obtained by using (single-agent) PPO. Moreover, all experiments are run on a machine with 20 Intel i9-9820X CPUs and 4 NVIDIA RTX2080 Ti GPUs, and averaged over 3 random seeds.
Table 3: Hyperparameters.
Hyperparameter
Value
optimizer
Adam
length of episode T minimum number of agents N maximum number of agents N maximum number of policy training episodes
20
2
200
2.107
maximum number of BR training episodes
1. 1-106
actor learning rate critic learning rate update every E episodes optimize K epochs each update critic loss coefficient e entropy loss coefficient C2 batch size m for computing CKA dimension of binary encoding &
3 3010
3 10
5
5
0.5
0.01
1000
12
C.4 APPROXIMATE NASHCONV
In this section, we provide more details on how to compute the approximate NashConv. Let Tex be the policy returned by method (PAPO other baselines) for the given game G(N). According to the definition given in Sec. 3. the NashConv of this policy is as follows:
NASHCONY(TON. = Σ max (' ( = ToN L. 14 Viss', 1 1o (11)
As the agents homogeneous, without loss of generality, compute the NashConv of repre- sentative agent Specifically, first train approximate BR policy for agent denoted no BR (deriving the exact BR policy for the agent is typically difficult in multi-player games). Then, we compute the approximate NashConv of the agent as:
NASHCON'
(s
i.RR { me
new )==1, (is T I.)
(12)
Roughly speaking, the approximate NashConv of agent the difference between value function of following the BR policy T the other agents are fixed to the current policy {T/
_iBR and his value function of following current policy Tex, given that
Tox J==lidi-
C.5 SIMILARITY MEASURE: CENTERED KERNEL. ALIGNMENT
In this work, we use centered kernel alignment (CKA) Korbllihh al. 2019) to measure the sim- ilarity between two policies generated by PAPO. Specifically, we aim to compute the similarity measure p(N) p(To=ha((N) (NV+)( Under CKA, the process is as follows. We randomly sample a batch of m states and fed them into the two generated policies Th= and TØ= (N+1)- Then, we obtain the output of each layer:
X input pmp
X-N hidden Ximaggaaa
X ourput
mIX Guarue
(13)
for To (N) and N X N+1 hidden and oupput respectively the number neurons of input, hidden, and output layers of the policies. Now, we can measure the similarity of each layer between the two generated policies, i.c.. we have Pinpur (N)
NN+1 for To Ta (N) where Imput- Ihiddens and Goupput are
CKA(XN imput iinutt Phidden (N) CKA(X hidden hidden), and Poutpar (N) CKA(XN outpur X XN+1)
Details on the computation of the CKA can be found in (Korbblith et al. 2019)
24

---

Published as a conference paper at ICLR 2023
D MORE EXPERIMENTAL RESULTS
In this section, we provide more results and analysis to deepen the understanding of our approach.
D.1 MORE EXPLANATIONS ON THE EXPERIMENTAL RESULTS
From Fig. Panel A, we can see that AugPPO and HyperPPO, as the two most natural methods, are competitive in terms of performance, but both are weaker than our PAPO. We hypothesize that the augmentation or hypernetwork alone would be individually insufficient. (1) HyperPPO, the population size information needs to first pass through the hypernetwork (which is typically much larger) before passing to the underlying policy network. This could be inefficient when the gradient of the embedding of the population size backpropagates through the deeper hypemetwork, which is similar to the observation in (Sarafian et al. 202T) where the context gradient did not backpropagate through the hypemeeworr. (2) In AugPPO, the population size information is directly augmented to the input of the policy network. However, the policy network is less expressive than the hypernet- work as it is typically much smaller than the hypernetwork. Therefore, by inheriting the merits of the two special cases, PAPO could achieve a better performance. However, as mentioned in Sec.F..2 instead of thoroughly investigating the two special cases and answering the question of which one is better (which could be more involved and outside the scope of our work), we propose a unified framework which encompasses the two options as two special cases.
Given the same training budget as PAPO, PPO-Large and AugPPO-Large cannot always generate approximate Nash policies for games with different population sizes. In this sense, could be the case that AugPPO-Large could perform worse than PPO-Large as it is more sensitive to the popu- lation size. However, we note that we cannot derive the conclusion that AugPPO-Large performs definitely worse than PPO-Large (in "Crowd in Circle" environment, AugPPO-Large performs bet- ter than PPO-Large in small-scale settings). Further investigating the difference between PPO-Large and AugPPO-Large could be outside the scope of this work.
D.2 TRAINING CURVES
In Fig. [ present the training curves of different methods in different environments. From the results, we can see that all methods have converged after a sufficient number of training episodes (2 107). Notice that theoretically computing the exact Nash policy typically difficult in multi- player games, if not impossible. In this sense, we would expect the methods to return an approximate Nash policy, given that they are trained with a large enough training budget.
During evaluation in given game, we train a new PPO as the representative agent's 's BR policy while other agents are fixed execute the approximate Nash policy. From the representative agent's perspective, the environment is stationary and hence, the problem of learning the BR policy is re- duced an RL problem. In Fig. [0] we present the BR training curves of different methods in a given game. As we can see, the BR training curves have approximately converged, ensuring that the computed approximate NashConv is a reasonable metric to assess the quality of learning.
Exploration
Taxt Matching
Crowd in Circle
Roaum
Retum
250
PPO Large
PPO
3.00
AugPPO
PAPO (Duss)
33
150
2000
JUO
20N0
DE00
30.00
10000
33.00
4300
10000
9130
10300
Egisode
Upisede
Episods
Figure 9: Training curves of different methods in different environments.
25

---

Published as a conference paper at ICLR 2023
Exploration
Taxi Matching
Crond Circle
220
70
700
180
Resuss
35
140
34
PP1
PMD-Larg
ArghttO-Lorg Lorge
UbyerITU
120
XugPPO
PAUOO LOuno
100
33
10000
3000
10000
Fprinds
Epiodde
Fpisods
Figure 10: BR training curves of different methods in different environments (N = 10).
D.3 CKA ANALYSIS
In this section, we provide more detailed analysis the CKA values. In Fig. 3 the function p(N) with the form of p(N) = No In fact, as mentioned in Sec. 5.1 there could
B we consider
be different forms of p(N). However, thoroughly investigating all forms of p(N) could be outside the scope of this work. Instead, we choose the with the form of inverse polynomial because it coincides with the intuition that when N goes to infinity, the similarity between two policies goes (i.e., upper-bounded by 1).
Intuitively, the most general case is p(N) - cN"+7' However, by computing the p-values, we found that some parameters (c and d) are less significant and hence, can be removed from p(N). In Fig. TT. we show the curves and Table 4 p-values. We
we present the values of the parameters and their see that, in most cases, only the parameter b has a significant influence the curve fitting, i.e., has small p-value. As a result, in Fig, (B), we consider that p(N) = N and from the results we see that the two parameters (a and b) are statistically significant, which means that this simpler formula is sufficient to quantitatively characterize the evolution of the CKA values over the population size.
Though the results in Fig. 3 (B) and Fig. provide some intuitions about the evolution of the agents' optimal policies with the population size, they do not rigorously prove the convergence of finite-agent games to MFG. In fact, rigorously proving the convergence could be more involved and maybe only achievable for some special cases such as in (Guo & Xul 2019). Nevertheless, our results provide an empirical verification, which has not been done before when the policies are represented by DNNs.
Exploration
Taxi Matching
Crowd in Circle
Ingut, Row
Hidden, Raw
Outpur,
Inpot, Fit
Hidden,
Outpot,
1.0
1.0
1.0
0.9
0.9
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0
50
100
150
200
0
50
100
150
200
100
150
200
Population Size
Population Size
Population Size
Figure 11: CKA values versus population size. p(N) = 1 4.
26

---

Published as a conference paper at ICLR 2023
Table 4: Values of parameters in P(N)
+v in different game environments.
Exploration
Layer
(p)
b(p)
c(p)
d(p)
Pingun 3.81e (0.97)
3.92e - (0.97)
4.68e (0.00)
-4.88e (0.00)
Phidden
4.09e (0.99)
6.74e (0.00)
2.98e (0.99)
-3.63e +0 (0.99)
Puupus
4.22c (0.99)
8.93e -1 (0.00)
1.02e (0.99)
-1.09e* (0.99)
Taxi Matching b(p)
Layer
a(p)
c(p)
d(p)
Pinput
8.96c -3 (0.99)
1.11e (0.09)
5.71e +0 (0.99)
6.09e +0
(0.99)
Phiddes
2.90e
(0.99)
1.28e
(0.16)
2.32e
(0.99)
-2.44e
(0.99)
Pompus
8.43e
(0.99)
4.52e (0.02)
1.84e
(0.99)
+6.26e
(0.99)
Crowd in Circle b(p)
Layer
a(p)
c(p)
d(p)
Pinpus
- -..55 4 (0.91)
-7.77e" (0.16)
1.75e +0 (0.91)
-1.62e +0 (0.91)
Phidden
+1.49c (0.99)
+2.65e (0.99)
1.46e (0.99)
-1.45e (0.99)
Ponnuu
-5.13c -2 (0.99)
-3.43e -1 (0.47)
1.05e
+1 (0.99)
-1.32e +1 (0.99)
27

---

Published as a conference paper at ICLR 2023
D.4 REPRESENTATIONS OF GAME STATES
In this section, we provide more results of the representations of different game states. To this end, first manually construct some natural and understandable game states according to the position of an agent on the map/circle. For example, the game state "Bottom-left" means the agent is located on the bottom-left of the map any time. In Fig. 12 Fig. [4] we show the UMAP embeddings of different game states. We found that the UMAP embeddings of some game states (e.g.. bottom-left and upper-right Exploration, bottom-left and bottom-right in Taxi Matching, center in Crowd in Circle) present a similar trend as mentioned in the main text while the UMAP embeddings of other game states do not have clear patterns. We hypothesize that some manually constructed high-level (coarse-grained) game states may consist of very different underlying (fine-grained) states of the agents. On the other hand, to some extent, the results imply that our PAPO may not be a smooth enough function of the population size. In this sense, in future works, proposing novel techniques to drift PAPO toward a smoother function of the population size is worth interesting and may further improve the performance of PAPO, making it more elegant solution.
Bottom-left
Bottom-right
Upper-left
Upper-right
N
20-
No15
20
20-
20
10-
N-
10
10-
10
-20
-20
-10
-10
-10
50
50
30
90-
30-
20-
N-190
20-
20-
20-
-00
-10
-10-
-10
-20
-20
-20
-20
Figure 12: UMAP embeddings of different game states in Exploration.
Bottom-right
Upper-center
-10
No10
-10
-10-
-10-
N-11
-20
-20
-20
-20
50
50
30
30
20
20
20-
20
No1ss
N-190
-10
N--20
Figure 13: UMAP embeddings of different game states in Taxi Matching,
28

---

Published as a conference paper at ICLR 2023
Left-end
Center
Right-end
20
20
20
10-
10-
N-5
10
-00
10
N-15
-20
-20-
30
50-
20
10-
10-
10
185
N-190
-10
100
-20-
N-200
-20-
200-
-10
1.0
10
20
-10
10
20
Figure 14: UMAP embeddings of different game states in Crowd in Circle.
29

---

Published as a conference paper at ICLR 2023
D.5 EFFECT OF POPULATION SIZE ENCODING
In this section, we provide more analysis to explore the effect of population-size encoding on the training process. In Fig. the approximate NashConv is less meaningful in this case, for completeness, we present the result- ing approximate NashConv in Fig. Fig. I6 The results in Fig. 15 correspond Fig. | which considers small-scale settings in Taxi Matching environment. The results in Fig. 16 correspond to Fig. 3 which considers larger-scale settings in different environments.
that the training of PAPO is collapsed when using RE. Though
125
100
Approx. NashConv
PPO-Naive
75
PPO
PAPO w/ RE
50
PAPO (Ours)
15
20
30
40
45
Population Size
Figure 15: Approximate NashConv versus population size in small-scale settings in Taxi Matching environment.
Exploration
Taxi Matching
Crowd in Circle
P900
PPC-lange
AngF00
DypertPo
w/R:
140
160
120
340
80
Approx. NashCoov
100
60
40
40
20
20
50
150
100
Population Size
Population
Population Size
Figure 16: Approximate NashConv versus population size in large-scale settings in different envi- ronments.
is well known that increasing the entropy of a policy at the beginning of training can incentivize it to explore the environment more widely and in turn improve the overall performance of the pol- icy (Haarnoja et al. 2018 Ahmed et al., 2019). In this sense, given the initialized PAPO, then we expect that the generated policies have high entropy, regardless of the input population size. More precisely, given any N, the generated policy Te= by the initialized PAPO would output an approximate uniform distribution over the action space A for given state st. Let be the uniform policy, i.e., (a)s:) = A for all E A. We can compute the KL-divergence between ( and Åt denoted as ((N) (MV Then, intuitively, the beginning of training, x(N) near 0 for different N's. After the neural networks of PAPO well trained, s(N) will be larger than O and vary with N.
In Fig. 17
present the results to verify the aforementioned conclusions. From the results, can see that, when using RE, the approximate uniform action distribution only holds for small N's, as shown in the first row. However, x(NV quickly increases when N is increasing, which could severely hamper the training of PAPO (see the comparison between "PAPO w/ RE (initial)" and "PAPO w/ RE (trained)""). In striking contrast, using BE can effectively avoid such a training collapse.
30

---

Published as a conference paper at ICLR 2023
1-7
Exploratton
Tast Matching
Crowd In Circle
1.0
3.5
PAPO BE (antaa))
2.0
244
3.55
2.0
3.0
2.5
(mntto))
1.5
022
1.5
1.0
0.5
Pop-22112 St0
Popudaiio 552
3
4
5
6
0.5
1.0
50
75
100
125
150
175
200
73
100
125
150
175
200
25
200 75
100
125
180
175
20
(ininaa)
12.5
15
PAPO BE (trined)
0.000
7.5
M M Mee Wn
0.005
5.0
Wh.
2.5
0.0
1:11
1c11
3.0
PAPO wi RE (miis))
(uained)
2.0
1.0
0.0
25
50
73
100
125
150
175
200
75
100
125
150
175
200
25
75
100
125
1.0
175
200
Pupultiin Stue
Popultian Size
Popuaaion Stac
Figure 17: The KL divergence versus population size.
31

---

Published as a conference paper at ICLR 2023
D.6 REWARD DISTRIBUTION SHIFT
One of the reasons we hypothesize why PPO performs poorly across different games is the reward distribution shift in the environments. To verify the intuition, for environment, we ran- domly initialized policy to sample 1,000 trajectories (episodes) and get the reward distribution by statistics. As shown in Fig. 18 Fig. 20, the reward distribution varies sharply with the increasing population size. Intuitively. PPO works poorly across the games as it does not consider the increas- ing diversity of reward signals resulting from the changes population size. In contrast, PAPO (and other population pqpulation-size-waarr awze- methods) possesses the ability to work consistently well across different games by explicitly taking the information the population size into account during training.
N-200
Frapunty
2
1.5
Rowaz0
10
12
23
13
10
10
Figure 18: Sampled reward distributions and trajectories in Exploration.
N-100
N-150
066
0.6
9.5
0.1
9A
0.2
000
00
0.0
8,0
Reward
40
Rovnnl
Figure 19; Sampled reward distributions and trajectories in Taxi Matching.
N-5
N-50
N-108
N-1 50
N-201
0.5
0.35
8.6
0.5
0.30
8.4
0.2
02
0.80
8.0
000
Reward
Reward
Reward
Reward
Rewaed
100
10.6
9.5
500
15
10
15
Figure 20: Sampled reward distributions and trajectories in Crowd in Circle.
32


Reviews:
[{"review_id": "eub0PMA1LVJ", "reviewer": "Reviewer_BBrB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The main topic of this paper is mean field games, which have been developed to study games with a very large number of players. A mean field game correspond to the asymptotic limit where the number of players becomes infinite. Here, the authors propose to combine the mean field point of view with a hypernetwork to learn policies that can be adapted to different (finite) sizes of populations. After introducing the machine learning model, they propose an algorithm to train the network. Last, the paper presents numerical experiments on several examples, by showing how the policy learnt by the algorithm provides and approximate Nash equilibrium on games of various population sizes. The authors also provide some measure of how the learned policy changes with the population size.", "summary_of_the_review": "To the best of my knowledge, this paper proposes a new approach to use learn a policy network that can be efficient for various population sizes. The extra flexibility is probably interesting for applications. I would recommend accepting the paper, although I believe that addressing the above questions would strengthen the contributions.", "strength_and_weaknesses": "Strength: From my point of view, the paper is well written and the question is interesting for the mean field game community. The fact that several examples from the literature are treated is also a positive point.\n\nWeaknesses: \n(1) It is not clear to me why the policies are stationary although the time horizon is finite. In principle, time-dependent policies could achieve a better reward than stationary ones. \n\n(2) I find it difficult to really understand the baselines. Since this is a numerical paper, I think it would be important to provide even more details in Appendix C.2. For example, the first method is simply referred as standard PPO but it is not really clear what the inputs are. Is it only the agent’s state (and the current time)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "dY64FBzUeE", "reviewer": "Reviewer_cUv7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the scaling laws of mean-field games by proposing a variant of independent proximal policy optimization (PPO), called Population-size-Aware Policy Optimization (PAPO), that trains Nash equilibrium agent policies that generalize across different number of agents in homogeneous games by utilizing hyper-network, population-size encoding as well as population-size as policy input. It is shown that PAPO outperforms other benchmark algorithms in terms of metrics such as NashConv, and the generated policies from PAPO are also used to obtain some observations and understandings of the convergence behavior of policies (i.e., scaling laws) with an attempt to further deepen the understanding of the connection between N-player games and mean-field games.", "summary_of_the_review": "Given the above comments, I think this paper is generally a well-written paper with some notable novelties. Hence I would like to recommend this paper for acceptance. ", "strength_and_weaknesses": "Strengths:\n* This paper proposes a way to learn policies that generalize across games with different number of players. To my knowledge, this is new.\n* This paper studies the scaling laws of policies as the number of players goes to infinity.\n\nWeaknesses:\n* Too much approximation/inaccuracies (both in terms of the algorithms used to compute the policies and the methods used to evaluate the policies) in the study of the scaling laws, making it not very convincing. \n* The convergence result of independent policy gradient in Appendix A does not seem to be right. The authors are confusing symmetric games with common interest games, and Proposition A.1.1 is incorrect. In fact, if Proposition A.1.1 is correct, then one should expect that all mean-field games are potential, which is wrong (and there were other earlier papers making such similar common mistakes). Also that's why I say that there are too much approximation, since there is indeed no clue of whether independent RL converges for symmetric (but not potential/common interest) games, even in the simplest settings (e.g., without function approximation, etc.). But this is not a major result of the paper so I think it's a relatively minor weakness. But please double check carefully and fix this issue if needed in any case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "adL0TVR03d", "reviewer": "Reviewer_BxNp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the learning of infinite-player Mean-Field Games (MFG) through training on finite-agent Markov Game (MG) approximations. By combining two existing techniques, namely augmentation and hypernetworks, the authors propose a PPO-based algorithm called PAPO, which is demonstrated to achieve better performance in several mean-field game environments.", "summary_of_the_review": "Although some solid empirical effort is made to deal with a relevant problem, this paper offers little understanding of any \"scaling law\" and greatly overstates its contribution. I cannot recommend acceptance of this paper in its current form.", "strength_and_weaknesses": "**Strength:**\n1. The problem that is studied, namely Markov games with varying number of players, is interesting and relevant. \n2. The experimental results seem fair and well-represented.\n\n**Weaknesses:**\n1. The contribution of this paper is largely overstated:\n  - The abstract claims that this work is \"the first attempt to bridge the two research fields [finite-agent MG and MFG]\", overlooking existing work on the theoretical foundation for finite-agent approximation (e.g. Saldi et al.) and algorithmic design (e.g. Li et al.).\n  - The title, the abstract and the introductory section promise extensive analysis of the \"scaling law\", which is revealed to be only a presentation of the similarity measure of output policies for one specific algorithm. There is no analysis of its meaning beyond a simple curve-fitting.\n2. The writing of this paper has much to be desired.\n\n\n-----------\nSaldi et al. Markov–Nash equilibria in mean-field games with discounted cost. 2018\n\nLi et al. Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9BjtZSiS9x", "reviewer": "Reviewer_g5d3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the problem of training neural network policies in many-player games that are capable of generalizing to any number of player, rather than just a specific number N. They accomplish this through two methods. The first is augmentation, in which the number of players N is an input into the policy network. The second is a hypernetwork that takes N as input and outputs the parameters of the neural network to use for the game that has N agents. They show in experiments that this beats baseline techniques such as PPO that does not take the number of players as input.", "summary_of_the_review": "I think there is a core contribution to this paper that is valuable and validated well in experiments, but I think the framing of the paper, particularly regarding the baselines, would make it much stronger.", "strength_and_weaknesses": "Strengths: The success of binary encoding seems like the highlight of the paper to me. The fact that it works across all the games and does so much better than raw encoding is non-obvious and useful. The experiments and ablations are also well done and the paper was well-written.\n\nWeaknesses: I think the paper uses inappropriate baselines. Figure 1, for example, compares PAPO to two baselines. The first is PPO-naive which shows how PPO trained on a game variant with 20 agents generalizes to game variants with 2 <= N <= 50 agents. The second is PPO, which shows how PPO trained on 2 <= N <= 50 agents (but without N as an input to the network) performs in game variants with 2 <= N <= 50 agents. Both of these baselines will clearly perform poorly because they don't condition on the actual game state. In my opinion the appropriate baseline would be the Raw Encoding option.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster