Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the scaling laws of mean-field games by proposing a variant of independent proximal policy optimization (PPO), called Population-size-Aware Policy Optimization (PAPO), that trains Nash equilibrium agent policies that generalize across different number of agents in homogeneous games by utilizing hyper-network, population-size encoding as well as population-size as policy input. It is shown that PAPO outperforms other benchmark algorithms in terms of metrics such as NashConv, and the generated policies from PAPO are also used to obtain some observations and understandings of the convergence behavior of policies (i.e., scaling laws) with an attempt to further deepen the understanding of the connection between N-player games and mean-field games.
strength_and_weaknesses: Strengths:
* This paper proposes a way to learn policies that generalize across games with different number of players. To my knowledge, this is new.
* This paper studies the scaling laws of policies as the number of players goes to infinity.

Weaknesses:
* Too much approximation/inaccuracies (both in terms of the algorithms used to compute the policies and the methods used to evaluate the policies) in the study of the scaling laws, making it not very convincing. 
* The convergence result of independent policy gradient in Appendix A does not seem to be right. The authors are confusing symmetric games with common interest games, and Proposition A.1.1 is incorrect. In fact, if Proposition A.1.1 is correct, then one should expect that all mean-field games are potential, which is wrong (and there were other earlier papers making such similar common mistakes). Also that's why I say that there are too much approximation, since there is indeed no clue of whether independent RL converges for symmetric (but not potential/common interest) games, even in the simplest settings (e.g., without function approximation, etc.). But this is not a major result of the paper so I think it's a relatively minor weakness. But please double check carefully and fix this issue if needed in any case.
clarity,_quality,_novelty_and_reproducibility: The writing is clear and relatively easy to follow. The generalization across different games with different number of players is novel, but I think the authors are over-claiming their contribution in terms of studying the scaling laws. There have been numerous works studying the convergence behavior of policies from both the theoretical and computational viewpoints as the number of players goes to infinity, like [A,B] and some other papers citing [C] (a few with rates/scaling laws of convergence as well), to name just a few. But the scaling law study in this paper seems to be more general than existing works, despite the approximation/inaccuracies mentioned above. The reproducibility looks good in general, but the detailed hyper-network structure in Appendix B.1 might better be made clearer with an illustration figure, and the shapes of $x$, $w_2(z)$, $g_2(z)$ and $b_2(z)$ and why they match might better be made clearer as well.

[A] Guo, X., & Xu, R. (2019). Stochastic games for fuel follower problem: N versus mean field game. SIAM Journal on Control and Optimization, 57(1), 659-692.

[B] Cabannes, T., Lauriere, M., Perolat, J., Marinier, R., Girgin, S., Perrin, S., ... & Elie, R. (2021). Solving N-player dynamic routing games with congestion: a mean field approach. arXiv preprint arXiv:2110.11943.

[C] Saldi, N., Basar, T., & Raginsky, M. (2018). Markov--Nash equilibria in mean-field games with discounted cost. SIAM Journal on Control and Optimization, 56(6), 4256-4287.
summary_of_the_review: Given the above comments, I think this paper is generally a well-written paper with some notable novelties. Hence I would like to recommend this paper for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the learning of infinite-player Mean-Field Games (MFG) through training on finite-agent Markov Game (MG) approximations. By combining two existing techniques, namely augmentation and hypernetworks, the authors propose a PPO-based algorithm called PAPO, which is demonstrated to achieve better performance in several mean-field game environments.
strength_and_weaknesses: **Strength:**
1. The problem that is studied, namely Markov games with varying number of players, is interesting and relevant. 
2. The experimental results seem fair and well-represented.

**Weaknesses:**
1. The contribution of this paper is largely overstated:
  - The abstract claims that this work is "the first attempt to bridge the two research fields [finite-agent MG and MFG]", overlooking existing work on the theoretical foundation for finite-agent approximation (e.g. Saldi et al.) and algorithmic design (e.g. Li et al.).
  - The title, the abstract and the introductory section promise extensive analysis of the "scaling law", which is revealed to be only a presentation of the similarity measure of output policies for one specific algorithm. There is no analysis of its meaning beyond a simple curve-fitting.
2. The writing of this paper has much to be desired.


-----------
Saldi et al. Markov–Nash equilibria in mean-field games with discounted cost. 2018

Li et al. Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach. 2021.
clarity,_quality,_novelty_and_reproducibility: - Some aspects of the problem description can be confusing:
  - It should be stressed that the finite-player Markov Games (MG) considered in this work are limited to those that are finite-player approximations of mean-field games. This property is described in the paper by being "homogeneous", which does not seem to be its standard usage. Note that this restriction is heavy: the MG has to be symmetric, and interaction can only happen through states., and not even the paper-scissors-rock game can fit within the definition of MG of this work. Although this formulation is sufficient enough for the setting of this work, it should nevertheless be emphasized that this work does not solve the standard MG problem.
  - The "scaling law" is limited to the similarity measure of $\pi^\ast_N$ and $\pi^\ast_{N+1}$. This approach (1) is not well defined considering Nash Equilibria may not be unique; (2) may not be a good measure at all of whether the Nash equilibria at time $N$ will perform well for different number of players.
- As far as I can tell, the algorithmic techniques, including the binary encoding of $N$ and the specific way to include hypernetworks, are novel.

summary_of_the_review: Although some solid empirical effort is made to deal with a relevant problem, this paper offers little understanding of any "scaling law" and greatly overstates its contribution. I cannot recommend acceptance of this paper in its current form.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the problem of training neural network policies in many-player games that are capable of generalizing to any number of player, rather than just a specific number N. They accomplish this through two methods. The first is augmentation, in which the number of players N is an input into the policy network. The second is a hypernetwork that takes N as input and outputs the parameters of the neural network to use for the game that has N agents. They show in experiments that this beats baseline techniques such as PPO that does not take the number of players as input.
strength_and_weaknesses: Strengths: The success of binary encoding seems like the highlight of the paper to me. The fact that it works across all the games and does so much better than raw encoding is non-obvious and useful. The experiments and ablations are also well done and the paper was well-written.

Weaknesses: I think the paper uses inappropriate baselines. Figure 1, for example, compares PAPO to two baselines. The first is PPO-naive which shows how PPO trained on a game variant with 20 agents generalizes to game variants with 2 <= N <= 50 agents. The second is PPO, which shows how PPO trained on 2 <= N <= 50 agents (but without N as an input to the network) performs in game variants with 2 <= N <= 50 agents. Both of these baselines will clearly perform poorly because they don't condition on the actual game state. In my opinion the appropriate baseline would be the Raw Encoding option.
clarity,_quality,_novelty_and_reproducibility: Overall the clarity is good but there are some issues with the paper that I'll list below:

1) I feel like the title isn't suited to the paper. When I saw "scaling laws" in the title I thought it referred to the size of the neural network (as is typical in machine learning papers these days). The authors apparently intend it to mean the number of agents, but even in that case it doesn't really make sense. This paper isn't about how performance and behavior changes as the number of agents grows. It's about finding a way to generate a policy that generalizes across arbitrary numbers of agents.

2) The authors reference Figure 1 in the intro but don't sufficiently explain it until the results section of the paper. I'm left wondering what this environment is, and what you are "transferring" from and to.

3) The number of citations in this paper is... excessive. The related works section is basically a list of any paper that might be slightly related. This is space that could be better used explaining this paper or explaining how a select set of papers are actually related.

4) It wasn't clear to me until the related work section that this paper is limited to Markov Games. I think that should be made clear in the abstract or at least the intro.

5) Saying Raw Encoding "cannot work in practice" seems like a strong statement that would require more explanation if you're going to make that claim. Is there simply a lack of evidence that it works, or is there some reason to think that it definitely cannot work?

6) It would be nice to see more explanation of why PAPO benefits from both augmentation and a hypernetwork. Is there a reason to think there might be some synergy between the two or are their contributions largely independent? Is there a reason to think that augmentation or hypernetwork alone would be individually insufficient in larger-scale experiments?

7) There is a typo in the last sentence of section 4.

8) In the figures, PPO-Large and AugPPO-Large have almost the same color, which makes them difficult to distinguish. However, if I'm reading the figure correctly, it looks like AugPPO-Large is doing universally worse than PPO-Large. Is there any reason to think this should be the case? Is this mislabeled?
summary_of_the_review: I think there is a core contribution to this paper that is valuable and validated well in experiments, but I think the framing of the paper, particularly regarding the baselines, would make it much stronger.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The main topic of this paper is mean field games, which have been developed to study games with a very large number of players. A mean field game correspond to the asymptotic limit where the number of players becomes infinite. Here, the authors propose to combine the mean field point of view with a hypernetwork to learn policies that can be adapted to different (finite) sizes of populations. After introducing the machine learning model, they propose an algorithm to train the network. Last, the paper presents numerical experiments on several examples, by showing how the policy learnt by the algorithm provides and approximate Nash equilibrium on games of various population sizes. The authors also provide some measure of how the learned policy changes with the population size.
strength_and_weaknesses: Strength: From my point of view, the paper is well written and the question is interesting for the mean field game community. The fact that several examples from the literature are treated is also a positive point.

Weaknesses: 
(1) It is not clear to me why the policies are stationary although the time horizon is finite. In principle, time-dependent policies could achieve a better reward than stationary ones. 

(2) I find it difficult to really understand the baselines. Since this is a numerical paper, I think it would be important to provide even more details in Appendix C.2. For example, the first method is simply referred as standard PPO but it is not really clear what the inputs are. Is it only the agent’s state (and the current time)?

clarity,_quality,_novelty_and_reproducibility: Typo:
Page 26: “posses” → “possesses” 

summary_of_the_review: To the best of my knowledge, this paper proposes a new approach to use learn a policy network that can be efficient for various population sizes. The extra flexibility is probably interesting for applications. I would recommend accepting the paper, although I believe that addressing the above questions would strengthen the contributions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the scaling laws of mean-field games by proposing a variant of independent proximal policy optimization (PPO), called Population-size-Aware Policy Optimization (PAPO), that trains Nash equilibrium agent policies that generalize across different number of agents in homogeneous games by utilizing hyper-network, population-size encoding as well as population-size as policy input. It is shown that PAPO outperforms other benchmark algorithms in terms of metrics such as NashConv, and the generated policies from PAPO are also used to obtain some observations and understandings of the convergence behavior of policies (i.e., scaling laws) with an attempt to further deepen the understanding of the connection between N-player games and mean-field games.

During the AC-reviewer meeting, reviewers are convinced with the part this paper considers a relatively new and very interesting problem, present good empirical results, and thus make solid contributions. Therefore we recommend acceptance. However, there are still some concerns we hope authors can address in the final version. (1) the title reads misleading. Multiple reviewers have pointed this out, that the word "scaling laws" typically mean things that are very different from the definitions in this paper. We would suggest to use a different word, instead of redefining "scaling laws" at the beginning of the abstract; (2) the author may want to consider the rigorous math definition of "scaling laws" (or actually the policy change according to the number of player). In particular, whether the Nash equilibria is unique in games discussed in this paper. If it's not unique, how to properly define the policy change/convergence when population size changes (i.e., which Nash in the new game should be comparing to). Since this is an empirical paper, we evaluate this paper mostly from the novelty and performance from the empirical perspective. Nevertheless, answering these basic questions rigorously can greatly improve the quality of this paper.
justification_for_why_not_higher_score: It's not very clear if the result of this paper is significant/ will lead to practical impact. The presentation needs to be improved,
justification_for_why_not_lower_score: The results are new and interesting. The experiments are solid.
Decision: Accept: poster