Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a new framework of  population learning that optimizes a single conditional model to learn and represent multiple diverse policies in real-world games.

Experiments on Mujoco Football and strategic games demonstrate its effectiveness. 
main_review: The main idea is to optimize a single conditional network to learn and represent diverse policies, and reduce the computational costs (which are very expensive in prior methods) to self-play.

The paper is well written and the derivations look correct to me. 

The experiments are strong, the authors illustrate that NeuPL can expected results of existing population learning algorithms on classical games, the visualization of learned policy population is helpful. 

I’m not sure if transferring just shared representation (encoder and memory as in Sec 2.2) can be named as skill transfer. 

The effectiveness of using this method to learn exploiters is significant. But I am wondering how much computation cost is reduced compared with prior work? If would be helpful to include a comparison. 

The connection to Schaul et al 2015 is interesting but somewhat vague, it would be helpful to elaborate more, do the authors mean that NeuPL can be viewed as UVFA?  

The underlying RL algorithm is MPO which arguably is not the state-of-the-art RL algorithm, what are the reasons behind using it? Does NeuRL also work with other value-based algorithms? 

It’s interesting that the effective population size plateaus at 12, regardless of the speciﬁed maximum capacities of the neural populations on stochastic games, did this also happen to other domains like Mujoco based environments? 
summary_of_the_review: A new framework of population training is proposed, the key idea is optimizing a shared conditional network to learn and represent diverse policies and embedding self-play. 

The paper is well presented with just a few confusing points. 

Experimental results are strong and have many interesting empirical findings.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes Neural Population Learning which I think extends PSRO in two aspects. First, it avoid the premature *good*-response. Second, it uses a conditional network to represent the population of policies, so as to enable skill transfer.  NeuPL also offers convergence guarantees under some assumptions. NeuPL is empirically verified in *rock-paper-scissors*, *running-with-scissors*, and MuJoCo Football.
main_review: **Strengths** 
1. The paper is written well and easy to follow. 
2. NeuPL is interesting, and it also converges to an N-step best response. I also like the idea of using conditional network to represent the population of policies. It could enable skill transfer and speed up the learning. 
3. The empirical study on MuJoCo Football is appreciated especially for population learning. But I recommend the authors should include more details for other researchers to reproduce the results.    


**Concerns/Questions**
1. How efficient is the LP Nash solver in Algorithm 3? Will it be the compute bottleneck when $N$ is large?
2. It is not clear to me that how the conditional policy can be used in *execution* since it makes $\sigma$ as input. 

**Minor comments**
1. Some figures are too blur to read.   
2. Some terms are not clear to me, for example, "tabula rasa". 
3. In Figure 8 (the right column), the matrix and figure do not match?  

summary_of_the_review: I think NeuPL is new and interesting. Also, NeuPL is also rigorously studied empirically. Although I have the question above mentioned, currently I feel the strengths of the paper outweigh the weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This submission provides an integrated and versatile NeuPL framework to improve the performance and convergence speed of population-based training algorithms by representing the entire population of policies within a single conditional policy network. NeuPL is very general, the commonly used population-based training algorithms, such as self-play, fictitious play, and PSRO can be regrade as its special cases by changing the meta-graph solver. The authors conduct extensive ablation experiments on some small games to validate the effectiveness of NeuPL from different perspectives. The results on the large-scale football environment also demonstrate NeuPL’s generalization ability.
main_review: **Strengths**:
1. The idea of representing an entire population of policies within a single conditional model to solve the shortcomings of existing population-based training algorithms is very elegant and novel.
2. The proposed NeuPL framework is very general and can realize a lot of current mainstream population-based training algorithms.
3. The ablation study is very detailed, which clearly demonstrate NeuPL’s effectiveness.

**Weaknesses**:
1. Although this submission is relatively well written, I think it may not be accessible to a wider range of audience since it contains too many concepts without proper explanation in the main text. For example, **a)** in the introduction section, the authors mentioned many concepts, such as game-of-skill, which should be difficult for readers who don't know much about this field to understand before reading the **game-of-skill hypothesis** proposed by [1]. **b)** In the preliminaries section, I think Figure 8 in the appendix is very helpful to understand some core concepts. I suggest putting this figure in the main text and using it as concrete examples to help readers understand these definitions. **c)** In section 1.2, the authors should introduce PSRO in more detail (perhaps in the appendix), because it is difficult for readers to understand NeuPL without PSRO. **d)** In the fourth line of the fourth page, is $E_{\sigma_i, \sigma_ j}$ a typo? I think it should be $E_{\Pi_\theta (\cdot| \sigma_i), \Pi_\theta (\cdot| \sigma_j)}$. **e)** It is better to redraw all the figures in the experiment section, all the current figures look blurry.
2. Some important related work is missing, such as [2]. Pipeline-PSRO [3] and [2] are also to accelerate the convergence speed of PSRO. It is best for the author to discuss some differences and connections between NeuPL and these methods. If possible, it is better to add some comparative experiments with [2] and [3] since the baseline PSRO-C is relatively weak.

[1] Czarnecki, Wojciech M., Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafiei, David Balduzzi, and Max Jaderberg. "Real World Games Look Like Spinning Tops." Advances in Neural Information Processing Systems 33 (2020).

[2] Smith, Max, Thomas Anthony, and Michael Wellman. "Iterative Empirical Game Solving via Single Policy Best Response." In International Conference on Learning Representations. 2020.

[3] Mcaleer, Stephen, J. B. Lanier, Roy Fox, and Pierre Baldi. "Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games." Advances in Neural Information Processing Systems 33 (2020): 20238-20248.
summary_of_the_review: The authors provide a framework called NeuPL to improve the performance and convergence speed of population-based training algorithms. The authors also conduct extensive experiments to validate the effectiveness and generalization ability of NeuPL. Although some important related work is missing and the writing can be further improved, this paper does solve some key problems in population-based training algorithms. Therefore, I recommend its acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Population based training (PBT) algorithms progressively grow a set of policies by adding best-responses to mixtures of the existing population. When RL is used as a best-response method the new policy is generally not a best-response but instead a good response. Moreover, the good-response is initialized tabula rosa potentially relearning responses to policies responded to in previous iterations. To address these concerns, this work proposes NeuPL, where a single policy is trained to represent the entire population by conditioning it on the opponent's mixture. Moreover, this proposed algorithm introduces the use of interaction graphs as a means to codify match-making in a continuously training population. NeuPL is restricted to zero-sum symmetric games with fixed player sizes. 
main_review: **Review**
This paper offers two ideas (1) concurrent training of new responses and fine-tuning of previous responses through paramterizing a single policy with the other-agent strategies, and (2) a method for representing population based training algortihms through interaction graphs. The former seems a reasonble way to prevent unnecessary relearning; however, it raises concerns of policy collapse, negative transfer, and questions of what exactly is transferred. Despite concerns, it appears that empirically this a direction worth exploring. The later, is a new lens to view population based training algorithms from, and I believe offers the community a language to discuss algorithms with that has room for fruitful discoveries. 

**Major Comments**

 - My impression from reading this paper is that the authors are proposing NeuPL as a more "general class of population learning algorithms" than PSRO (Sec 1.2). However, this seems very disingenuous because where NeuPL offers a few nicer generalizations it also restricts the class of games that can be studied. The authors need to fairly discuss their work limitations wrt prior work. In particular, acknowledging explicitly NeuPL is restricted to zero-sum symmetric games with fixed player sizes (as the policy's observation space is dependent on the game's number of players), where PSRO is not. Another area for limitations is policy collapse, or gradient interference when co-training dramatically different or contrary responses.
 - The literature review is missing related work and fails to place this work in the context of prior work that is shallowly cited. Last year's ICLR had a work that [1, 2] looked at transferring response knowledge across PSRO epochs, directly relevant to the problems being addressed here. Moreover, P2SRO [3] is shallowly cited when it is co-training across PSRO epochs. I also think it would be advantageous to compare and contrast with Self-Play, directly discussing that the changes are an expanded observation space alongside the matchmaking. Otherwise, using self-play to refer to playing against homogenously parameterized policies must be inferred. This work also has some indirect connections to [4], maybe not worth mentioning explicitly (opponent strategies used at training time instead of inference).
 - Comparing methods using only RPP, a dynamic measurement, may not paint a detailed enough picture to truly understand relative performance of two methods. For example, assuming weak RL, one method may settle into Rock-like policies while another into Paper-like policies. Creating a strict ordering in performance, and then chase each-other around the search space simply due to a staggering of learning. It would be beneficial to include absolute measurements relative to known solutions, or compare relative to a held-out static rich pool of evaluation policies. In other words, the algorithms have no equilibrium selection bias. 
 - A major claim that the authors seek to address is that good-responses are being generated instead of best-responses. Have the authors considered that this may in fact be a feature? Empirically, I have noticed that noise introduced can be beneficial for strategy space exploration. Serendipitously, I could not find a direct argument that NeuPL addresses this problem, could the authros please elaborate? I suspect that the continuous training of all policies is the argument, but it's not clear why longer training time actually solves this problem. In my experience the truncation is because the reward marginals for exponentially more training are tiny.
 - I am having trouble reconstructing PSRO from the variation of PSRO presented in Algorithm 2. I suspect this is because "N", "M", and "N-step ABR" are undefined or inconsistently used. M is not ever defined, which I believe is being used here to track PSRO iterations, but no outer loop for PSRO is presented. Is N-step BR meant to refer to training a new policy for each i \in N player? If so, for symmetric games, as this work considers, only a single BR step is necessary (train i vs -i, and give all players the new policy).
 - How were the hyperparemeters selected for NeuPL and the baseline methods? Exploration schedules in particular seem particularly challenging to define well for NeuPL.


**Minor Comments (not impacting review)**

- Sec 0, Par 3, "... network via self-play" I think overloading self-play this early on could be very misleading given its established definition.
- Sec 1.2, Par 2, "opponent's meta-game strategy \sigma_i" I think you want \sigma_{-i}
- A major limitation of this work is that policies must be homogenous. So for example, if you wished to initialize the population with a diverse set of known heuristic strategies this would require some changes and not offer the same benefits that could be gleamed from competiing methods. Not that this is a setting realized currently in the literature. 
 - Truncation of training still seems necessary for Alg-5, where it in effect represents a trade-off for good-ness of response and cost incurred from operations on the interaction graph. Did the authors explore looking at this dimension at all?

[1] Smith, Anthony, Wellman. Iterative Empirical Game Solving ia Single Policy Best Response. 2021.
[2] Smith, Anthony, Wellman. Learning to Play against Any Mixture of Opponents. 2020.
[3] McAleer, Lanier, Fox, Baldi. Pipeline PSRO: A Scalable Approach For Finding Approximate Nash Equilibria in Large Games. 
[4] Foerster, et al.. Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning. 2017.
summary_of_the_review: Overall I think the paper is sensible to appear at ICLR, but would not fight for it or be upset if it was not accepted.

The ideas presented within the work are interesting and provide a way to frame PBT methods and could facilitate the design of many future algorithms. These ideas are supported with some empirical demonstrations; however, my concerns regarding: insufficient discussion of place in literature, absent discussion of how hyperparameters are arrived at, and lack of quantitative analysis on the later experiments; leave me with reservations.
review_title: null

Meta-review: The authors propose a new framework of population learning that optimizes a single conditional model to learn and represent multiple diverse policies in real-world games. All reviewers agree the ideas are interesting and the empirical results are strong. The meta reviewer agrees and recommends acceptance.
Decision: Accept (Poster)