NeuPL: Neural Population Learning
Authors: Siqi Liu
Luke Marris
Daniel Hennes
Josh Merel
Nicolas Heess
Thore Graepel
Venue: ICLR 2022 Poster (2022)
Keywords: Multi-Agent Learning
Game Theory
Population Learning

Abstract: Learning in strategy games (e.g. StarCraft, poker) requires the discovery of diverse policies. This is often achieved by iteratively training new policies against existing ones, growing a policy population that is robust to exploit. This iterative approach suffers from two issues in real-world games: a) under finite budget, approximate best-response operators at each iteration needs truncating, resulting in under-trained good-responses populating the population; b) repeated learning of basic skills at each iteration is wasteful and becomes intractable in the presence of increasingly strong opponents. In this work, we propose Neural Population Learning (NeuPL) as a solution to both issues. NeuPL offers convergence guarantees to a population of best-responses under mild assumptions. By representing a population of policies within a single conditional model, NeuPL enables transfer learning across policies. Empirically, we show the generality, improved performance and efficiency of NeuPL across several test domains. Most interestingly, we show that novel strategies become more accessible, not less, as the neural population expands.

Published as a conference paper at ICLR 2022
NEUPL: NEURAL POPULATION LEARNING
Siqi Liu
Luke Marris University College London DeepMind marri s wowosegoooccooooooo com
Daniel Hennes DeepMind
University College London DeepMind
hennes@googlee.com
Hlusiqi@google....
Josh Merel*
Nicolas Heess DeepMind heess@googla com
Thore Graepel
DeepMind
University College London t graepe @ucl ac uk
j jmme rel @gma com
ABSTRACT
Learning in strategy games (e.g. StarCraft, poker) requires the discovery of diverse policies. This is often achieved by iteratively training new policies against existing ones, growing a policy population that is robust to exploit. This iterative approach suffers from two issues in real-world games: a) under finite budget, approximate best-response operators at each iteration needs truncating, resulting in under-trained good-responsed populating the population: b) repeated learning of basic skills at each iteration wasteful and becomes intractable the presence of increasingly strong opponents. In this work, we propose Neural Population Learning (NeuPL.) as a solution to both issues. NeuPL offers convergence guarantees to population of best-responses under mild assumptions. By representing a population of policies within single conditional model, NeuPL enables transfer learning across policies. Empirically, we show the generality, improved performance and efficiency of NeuPL across several test domains" Most interestingly, we show that novel strategies become more accessible, not less, as the neural population expands.
The need for learning not one, but a population of strategies is rooted in classical game theory. Consider the purely cyclical game of pock-paper-scisee the performance of individual strate- gies is meaningless as improving against one entails losing to another. By contrast, performance can meaningfully examined between populations. population consisting of pure strategies puper) does well against a singleton population of {scissors) because in the meta-game where both populations are revealed, a player picking strategies from the former can always beat a player choosing from the latter" This observation underpins the unifying population learning framework of Policy Space Response Oracle (PSRO) where a new policy is trained to best-respond to a mixture over previous policies each iteration, following meta-strategy solver (Lanctot et al.) (017). Most impressively, /Vinyals et al. (2019) explored the strategy game of StarCraft with a league of policies, using a practical variation of PSRO. The league counted close to a thousand sophisticated deep RL agents as the population collectively became robust to exploits.
Unfortunately, such empirical successes often come considerable costs. Population learning algorithms with theoretical guarantees are traditionally studied in normal-form games (Brown 1951 McMahan et al. 2003) where best-responses can be solved exactly. This is stark contrast to real-world Game-of of-Silllsss (Czarnecki et al. 2020)) such games are often temporal in nature, where best-responses can only be approximated with computationally intensive methods (e.g. deep RL). This has two implications. First, for a given opponent, one cannot efficiently tell apart good-responses that temporarily plateaued at local optima from globally optimal best-responses. As result, approximate best-response operators often truncated prematurely, according to hand-crafted schedules (Lanctot ct al. 2017 Mcaleer et al. 2020). Second, real-world games often afford strategy-agnoste transitive
Currently at Reality Labs, work carried out while at DeepMind.
Work carried while DeepMind.
See https://// / /neupl.... g thub. o /ewo//////////// for supplementary illustrations.
This formally quantified by Relative Population Performance, see Definition A.1 Balduzzi et al. 2019

---

Published as a conference paper at ICLR 2022
1
1/6
0
0
0
0
0
0
0 0
1/8
0
1
1/4
1
0
1/4
p
p
0
2
1-P
3
3
Population Self-Play
Strategic Cycle
Fictitious Play
PSRO-NASH
Figure I:
Popular population learning algorithms implemented as directed interaction graphs (Bot- tom), equivalently,
of meta-game mixture strategies E E RR fo, ) (Top). A directed edge from to j with weight oij indicates that policy i optimizes against j. with probability dij Unless labeled, out edges from each node weighted equally and their weights sum up to
skills that are pre-requisite strategic reasoning. Learning such skills from scratch at each iteration in the presence of evermore skillful opponents quickly becomes intractable beyond few iterations.
This iterative and isolated approach is fundamentally at odds with human learning. For humans, mastering diverse strategies often facilitates incremental strategic innovation and learning about new strategies does not stop us from revisiting and improving upon known ones (Caruana) 1997. Krakauer et al., 2006). In this work, we make progress towards endowing artificial agents with similar capability by extending population learning to real-world games. Specifically, we propose NeuPL, efficient and general framework that learns and represents diverse policies in symmetric zero-sum games within a single conditional network, using the computational infrastructure of simple show that NeuPL converges sequence of iterative best-responses under certain conditions (Section 1.3). Empirically, we illustrate the generality of NeuPL by replicating known results of population learning algorithms the classical domain of rock- paper-scissors as well as its partially-observed, spatiotemporal counterpart rumning-we rumning-with--ciin with-seissoooss (Vezhnevets al. 2020) (Section 2.1). Most interestingly, we show that NeuPL enables transfer learning across policies, discovering exploiters to strong opponents that would have been inaccessible to comparable baselines (Section [.2]]. Finally, we show the appeal of NeuPL. the challenge domain of MuloCo Football (Liu et al.) (2019) where players must continuously refine their movement skills in order to coordinate as a team. this highly transitive game, NeuPL naturally represents a short sequence of best-responses without the need for carefully chosen truncation criteria (Section) 2.4).
self-play (Section 1.2). Theoretically,
METHODS
Our method is designed with two desiderata in mind. First, at convergence, the resulting population of policies should represent sequence of iterative best-responses under reasonable conditions. Second, transfer learning can occur across policies throughout training. In this section, we define the problem setting of interests as well as necessary terminologies. We then describe NeuPL, our main conceptual algorithm well as its theoretical properties. To make it concrete, we further consider deep RL specifically and offer two practical implementations NeuPL for real-world games.
1.1 PRELIMINARIES
Approximate Best- (ABR) in Stochastic Games We consider symmetric zero-sum Stochastic Game (Shapley 1953) defined by O, A, P, R, Po) with S the state space, O the observation space and X the state for both players, Given joint actions (a,, a)) A A, the state follows the transition distribution P S x A x A Pr(S). The reward function R S - IR IR defines the rewards for both players in state s;, denoted R(ss)) (re. - rr)). The initial state of the environment follows the distribution Pa- In given state St, players act according to policies (T(-loge), n' (=(ooz2))) Player
O x O the observation function defining the (partial) views of
achieves expected return of J(m,, m') =r EE. rz] against n'. Policy n²" is a best response x'
2

---

Published as a conference paper at ICLR 2022
if Vπ. J( 2 [1T, x'). We define R t ABR(T, x²) with J(A, J[T, x²). In other words, an ABR operator yields policy á that does no worse than я, in the presence of an opponent x'.
Meta-game Strategies in Population Learning Given a symmetric zero-sum game and set of N policies : {T, /iz1. we define a normal-form meta-game where players' i-th action corresponds to executing policy for one episode. A meta-game strategy thus defines probability assignment, or an action profile, over II. Within II, we define E IR N < (VIL(II) to be the expected payoff's between pure strategies of this meta- meta-game game equivalently, Uej := J TT)) in the underlying game. We further extend the ABR operator of the underlying game to mixture policies represented by a, such that ft <- ABR ( 0, IT) with (po' x²]] ( x²)]. Finally, we define f: IR | + R!4l| to be a meta-strategy solver (MSS) with a +- ((4) and F RRNN + N meta- peta-graph solver (MGS) with Σ t F(2)). The former formulation is designed for iterative optimization of approximate best-responses as in Lanctot et al. (2017) whereas the latter is motivated by concurrent optimization over a set of population-leve objectives as in Garnelo et al. (202T). In particular, Σ E RNNN == { defines N population-level objectives, with स; optimized against the mixture policy represented by o; and II. As such, Σ E RN*N XN corresponds to the adjacency matrix of interaction graph. Figure [ illustrates several commonly used population learning algorithms defined by or equivalently, their interaction graphs.
1.2 NEURAL POPULATION LEARNING
We now present NeuPI. and contrast it with Policy-Space Response Oracles (PSRO, Lanctot et al. (017)) which similarly focuses on population learning with approximate best-responses by RL.
Algorithm 1 Neural Population Learning (Ours)
Algorithm 2 PSRO (Lanctot et al. 2017)
1: o)
Conditional neural population net.
1: II {TO}
Initial policy population. Initial meta-game strategy.
2: == [
P Initial interaction graph.
2: UNIF(I))
3: F:RNXN_ + RNX
Meta-graph solver.
3: : | - R !
> Meta-strategy solver.
4: while true
4:
5: m + (To(-1s, oz) J
b Neural population.
5: for i E [[A]]] do
D N-step ABR.
6:
for o; E UNIQUE(>) do
6: Initialize Te,
7:
IIg (l)
7: no, + oB,
8:
11 + ABR(Ug
o.. 11)
Self- play.
8: 11 t 11 {TT,)
9:
u e ))T)
(Optional) if F adaptive.
9:
2 EVAL.
Empirical payoffs.
10:
Σ+ F(()
(Optional) if F adaptive.
10:
a 5 F(L))
11:
return No. Σ
return II
NeuPL deviates from PSRO in two important ways. First, NeuPL suggests concurrent and continued training of all unique policies such that no good-response features in the population prematurely due early truncation. Second, NeuPL represents an entire population of policies via a shared conditional network [I((- or) with each policy [Ig(-|s, di)) conditioned on and optimised against meta-game mixture strategy og enabling transfer learning across policies. This representation also makes NeuPI. general: it delegates the choice of effective population sizes [UNIQUR(E) < [Σ| N to the meta-graph solver as a; = a, implies [- a.) = [- a,) (cf. Section 2.1 Finally,
NeuPL allows for cyclic interaction graphs, beyond the scope of PSRO. We discuss the generality of NeuPL in the context of prior works in further details in Appendix D
N-step Best-Responses via Lower- Triin--ri Graphs A popular class of population learning algorithms seeks to converge to a sequence of N iterative best-responses where each policy n; is best-response to an opponent meta-game strategy with support over a subset of the policy population I2;; = { J In NeuPL, this class of algorithms are implemented with meta-graph solvers that return lower-triangular adjacency matrices Σ with Σ,sj = O. Under this constraint, do becomes a zero vector, implying that Hl (-]s, ou) does not seek to best-respond to any policies. Similar to the role of initial policies {To) } in PSRO (Algorithm 2}. IIg (s, op) serves
starting point for the sequence of N-step best-responses and any fixed policy can be used. We note that this property further allows for incorporating pre-trained policies in NeuPL. as we discuss in Appendix D.1
3

---

Published as a conference paper at ICLR 2022
Algorithm 3 A meta-graph solver implementing PSRO-NASIL.
I: function N(14)
bu NxN the empirical payoff matrix.
Initialize meta-game strategies XNN with zeros.
3: for i E (I....,N 1) do
4: Di41.1 +- NAL
LP Nash solver, see Shoham & Leyton-Brown 2008
5:
return 2
One prominent example is PSRO-NASII, where mi is optimized to best-respond to the Nash mixture policy over 21 This particular meta-graph solver is shown in Algorithm 3
1.3 CONVERGENCE TO N-STEP BEST- RESSORESP VIA NEUPL
Under certain assumptions on the best-response operator, interaction graph, and meta-graph solver (MGS) we can construct proofs that NeuPL converges N-step best-response. We introduce the term grounded (Section C to refer to interaction graphs and MGS that have a structure that imposes convergence unique set of policies. Certain interaction graphs grounded, in particular, lower- triangular graphs are one such class which describe an -seppp best response. In addition, certain MGSs grounded, in particular, ones that operate the sub- sal--payff and output lower-triangular interaction graph, F :uci,ei + Li,<i. The lower- triangula maximum entropy Nash equilibrium (MENE) one such grounded MGS. Therefore with sufficiently large N, NeuPL will converge to normal-form Nash equilibrium. See Section C for the full definitions, theorems and proofs.
1.4 NEURAL POPULATION LEARNING BY RL
We now define the discounted return maximized by IIa(- o;) in Equation We denote P(at) as the probability distribution over policy i's opponent identities σj E {o1...... oN Intuitively, each policy is maximizing its expected returns in the underlying game under a double expectation: the first taken over its opponent distribution, with oj ~ P(az) and the second taken under the game dynamics partly defined by the pair of policies (To(--los, 0.), (0(1)''))))))
Ja. = E P(r)
E
[2 Enn'
(1)
To optimize IT by RI, jointly train opponent-conniitinee action-valud function, approx- imating the expected return of choosing an action a given an observation history Ogt. follow- ing 1la(- o:) thereafter in the presence of 1l-(-log.. σj), denoted by Q(ose, ae oi, o.
Enoo(oo.ss...o(( |Dt+T " a.} with the discount factor. In the case of ABR by deep RL, we could additionally approximate the expected payoffs matrix 44 by learning a payoff esti- En-TTo(- (ooo, (Qo(o, a, og or )))
mator (a, dy) minimizing the loss Lij Ee-D [ow where the expectation taken over the state visitation distribution D defined by the pair of policies and the environment dynamics P. In other words, du (ai, oÿ) approximates the expected return of n- o;) playing against we can evaluate U efficiently. without explicitly evaluating all policies each iteration.
o,). By connecting payoff matrix Ui to the learned Q function,
Finally, we propose Algorithm 4 in the setting where the meta-graph solver Youns + F(U)) is constant function and extends instance, population learning algorithms such as Fictitious Play (Brown 1951) implement static interaction graphs while algorithms such as PSRO (Lanctot al. 2017) rely on adaptive MGS.
( the meta-graph solver adaptive in U. For
EXPERIMENTS
In this section, we validate different contributions of NeuPI. across several domains. First, we verify the generality of NeuPL. from aspects: a) NeuPL. recovers expected results of existing population learning algorithms (Brown) 1951 Heiiniih et al. 2015 Lanctot et al. 2017) on the
We present the case for action-value function but value function could he used instead.
4

---

Published as a conference paper at ICLR 2022
Self-Play
Strategic Cycle
Fictitious Play
Figure 2: Neural Population Learning in peck-paper-scisoors induced by static interaction graphs. Policy distributions are colored by training iteration from red (earliest) to green (latest). (Left) learning by self-play. (Middle) neural population of 4 strategies exploring strategic cycle. (Right) neural population of 6 strategies iteratively best respond to the average previous strategies.
classical game of rack-paper-scissors where we visualize the learned policy population over time and; b) NeuPL generalises to the spatiotemporal, partially observed strategy game running- with-scissors (Vezhnevets et al. 2020), where players must infer opponent behaviours through tactical interactions. Second, we show that NeuPL. induces skill transfer across policies, enabling the discovery of exploiters to strong opponents that would have been out-of-reach otherwise. This property translates to improved efficiency and performance compared to PSRO-NASH baselines, even under favorable conditions. Lastly, we show that NeuPL scales the large-scale Game-of-Skee of MuJoCo Football (Liu et al. 2019) where concise sequence of best-responses are learned, reflecting the prominent transitive skill dimension of the game.
In all experiments, we use Maximum A Posterior Optimization (MPO, Abdolmaleki et al. (0118) as the underlying RL algorithm, though any alternative can be used instead. Similarly, any conditional architecture can be used to implement IīA. Our specific proposal reflects the spinning top geometry (Czarnecki et al. (020)) so as to encourage positive transfers across polices. Further discussions on the network design is available in Appendix B.2 B.2
2.1 NEUPL GENERAL?
Static Interaction Graphs Figure illustrates the effect CLUUL-RRLSSTATEE implementing popular population learning algorithms in the purely cyclical game of (Left) shows that learning by self-play leads to the policy cycling through the strategy space indef- initely, as expected in such games (Balduzzi et al. (201)). By contrast, Figure 2 (Middle) shows the effect of a specialized graph that encourages the discovery of a strategic cycle as well as a final strategy that trains against the others equally. As result, obtain population that implements the pure strategies of the game as well as an arbitrary strategy. This final strategy needs not be the Nash of the game any strategy would achieve return of zero. Finally, Figure (Right) recovers the effect of Fictitious Play (Brown) 1951 Heiinrich et al. 2015) where players at each iteration optimize against the "average" previous players. We initialize the initial sink strategy to be exploitable, heavily biased towards playing rock") The resulting population, represented by (Ta {=) learned execute 6 strategies, starting with "pure-rock" which is followed by its best-response "pure-paper", with subsequent strategies gravitating towards the Nash equilibrium (NE) of this game.
Figure
Adaptive Interaction Graphs Figure 3 illustrates that NEUPL-RL- ADAPPIVE with FPSRO-N recovers the expected result of PSRO-NASH in rock-paper-sciseee -eipessscissor. Specifically, the first three strategies gradually form a strategic cycle and converge to the pure strategies of the game. As the cycle starts to form, the final strategy, best-responding to the NE over previous ones, shifted to optimize against mixture of pure strategies. These results further highlight an attractive property of NeuPL- - the number of distinct strategies represented by the neural population grows dynamically in accordance with the meta-graph solver (comparing Σ at epoch 0 and 71). In particular, distinct
"This choice is important: had the sink strategy heen initialized to be the Nash mixture strategy, subsequent strategies would be "uninteresting" as strategy improve upon the sink strategy.
5

---

Published as a conference paper at ICLR 2022
epoch
appn 285
etoelagy
popott
wegh
6C
Figure 3: NeuPI. in rak-paaee-sscisorss induced by an adaptive interaction graph implementing PSRO-NASH. An epoch lasts 10 iterations. (Top) strategy space explored by the neural population of strategies over time. (Middle) the learned payoff estimates between the population of strategies. (Bottom) Adjacency matrices representing interaction graphs as responses to payoff matrices.
objective for strategy i + 1 is introduced if and only if [o(--oc ox)) gains support in the NE over 11 Unlike prior works (Lanctot et al. 2017 Mcaleer et al. 2020), the effective population size is driven by the meta-graph solver, rather than handcrafted truncation criteria. This is particularly appealing in real-world games, where we cannot efficiently determine if a policy has converged to a best-response temporarily plateaued at local optima. In NeuPL, one needs to specify the maximum number of policies N to be represented in the neural population, yet the population need not optimize N distinct objectives the start, is it required
converge N distinct policies at convergence. The number of distinct polices represented in the neural population is a function of the strategic complexity of the game, the capacity of the neural network, the effectiveness of the ABR operator and the nature of the meta-graph solver. We recall this property in rumning-withh sichsssssoors and in MuJoCo Football, which afford varying degrees of strategic complexities.
Stochastic Games rumning-with-scing -oserveed setting. Using first-person observations of the game 4x4 grid in front of the agent), each player collects resources representing "rock", "paper" and "scissors" so as to counter its opponent's 's hidden inventory. At the end of an episode or when players confront each other through "tagging", players compare their inventories and receive rewards accordingly. To do well, one must infer opponent behaviours from its partial observation history 0ge - if "rock"s went missing, then the opponent may be collecting them; if the opponent ran past "scissors". then may not be interested in it. We describe the environment in details in Appendix B.I Figure shows that NeuPL with FPSRO-N leads population of sophisticated policies. As before, sink policy to be exploitable and biased towards picking up "rock"s. Early in training, we note that the first three policies the neural population implement the pure-resource policies of "rock". "paper" and "scissors" respectively, as evidenced by their relative payoffs. In contrast to mack-papr-sccssons the mixture of pure-resource policies is exploitable the sequential setting, where the player can observe its opponent before implementing counter strategy. Indeed, policy Lla(-los, 04) observes and exploits, beating the mixture policies at epoch 680, Following FPSRO-N. 1Ig (- (-log, σs) updates its objective to focus solely on this newfound NE over mT developing deceptive counter strategy.
(Vezhnevets et al. 2020) extends pokkppper-sccisoors to
the spatiotemporal and
set the initial
Figure (Left) quantitatively verify that NeuPL. implementing FPSRO-N indeed induces policy popu- lation that becomes robust to adversarial exploits as the population expands. To this end, we compare independent policy populations by their Relative Population Performance (RPFF) Appendix A. across 4 independent NeuPL experiments with different maximum population sizes. As expected, we observe that neural populations representing more best-response iterations are less exploitable. Additionally, population size greater than 8 has limited impact on learning, both in terms of marginal exploitability benefits and rate of improvement against smaller populations (shown in blue and or- ange). This further mitigate the concern of using a larger maximum population size than necessary. In fact, Figure 5 (Right) shows that the effective population sizes UNIQUE(E)) plateau at 12 across maximum neural population sizes. We hypothesise that this is due to the increased difficulty in
A negative RPP(2, D) implies that all mixture policies in exploitable by mixture policy in D.

---

Published as a conference paper at ICLR 2022
spech
R2cch 340
opIch 6E0
spoci 1080
epech 1961
spech 1701
EOSCE 2041
oposh 2312
Figure 4: A NeuPL population developing an increasingly sophisticated set of diverse policies in running-with-scixsors The interaction graph is updated every 1,000 gradient updates (an epoch). (Top) the learned payoff estimates U between the neural population of policies as training progresses. (Bottom) interaction graphs FPSRO-N (L)) as response the corresponding payoff matrix.
discovering reliable exploiter beyond 12 iterations in this domain - FPSRO.N n forms curriculum where new objectives are introduced and only if the induced meta-game contains novel meta-game strategies worth exploiting, regardless of the maximum population size specified. We emphasize that the only variable across these independent runs is their respective maximum neural population size.
Relative Population Parformance through Training
Eflective Neural Population Size through Training
Reterive Population Parfomance
Efrectv Papubtion Stze
10-
-20
max population size
-20-
52
16
3P
-40
gradlent step
gradient
Figure 5: (Left) Relative Population Performance comparing independent NeuPL runs of different
maximum population sizes. (Right) Effective population size time, driven by FPSRO-N-
2.2 DOES NEUPL ENABLE TRANSFER LEARNING ACROSS POLICIES?
In contrast prior works that train new policies iteratively from scratch, NeuPL represents diverse policies with explicit parameter sharing. significance of transfer learning across iterations. Specifically, we verify that the shared representation learned by training against fewer, weaker policies early in training, facilitates the learning of exploiters stronger, previously unseen opponents. To this end, set of randomly initialized MPO agents with partially transferred parameters from different epochs of the experiment shown in Figure 4 are trained against fixed mixture policies defined by _1 with 0" and " = FPSRO-N (2l*) obtained at epoch 1,200 of the same experiment. In other words, the objective for each agent is beat a fixed NE over pre-traie trained policies [IIg- (alos, [ with specific n. 6 shows the learning progressions of the agents for n E {2, 4, 7). with transferred parameters taken at epoch 0 (red) upto 1,000 (green).
the two approaches and illustrates the
Against an easily exploitable opponent mixture (NE over the first two pure-resource policies), an agent with randomly initialized parameters (red) remains capable of learning an effective best response, albeit at slower pace. This difference becomes much more apparent against competent mixture policies that execute sophisticated strategies (NE over the first or 7 policies) the randomly initialized agent failed counter its opponent despite prolonged training while agents with partially transferred parameters successfully identified exploits, leveraging effective representation of the environment dynamics and of diverse opponent strategies. By transferring skills that support sophisticated strategic decisions across iterations, NeuPL enables the discovery of novel policies that inaccessible randomly initialized agent. In other words, learning incremental best responses
7

---

Published as a conference paper at ICLR 2022
NauPL agent spochw agents
Againet Againat
spoch
Eincodee
your inttskine do-d
Encoow
500
Marmory
*
Pollyy
Policy
2308
gradeet cimp
goNdust stop
gredlect
Figure 6: Learning progression of exploiters against incremental Nash mixture policies obtained via NeuPL training (as shown in Figure 4). The red curve corresponds to learning an exploiter from random initialization fully and the green curve corresponds transferring encoder and memory components from the trained NeuPL network at epoch 1,000. Each experiment is repeated five times.
becomes easier, not harder, as the population expands. This is particularly attractive in games where strategy-agnostic skill learning is challenging in itself (Vinyals et al. 2019 Liu al. 202T).
2.3 DOES NEUPL OUTPERFORM COMPARABLE BASELINES?
Evaluating NeuPL population at different steps against fixed PSRO policy populations obtained after iterations.
Relative Population Performance
exge Ceseeeenn Cameena
Expertment
PSND-201002
PSRO-C40200
gradient step
Figure 7: Relative Population Performance between a NeuPL population and policy populations obtained in PSRO baselines. Each PSRO variant is repeated over 3 trials, shown in shades.
We compare a NeuPL population implementing FPSRO-N to 4 comparable baselines implementing variants of PSRO. Since PSRO does not prescribe truncation criteria each iteration, we investigate PSRO baselines with 100k and 200k gradient steps per iteration respectively. Further, the effect of continued training across iterations, by initializing new policies with the policy obtained at the end of the preceding iteration instead of random initialization. We refer to this continued variant as PSRO-C. All PSRO populations are initialized with the same initial policy used for the Fillustrates the quantitative benefits of NeuPL, measuring RPP between NeuPL. population of maximum population size of 8 against the final population of policies obtained via PSRO after 7 iterations. The vertical dashed lines indicate that both the NeuPL population and the PSRO population have cumulatively undergone identical amount of training to allow for fair comparison. In purple, show the effective population size of the NeuPL population which has been shown previously in Figure We make the following observations: i) with a population size of 8, the NeuPL population successfully exploits PSRO baselines representing an equal number of policies, even if the latter performed twice many gradient updates; ii) the increase in RPP coincides with an increase in the effective population size, from 5 to 8, reaching the maximum number of distinct policies that this NeuPL. population can represent; iii) the amount of training each PSRO generation received has limited impact on the robustness of policy populations at convergence. This corroborates our observations in Figure 6 where the agent (in red) failed to exploit strong opponents despite continued training. Interestingly, PSRO-C proves equally exploitable. We hypothesize that the learned policies failed develop reusable representations that support diverse strategic decisions. Details of the PSRO basclines across seeds are available in Appendix demonstrating the strategic complexities captured the PSRO baseline populations.
consider
NeuPL population. Figure 7
8

---

Published as a conference paper at ICLR 2022
2.4 DOES NEUPL SCALE TO HIGHLY TRANSITIVE CAME-OF-SKILLLE
a game is purely transitive, all policies share the same best-response policy. In this case, self-play offers natural curriculum that efficiently converges to this best-response (Balduzzi al. 2019).
Nevertheless, this approach is infeasible in real-world games as one cannot rule out strategic cycles in the game without exhaustive policy search. The MuJoCo Football domain (Liu et al. 2019) is one such example: it challenges agents to continuously refine their motor control skills while coordinated team-play intuitively suggests the presence of strategic cycles. In such games, PSRO challenging proposal as it requires carefully designed truncation criteria. If an iteration terminates prematurely due to temporary plateaus in performance, good-responses are introduced and convergence slowed unnecessarily; if iterations rarely terminate, then the population may unduly delay the representation of strategic cycles. In such games, NeuPL offers an attractive proposal that retains the ability to capture strategic cycles, but also falls back to self-play if the game appears transitive.
Figure shows the learning progression of NeuPL. implementing FFSRO in this domain, starting with sink policy that exerts torque all actuators. We observe that Ty (-log, 02) exploits the sink policy by making rapid, long-range shots which is in turn countered by [g(- da), that intercepts shots and coordinates
team score. Impressively, the off-ball blue player learned to blocked off defenders, creating scoring opportunities for its teammate. With the interaction graph focused on the lower diagonal elements, this training regimes closely matches that of self-play.
training epoch
2.0e+03
1.6e+07
3.16+07
4.7e+07
(2) red
bus (3) vs (2)
payoffs
graphts
NNW Rene
Figure 8: NeuPI. in 2-vs-2 MuJoCo Football implementing FPSRO The red/blue/white traces correspond to the trajectories of red/blue players and the ball respectively.
3 RELATED WORK
Prior works attempted at making population learning scalable, motivated by similar concerns as ours. Mcaleer et al. (2020) proposed pipeline PSRO (P2SRO) which learns iterative best-responses concurrently in a staggered, hierarchical fashion. P2SRO offers principled way to make use of additional computation resources while retaining the convergence guarantee PSRO. Nevertheless, it does not induce more efficient learning per unit of computation cost, with basic skills re-learned at each iteration albeit asynchronously. contrast, Smith et al. (2020a) focused on the lack transfer learning across best-respone respones iterations and proposed "Mixed-Orackee over previous iterations is accumulated via an ensemble of policies. In this setting, each policy is trained to best-respond to meta-game pure strategy, rather than mixture strategy as suggested by the meta-strategy solver. To approximately re-construct a best-response to the desired mixture strategy, Q-mixing (Smith al. (2020b) re-weights expert policies, instead of retraining a new policy. In comparison, NeuPL enables transfer while optimising Bayes-optimal objectives directly.
where knowledge acquired
4 CONCLUSION AND FUTURE WORK
We proposed efficient, general and principled framework that learns and represents strategically diverse policies in real-world games within a single conditional making progress towards scalable policy space exploration. In addition to exploring suitable technique from the multi-tass continual learning literature, going beyond the symmetric zero-sum setting remain interesting future works, too, discussed in Appendix D

---

Published as a conference paper at ICLR 2022
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posterioni policy optimisation. In International Conference on Learning Representations, 2018. URL htttt::::://ppe/e//////
net htt.://nt///ore forum/rrmmmmmm ?id- www.-ddd-SSSSPP.. S1ANxQWOD)
David Balduzzi. Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg. and Thore Graepel. Open-ended learning in symmetric zero-sum games. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 434 443. PMLR, 09-15 Jun 2019. URL Bttp:ttt:///pceeeeeete ngs mlr press/ v htt.//alduzzzo 1 19a. html
George W. Brown. Iterative solution of games by fictitious play. Activity Analysis of Production and Allocation, 1951.
Rich Caruana. Multitask learning. Mach. Learn, 28(1))41 July 1997. ISSN 0885-6125. doi: 10.1023/4410073776666744
1023/A.0224.000 100737900007777
Wojeiech M. Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafici, David Balduzzi, and Max Jaderberg. Real world games look like spinning tops. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17443-17454. Curran Asso- ciates, Inc., 2020. URL fttttps:/prroeeeedi:: Pappdddapppppppppp
neur meurrips. ips cc/ppaper/2222//11ee
ca 1 77276199778377555d677666666
Marta Garnelo, Wojeiech Marian Czarnecki, Siqi Liu, Dhruva Tirumala, Junhyuk Oh, Gauthier Gidel, Hado van Hasselt, and David Balduzzi. Pick your battles: Interaction graphs as population- level objectives for strategic diversity. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, pp. 1501-1503, Richland, SC. 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978143038.773.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games, In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 805- Lille, France, 07-09 Jul 2015. PMLR. URL http: http:://procee.nngs...... mir htt.:/accia..........e.ee////aaa html
John W Krakauer, Pietro Mazzoni, Ali Ghazizadeh, Roshni Ravindran, and Reza Shadmehr. General- ization of motor learning depends on the history of prior action. PLOS Biology, 4(10):1-11, 09 2006. doi: w/ooorall.pboooooo..
URL
1371/ jourru
pbio. 0040316)
Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. unified game-theoretic approach to multiagent reinforcement learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL http:uuttttttt//ot//oeooo httpt:/ep:///pooeennnggg./ htw.:/eeeeeeinn....n............ neurips wttppppr/pppppp..///e ffle 1e23feelee9955cco (33
.ddf
Siqi Liu, Guy Lever, Nicholas Heess, Josh Merel, Saran Tunyasuvunakool, and Thore Graepel. Emer- gent coordination through competition. In International Conference Leurning Representations, 2019.
net// wwwp:eeettoormmmmm wow.foorrmmiiaeeer BB jR5kml
Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, S. M. Ali Eslami, Daniel Hennes, Wojciech M. Czamecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, Noah Y. Siegel, Leonard Hasenclever, Luke Marris, Saran Tunyasuvunakool, H. Francis Song, Markus Wulfmeier, Paul Muller, Tuomas Haarojja, Brendan D. Tracey, Karl Tuyls, Thore Graepel, and Nicolas Heess. From motor control to team play in simulated humanoid football. arXiv: 1219 [cs], 2021. URL httphhtt///ae .rg abs 121 961
10

---

Published as a conference paper at ICLR 2022
Luke Marris, Paul Muller, Mare Lanctot, Karl Tuyls, and Thore Graepel. Multi-agent training beyond zero-sum with correlated equilibrium sota-solvers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Leaming. volume 139 of Proceedings of Machine Learning Research, pp. 7480-7491. PMLR, 18-24 Jul 2021. URL http:://rooceedings. m wwwpss/swvvs3..
serrrislllaa a2la.
Stephen Mcaleer, JB Lanier. Roy Fox, and Pierre Baldi. Pipeline psro: A scalable approach for finding approximate nash equilibria in large games. In H. Larochelle, M. Ranzato. R. Hadsell, F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 20238 20248. Curran Associates, Inc., 2020.
ps Paper.pale
H. Brendan McMahan, Geoffrey J. Gordon, and Avrim Blum. Planning in the presence of cost functions controlled by adversary. In Proceedings of the Twentieth International Conference International Conference on Machine Learning, ICML'00 03, pp. 536-543. AAAI Press, 2003. ISBN 978-1-57735-18994
Paul Muller, Shayegan Omidshafiei, Mark Rowland, Karl Tuyls, Julien Perolat, Siqi Liu, Daniel Hennes, Luke Marris, Marc Lanctot, Edward Hughes, Zhe Wang, Guy Lever, Nicolas Heess, Thore Graepel, and Remi Munos. generalized training approach for multiagent learning. In International Conference on Learning Representations, 2020. URL htt.https://// hattps://operrevieee net/ Foo.oommmdd-dkkkkkar.
L. S. Shapley. Stochastic games. Proceedings of the National Academy of Sciences, 39(10):
1095-1100, 1953. ISSN 0027-8424. doi: (0.10773ppa..39.10... URL htt.http::/ /www
pnas. mnaa.orr/ccooteet//////////
Publisher: National Academy of Sciences _eprint:
Yoav Shoham and Kevin Leyton-Brown. Multiagent systems:
Algorithmic, game-theoretic, and logical foundations. In Multiagent systems: Algorithmic, game-theoretic, and logical foundations, chapter 4. Cambridge University Press, 2008
Max Smith, Thomas Anthony, and Michael Wellman. Iterative empirical game solving via single policy best response. In International Conference on Learning Representations, 2020a.
Max Olan Smith, Thomas Anthony, Yongzhao Wang, and Michael P Wellman. Learning play against any mixture of opponents. arXiv preprint arXiv:2009. 14180, 2020b.
Alexander Vezhnevets, Yuhuai Wu, Maria Eckstein, Rémi Leblond, and Joel Z Leibo. OPtions as REsponses: Grounding behavioural hierarchies multi-agent reinforcement learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pruceedings of Machine Learning Research, pp. 9733-9742, PMLR, 13- m.lr. wwwes/vvlls 2e119//eehneveets222.. htmllll..mmmm.mmmmmmmmmm..m..m
18 Jul 2020. URL
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575(7782)3500-34 2019.

---

Published as a conference paper at ICLR 2022
A METHODS
A.1 POPULATION LEARNING ALGORITHMS AS INTERACTION GRAPHS
Figure I shows several population learning algorithms represented as directed interaction graphs (Grnnelo et al. 2021). In particular, Policy Space Response Oracle with a Nash meta-game solver (PSRO-NASH) (Lanctot al. 2017) proposes learn best-response म; to the Nash equilibrium over policies Ii; at each iteration, resulting in an adaptive interaction graph. Specifically, the out-edges of node i + 1 are weighted according to the Nash equilibrium over the first policies (e.g.
shown as (1 p,P) for node 3 in Figure
A.2 RELATIVE POPULATION PERFORMANCE
Relative population performance was first introduced in Balduzzi et al. (2019) as the performance of individual agents meaningless in purely cyclical games. Intuitively, relative population performance measure of (BB, 2) implies that there exists mixture policy in the population B that achieves a payoff at least o (B, D) against all mixture policies in the opponent population D. This measure is defined formally in Definition A.I
Definition A.1 (Relative Population Performance). Given two populations of policies B, 2 and let (p, q) be Nash equilibrium over the zero-sum game Uss D RRM Relative Population Performance measures their relative performance: v(B, D) p Uu. q.
A.3 NEUPL WITH STATIC INTERACTION GRAPH AND ABR BY RL
In this section, we describe a specialised implementation of NeuPL given static meta-graph solvers F with an approximate best-response operator implemented via deep Reinforcement Learning. The algorithm is described in Algorithm
Algorithm 4 Neural Population Learning by RL (static F)
1: function RUN-EPISODES(E, 1T,
Visualized in Figure (Right).
b Initialize actor, critic trajectory buffers. b Generate trajectories for M episodes. 4: Sample az UNIQUE(E) uniformly (excluding sink nodes) and Ø; P(a:). Match--aking.
DD f {}: DQ + 0
3: for Episode E , M) do
5: Sample trajectories ; T from interactions between (Ioo-lo,, II (-(oo',,3)))
Dn < Dn U DQ f- Do " UT's
7:
return Pn, DQ
8:
9:
function
STATTICC, 7)
b E NxN the static interaction graph.
10:
Initialize HL (a os). the task-conditioned policy.
11:
Initialize (o,
o.). the action-value function.
12:
for Iteration {1, (1...... T') do
T the number of iterations for.
13:
Du, Pq RUN- CUN-EPISODES(E. Io, M)
14:
Optimize policy II, from Dn and Qo from Dq by RL.
A.4
NEUPL WITH ADAPTIVE META-GRAPII SOLVERS AND ABR BY RL
Building on Algorithm we now extend to the case of adaptive meta-graph solvers IF that are functions of empirical payoff matrices U. Specifically, the set of meta-game strategies the set of policies seek to best-respond to are given by + F(U). This algorithm described Algorithm
B EXPERIMENTS
RUNNING-ITH-SCISSNN ENVIRONMENT
Figure shows example view of the running-witth-scissee with-scissors environment at the start of an episode. The dashed squares shows the type of resources initialized the enclosed area with some of them consistently initialized with one type of resources while others randomly initialized with one of the
12

---

Published as a conference paper at ICLR 2022
Algorithm 5 Neural Population Learning by RL (adaptive F)
I: function NEUPL--L--DAPPTIVEE K, T,,,
FFF NXW RRNNN
Initialize II [a mt), the neural population network.
3: Initialize Qu( Q, 0;; dj), the action-value function.
4: Initialize du (a.. the empirical payoff estimator.
5: Let Ee E R RNNN f- 1/N.
All-to-all interactions. K epochs to run. D Re-compute payoff estimates. Update the interaction graph. D T iterations per epoch. D M episodes per iteration. the prop. of evaluation episodes.
6: for Epoch E {1..... K] [ do
7: Vi,j: Ui <- (o), oT)) the payoff matrix over N policies. DERN NNN + F(L)) the interaction graph over policies.
9:
for Itoration E f1, , T') do
10:
M x -)) x) Optimize policy mo from Dn and Qo from Dq U Di by RL. Optimize &w from Do U UD by minimizing Lij.
Du. RUN-E BUUN-EIIODDS(E, Dig RUN-EPISODES
11:
13:
Player
Rock
Random
Paper
Player B view
Tagging
Sciesors
Figure 9: An example view of the running- environment upon initialization.
three types. On the right, visualize the 4x4 pixel observations of the players. The visual observation is oriented along each player's forward direction. In addition to the visual observations, each player observes their current inventory the three types of resources, expressed terms of their normalized ratios. Each player is initialized with an equal weight inventory at the start of an episode. To move around, player can turn left, turn right, strafe left, strafe right, move forward or move backwards. Finally, each player can proactively seek out its opponent and "tag" it to terminate the game. A player is considered tagged if falls into the tagging area of its opponent. Bottom right view illustrates the shape of the tagging area in front of player. If neither player tags its opponent, the game ends after a fixed number of 500 steps. On the terminal step, the game resolves by comparing the inventory of both players and the rewards are assigned according the classical anti-symmetric payoff matrix of rock-paper-scisson On all other steps, both players receive zero rewards.
B.2 CONDITIONAL NETWORK ARCHITECTURE FOR NEUPL
Figure 10 (Left) illustrates the general network architecture of a NeuPL population for a typical Q-learning based RL agent. In particular, the encoder, memory and policy head network modules shared across all policies within the neural population with the conditioning variable a introduced at the final policy head layer via concatenation. This reflects the hypothetical Game-of-She geometry proposed in Czarnecki et al. (2020). where each policy can be understood as point within "spinning-top" volume. Each policy interpreted as a combination of strategy-agnostic transitive skills (c.g. movement skills for embodied agents; representational capability of past observations in partially-observer games) and a cyclic strategic element that decides which mixture policy to best-respond to. leveraging its transitive skills.
At a high-level, the goal of NeuPL to represent a compact set of policies that corresponds to the top layer of this "spinning-top" geometry, or the NE of the game. As such, proposed conditional network architecture facilitates the sharing of strategy-agnostic transitive skills across policies, while isolating graaeey-sppccif decisions that may call for different actions in the same state within the
13

---

Published as a conference paper at ICLR 2022
NeuPL
NE of thee
Interaction graph
Self- Play Episode
Encoder
.)
Memory
clotegi..ccc jer oddi-inne)
smanggpagreetl rmpatti chls
[e, 9)
Nyllooa) (
ON
o)
Figure 10: (Left) The conditional neural population network architecture for a Q-learning based RL agent. The "Ø" sign denotes a concatenation of all inputs tensors and dashed components are only used for training (and not necessary for acting); (Middle)) A diagram illustrating the separation between the strategy-agnostic transitive skill dimension captured by the shared parameters 0 and the strategic cycles captured by a at skill-level; (Right) A diagram illustrating the mechanism of a NeuPL self-play episode: at the start of an episode, a pair of conditioning vectors are sampled (os, o;)) from the interaction graph Σ. The pair of policies, using the same conditional policy network 1l 0g. J), act simultaneously in the enviromment. The circles correspond to the conditioning vectors, observations and actions for the exploiter policy (shaded) and its opponent (blank).
final apponent-conditionn policy head network, mitigating the concern of negative transfer. We note that while we limited ourselves to the simplest conditioning architecture in this work, investigating conditional network architecture with suitable inductive biases could be an interesting future direction.
B.3 HYPER-PARAMEN PAPAMEER & TRAINING INFRASTRUCTURE
For experiments on the normal form game of tock-paper-seissorss the "Encoder" and "Memory" components are omitted and the policy head network ne alone is parameterized and learned, with its only input g- The policy and action-value networks are both parameterized by 4-layer MLPs, with 32 neurons at each layer. We use small entropy cost of 0.01, learning rates of 0.001 and 0.01 for the main networks and the MPO dual variables (Abdolmalcki et al. 2018) respectively. As is typical an MPO agent, the online network parameters are copied to target networks periodically every 10 gradient steps. For the spatiotemporal, partially observable game of rumning- use small convolutional network encode the agent's 's partial observation of the environment (a 4x4 grid surrounding itself). The encoding is further concatenated with the agent's 's own inventory information and encoded through a 2-layer MLP with 256 neurons each, with re relu u activation. The memory module corresponds to recurrent LSTM network, with hidden size of 256. The policy, action-value networks are parameterized as 4-layer MLP networks. The learned payoff estimator (2(a,, or)) is parameterized as 3-layer MIP network. The learning rate of the agent networks is set to 0.0001 while the MPO dual variables are optimized with a learning rate of 0.001. The online network parameters are copied target networks every 100 gradient steps. For the MuJoCo Football environment, we used a domain specific encoder network that encodes egocentric observations of each player individually. The state representation is then implemented as weighted sum of per-play per-player embedding, using learned attention mask. Instead of outputting discrete categorical action profile each state, the policy head network trained to output a Gaussian distribution with learned mean and variance distribution parameters. As is common in continuous control literature, we used elu u as the activation function berween network components. The rest of the network architecture and ppper-parameterss remain consistent with that of network architecture and hyper-parameters are used across all experiments for the same environment.
we
The same
Figure 10 (Right) illustrates how self-play experience data is generated for NeuPL population at training time. At the start of an episode, a pair of conditioning vectors is sampled (a or,) and used to condition the pair of policies interacting in the game. In running running-with--ciie with-scissors, each NeuPL experiment uses 128 actor workers running the policy environment interaction loops and a single TPU-v2 chip running gradient updates the agent networks. The same computational resources
14

---

Published as a conference paper at ICLR 2022
Experiment
13RO-C8100K
PBRD-C#200K
PSRD881004 100K
PSRD0200K
payoff
Figure 11: Visualization of empirical payoff matrices for each PSRO population after iterations across three independent trials. Each iteration lasts 100k or 200k gradient steps. PSRO-C indicates that the policy trained at iteration i is initialized from the policy obtained at iteration instead of from random initialization. payoff of 1.0 (-1.0) indicates a win (loss) probability 100% for the row player.
used across different maximum population size for NeuPL as well as the PSRO baseline experiments. For MuloCo Football, 256 CPU actors are used per learner. For the game of pupp-spprssssssors, single CPU worker used instead. In rumning-with-scisson all experiments converge within 2 days wall-clock time. For all experiments using NeuPL, an evaluation split 0.3 is used.
B.4 BASELINE PSRO EXPERIMENTS
Figure | visualizes the empirical payoff matrices obtained in 12 independent experiments. In each experiment, we develop discrete population of policies via 8 iterations of PSRO training. In particular, we show the payoff matrices between trained policies where each PSRO iteration lasts 100k or 200k gradient steps. PSRO-C further shows the effect initializing each iteration by the policy obtained at the end of the preceding iteration instead of random initialization prescribed in Laactto al. 2017). We note that across all experiments, policy populations successfully recover the rock-papers scissors dynamics among the first three policies. In most but not all trails, PSRO also manages to learn reliable exploiter of the mixture of the three pure strategies. Finally, we note that continued training appears facilitate the discovery of richer strategic cycles, and so does allowing each PSRO iteration to perform more gradient updates.
B.5 SENSITIVITY TO THE CHOICE OF HYPER- PARAMETERS
Figure 12 investigates the sensitivity of NeuPL the choice of hyper-parametenn In particular, NeuPL introduces two additional hyper- perper-paramterss the proportion of simulation episodes used for learning pairwise empirical payoffs (a in Algorithm 55) and the interval between interaction graph update (7 in Algorithm (5). Intuitively, if is too low, the empirical payoff estimator risk being under-trained when its estimations are used by the meta-graph solver while value too high would delay the policy learning due insufficient simulation data being generated for policy learning. When comes to the meta-graphe update interval, a value too high may slow down strategic exploration the approximate best-response operators have already converged to best-responses while value too low may lead to noisy gradients the optimization process, as the set of learning objectives may change frequently.
15

---

Published as a conference paper at ICLR 2022
NeuPL: sensitivity to hyper-parameters (vs. PSRO-C@200K,
Reeatve Populaation Performanoe
Relavve Poppaation Performancs
Experiment apa
Experiment eps eps 0.3: 2000
ops 0.3:
1000
0.5:
T= 1000
0.5: T 2000
203,000
400,C00
600.000
800.005
200,000
400 000
$00.000
500,000
gradiant step
gradient step
Relative Populatton Performrnon
Relatvv Population Performancs
Experiment ops eps = 0.3: - 500
Experiment ops
eps 0.3: 5000
0.5; T 500
0.5: T 5000
202,000
400,000
600.000
800.00
200.002
400,000
500,000
600,030
gredient stop
gradient stop
Hyper- Aarraeerrr sweep proportion of matches used for empirical payoff evaluation (e) and interaction graph update interval in gradient steps (T). Population-level performance is measured in RPP, relative the same held out population PSRO- -0000K with seed 1.
Figure 12:
In practice, we observe that NeuPL is reasonably robust to these hyper-parameters across a wide range of choices. Across all experiments, evaluate the NeuPL population's 's relative population performance against a fixed, held-out population obtained via PSRO- c0200k, seed 1.
B.6 TEST-TIME EXECUTION OF LEARNED POLICIES
A distinctive property of population learning is that obtain population of policies that be executed individually at test time. In NeuPL, the population of policies is jointly defined by two elements: conditional network I- or) and of meta-game strategies Σ = {o, J derived from the empirical payoffs between policies. To execute a mixture policy defined over the population, it suffice execute [oo(-oos,, 6) with Pr(E).
Specific mixture policies have well-understood properties. For instance, playing the NE mixture policy implies that an opponent who has access to the same of policies is indifferent to playing any mixtures. This could be principled option in the absence of prior knowledge about one's we's opponent.
C CONVERGENCE PROOFS
In this section we detail the conditions upon which NeuPL will converge to a unique sets of policies. First, we discuss the theory when the interaction graph is static and grounded (Section C.I). p. Then we discuss the theory when the interaction graph is a function of the payoff matrix, so-called meta-graph solver (Section C.2). Finally we discuss specific function, Nash Equilibrium meta- -ataaagraph solver, that is popular in the literature, and prove NeuPL's convergence to normal-fome Nash Equilibrium under certain conditions (Section
C.1 GROUNDED INTERACTION GRAPHS
Assumption C.1 (Unique, Exact, Finite Best Response). Assume that we have access to best response (BR) operator that responds to policy, Rns (or equivalently a mixture over policies) exactly converges unique best response, Ta+1 BR(Tx), in finite time.
When using RL as the BR operator, uniqueness can be approximated with a small entropy term on policy being optimized. Therefore, in the situation where there is more than one best response, the
16

---

Published as a conference paper at ICLR 2022
BR operator will opt for the one with maximum entropy. This is common additional loss that is often added to RL agents, and is also believed to aid exploration (Haarnoja et al. 2018).
The second assumption, that the BR will be exact, is idealised. With sufficient model capacity, enough time, and appropriate hyper-parameee annealing schedules, RL can get close to an exact BR.
The final assumption, that the BR operator converges in finite time, is assumed because proofs follow easily from this assumption. It may be possible to relax this assumption to converge as t - oo, but leave this for future work.
Definition C.I (Grounded Interaction Graph). An interaction graph's edges define how policies should best respond to each other. We call an interaction graph grounded if its structure imposes convergence to a unique set of policies, where each policy is unique, exact, finite BR over opponent policies defined by the interaction graph.
Theorem (Grounded Lower Triangular). All lower-triangular interaction graphs (Σsz) = 0) are grounded.
Proof. The first of a lower triangular matrix is all zeros meaning that the first policy does not respond to any other policies. The first policy is therefore static (and arbitrary) and does not change over time. The second row only responds to the first policy. Since the first policy is static and are using a unique, exact and finite BR, the second policy will converge exactly to a unique policy. The nth row will respond to the converge to uniquely, so will the nth policy.
previous policies. Since over time all previous policies will
It is easy to imagine other grounded interaction graphs with interesting structure, however we will only focus on lower-triangular graphs for the purpose of this section. Research into other grounded interaction graphs may be interesting future work.
Definition C.2 (Static Interaction Graph). We call an interaction graph static if it does not change over time: E
We now prove first NeuPL result, that it can find an exact N-step best response under certain assumptions.
Theorem (NeuPL Static Lower- Trianglar Exact N- N-Step Best Response). NeuPL with a static, lower triangular interaction graph, arbitrary fixed initial policy, To, and N more policies, will converge to an exact N N-step best response, assuming that the BR is unique, exact, and converges in finite time.
Proof. The bulk of this proof can recycle the arguments in Theorem Note that NeuPL trains its conditional policies in parallel according to an interaction graph. If that graph is lower-triangular it grounded, and NeuPL will find the conditioned policies corresponding to the N-step best response.
C.2 GROUNDED CRATH-GRAPH SOLVER
is possible to extend this result on any deterministic function acting on a sub-payoff, producing a lower triangular interaction graph.
Definition C.3 (Grounded Meta-Graph Solver). A meta-graph solver is function that takes the payoff matrix as an argument and outputs an interaction graph as an output. We call a meta-graph solver, F. grounded if, when using unique, exact, finite BRs, it converges to grounded interaction graph in finite time.
Theorem (Any Deterministic Lower- Triee--rrannulr Meta-Graph Solver is Grounded), Any determinis- tic function, F. that maps a sub- sub-payoff, Uzi row, Ei in lower-triangular interaction graph is grounded meta-graph solver.
E = F(UZi,e))
Proof. The difficulty here is that in general, as the policies change, so will the payoff, and hence so will interaction graph, etc. Similar the arguments in Theorem the first policy is static and does not.
17

---

Published as a conference paper at ICLR 2022
change. The second policy can only respond to the first policy (under the lower- trianglri constraint), so the mneaa-graph solver has no flexibility to do otherwise, and therefore the second policy will also converge to a unique result. Note that these policies converge. will their sub sub-payoffs, U=a, Therefore, for row any deterministic mapping F :Maiess - Ei,ci will result in a unique set of policies.
We make further claim about NeuPL.: that it will converge unique set of conditioned policies under certain grounded meta-graph solvers.
Theorem (NeuPL. Deterministie Lower- TriingTria Exact N-Step Best Response). Assuming any deterministic lower-triangular meta-grapht solver, arbitrary fixed initial policy, no, and N more policies, NeuPL will converge to an exact N-step best response, assuming a unique, exact and finite BR.
Proof. Similar structure to Theorem 2 and Theorem
Of course, the more general result, that NeuPL will also converge for any grounded meta-graph solver is also true.
C.3 NASH EQUILIBRIUM META- META-GRRAH SOLVER
Note that the maximum entropy Nash Equilibrium (MENE) is such as grounded meta-graph solver. This will result in an algorithm similar PSRO-NASH (Lanctot et al. 2017) Double Oracle (McMahan et al. 2003).
Theorem (NeuPL. Nash Exact N-Step Best Response). Using an interaction-graphe petrrcttoongrrapfffuniiii that maps payoff to a lower triangular NE distribution:
Gn,en NEUUNNEEU--00 <n.<n)
(3)
With such function, arbitrary fixed initial policy, and sufficiently large N policies, NeuPL will converge to an exact normal-form Nash Equilibrium (NE), assuming a unique, exact, finite BR.
Proof. Starting from the proof in Theorem 4 we follow additional arguments from DO (McMahan al. 2003) to prove that for sufficient N will converge normal form NE.
Of course, many other algorithms can be recovered using specific meta-graph solvers Muulerr et al. 2020 Marris et al. 2021).
D GENERALITY OF NEUPL AND DISCUSSION ON FUTURE WORKS
For simplicity of presentation, we focused on the simple setting of learning from scratch in symmetric zero-sum games. In this section, we discuss elements needed towards applying NeuPL more generally from several aspects.
D.1 INCORPORATING PRIOR KNOWLEDGE IN NEUPL
As we alluded to in Section 1.2 the formulation of NeuPL offers a principled way to incorporate prior knowledge in the form of pre-trained policies. short. pre-trained policies can be incorporated the same way as the sink policy, with the requirement that it can only gain in-edges in the interaction graph. Figure [3 offers an illustration of such an example, where the population includes 2 pre-trained policies (Ilg (- (-lost, 11) and [lo(-lo<t, o2)) while 1I. (-lost: 0a) is optimized to best-respond to mixture over its predecessors according to a suitable meta-graph solver. As an implementation details due to the
of neural networks, σı and d2 need be unique vectors and they need be treated specifically in the corresponding MGS as they no longer represent valid probability distribution and should be excluded from policy training, similar the treatment of the sink policy.
Finally, we note that our proposed conditional network architecture is in fact synergistic with the of pre-trained policies, beyond naively including pre-trained expert policies as opponents in
18

---

Published as a conference paper at ICLR 2022
1 1-p
-1
0
0
3
0
Figure 13: Example NeuPL experiment with an interaction graph incorporating pre-trained policies.
the population. This is because the shared action-value function Qo(ogr, Oi, o.)) would learn to predict expected returns between fixed pre-trained expert policies, kick-starting the learning of encoder and memory components in the underlying game. As proposed in Figure [T0 this kick-started representation learning process should in principle transfer to the learning of other policies within the neural population well, rendering it attractive proposal.
D.2 GENERALIZATION N-PLAYER GENERAL-SUM GAMES
NeuPL. can in principle extend to n-player general-sum games. We offer elements of the solution in this section while deferring thorough investigation in this direction dedicated future work.
Consider an n-player general-sum game where the i-th player can play one of M' policies, the induced meta-game corresponds normal-form game between n players, each selecting a policy to execute for episode for one player. This yields empirical payoff tensors U = {2' Im with u' IRM' M XM the payoff tensor for player given all players' policy selections.
Coarse Correlated Equilibrium (CCE): in this setting, solving for NE becomes intractable but solution concepts such as CCE could readily used instead (Marris et al. 202T). This leads to solver that takes on the form P + F(U) with P E R MM..XM"MMMMM' a joint diint-distribution over the cartesian product of policy choices across players. This solver can thus be used in place of the meta-strategy solver, similarly and repeatedly invoked by the meta-graph solver. Note that instead of obtaining a marginal distribution for given player as in NE, CCE offers the joint distribution which can be marginalized for each player.
Heterogeneous Neural Populations: in the n-player setting, different players may take on dif- ferent roles in the game and admit entirely different observation, action spaces. This implies that heterogeneous neural populations are needed. Specifically, each player's policies can be represented by its neural population of maximum capacity M and its own n-dimensional marginal inter- action tensor. The i-th neural population is defined by a conditional model IT (-loss, of) and marginalized interaction tensor Σ R M 2 M (o k= derived from incremental CCE joint distributions.
MM
19


Reviews:
[{"review_id": "vOrTuvPgcC1", "reviewer": "Reviewer_bxQq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea is to optimize a single conditional network to learn and represent diverse policies, and reduce the computational costs (which are very expensive in prior methods) to self-play.\n\nThe paper is well written and the derivations look correct to me. \n\nThe experiments are strong, the authors illustrate that NeuPL can expected results of existing population learning algorithms on classical games, the visualization of learned policy population is helpful. \n\nI’m not sure if transferring just shared representation (encoder and memory as in Sec 2.2) can be named as skill transfer. \n\nThe effectiveness of using this method to learn exploiters is significant. But I am wondering how much computation cost is reduced compared with prior work? If would be helpful to include a comparison. \n\nThe connection to Schaul et al 2015 is interesting but somewhat vague, it would be helpful to elaborate more, do the authors mean that NeuPL can be viewed as UVFA?  \n\nThe underlying RL algorithm is MPO which arguably is not the state-of-the-art RL algorithm, what are the reasons behind using it? Does NeuRL also work with other value-based algorithms? \n\nIt’s interesting that the effective population size plateaus at 12, regardless of the speciﬁed maximum capacities of the neural populations on stochastic games, did this also happen to other domains like Mujoco based environments? ", "summary_of_the_paper": "The authors propose a new framework of  population learning that optimizes a single conditional model to learn and represent multiple diverse policies in real-world games.\n\nExperiments on Mujoco Football and strategic games demonstrate its effectiveness. ", "summary_of_the_review": "A new framework of population training is proposed, the key idea is optimizing a shared conditional network to learn and represent diverse policies and embedding self-play. \n\nThe paper is well presented with just a few confusing points. \n\nExperimental results are strong and have many interesting empirical findings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qS8gUWUkdct", "reviewer": "Reviewer_TfQw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Review**\nThis paper offers two ideas (1) concurrent training of new responses and fine-tuning of previous responses through paramterizing a single policy with the other-agent strategies, and (2) a method for representing population based training algortihms through interaction graphs. The former seems a reasonble way to prevent unnecessary relearning; however, it raises concerns of policy collapse, negative transfer, and questions of what exactly is transferred. Despite concerns, it appears that empirically this a direction worth exploring. The later, is a new lens to view population based training algorithms from, and I believe offers the community a language to discuss algorithms with that has room for fruitful discoveries. \n\n**Major Comments**\n\n - My impression from reading this paper is that the authors are proposing NeuPL as a more \"general class of population learning algorithms\" than PSRO (Sec 1.2). However, this seems very disingenuous because where NeuPL offers a few nicer generalizations it also restricts the class of games that can be studied. The authors need to fairly discuss their work limitations wrt prior work. In particular, acknowledging explicitly NeuPL is restricted to zero-sum symmetric games with fixed player sizes (as the policy's observation space is dependent on the game's number of players), where PSRO is not. Another area for limitations is policy collapse, or gradient interference when co-training dramatically different or contrary responses.\n - The literature review is missing related work and fails to place this work in the context of prior work that is shallowly cited. Last year's ICLR had a work that [1, 2] looked at transferring response knowledge across PSRO epochs, directly relevant to the problems being addressed here. Moreover, P2SRO [3] is shallowly cited when it is co-training across PSRO epochs. I also think it would be advantageous to compare and contrast with Self-Play, directly discussing that the changes are an expanded observation space alongside the matchmaking. Otherwise, using self-play to refer to playing against homogenously parameterized policies must be inferred. This work also has some indirect connections to [4], maybe not worth mentioning explicitly (opponent strategies used at training time instead of inference).\n - Comparing methods using only RPP, a dynamic measurement, may not paint a detailed enough picture to truly understand relative performance of two methods. For example, assuming weak RL, one method may settle into Rock-like policies while another into Paper-like policies. Creating a strict ordering in performance, and then chase each-other around the search space simply due to a staggering of learning. It would be beneficial to include absolute measurements relative to known solutions, or compare relative to a held-out static rich pool of evaluation policies. In other words, the algorithms have no equilibrium selection bias. \n - A major claim that the authors seek to address is that good-responses are being generated instead of best-responses. Have the authors considered that this may in fact be a feature? Empirically, I have noticed that noise introduced can be beneficial for strategy space exploration. Serendipitously, I could not find a direct argument that NeuPL addresses this problem, could the authros please elaborate? I suspect that the continuous training of all policies is the argument, but it's not clear why longer training time actually solves this problem. In my experience the truncation is because the reward marginals for exponentially more training are tiny.\n - I am having trouble reconstructing PSRO from the variation of PSRO presented in Algorithm 2. I suspect this is because \"N\", \"M\", and \"N-step ABR\" are undefined or inconsistently used. M is not ever defined, which I believe is being used here to track PSRO iterations, but no outer loop for PSRO is presented. Is N-step BR meant to refer to training a new policy for each i \\in N player? If so, for symmetric games, as this work considers, only a single BR step is necessary (train i vs -i, and give all players the new policy).\n - How were the hyperparemeters selected for NeuPL and the baseline methods? Exploration schedules in particular seem particularly challenging to define well for NeuPL.\n\n\n**Minor Comments (not impacting review)**\n\n- Sec 0, Par 3, \"... network via self-play\" I think overloading self-play this early on could be very misleading given its established definition.\n- Sec 1.2, Par 2, \"opponent's meta-game strategy \\sigma_i\" I think you want \\sigma_{-i}\n- A major limitation of this work is that policies must be homogenous. So for example, if you wished to initialize the population with a diverse set of known heuristic strategies this would require some changes and not offer the same benefits that could be gleamed from competiing methods. Not that this is a setting realized currently in the literature. \n - Truncation of training still seems necessary for Alg-5, where it in effect represents a trade-off for good-ness of response and cost incurred from operations on the interaction graph. Did the authors explore looking at this dimension at all?\n\n[1] Smith, Anthony, Wellman. Iterative Empirical Game Solving ia Single Policy Best Response. 2021.\n[2] Smith, Anthony, Wellman. Learning to Play against Any Mixture of Opponents. 2020.\n[3] McAleer, Lanier, Fox, Baldi. Pipeline PSRO: A Scalable Approach For Finding Approximate Nash Equilibria in Large Games. \n[4] Foerster, et al.. Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning. 2017.", "summary_of_the_paper": "Population based training (PBT) algorithms progressively grow a set of policies by adding best-responses to mixtures of the existing population. When RL is used as a best-response method the new policy is generally not a best-response but instead a good response. Moreover, the good-response is initialized tabula rosa potentially relearning responses to policies responded to in previous iterations. To address these concerns, this work proposes NeuPL, where a single policy is trained to represent the entire population by conditioning it on the opponent's mixture. Moreover, this proposed algorithm introduces the use of interaction graphs as a means to codify match-making in a continuously training population. NeuPL is restricted to zero-sum symmetric games with fixed player sizes. ", "summary_of_the_review": "Overall I think the paper is sensible to appear at ICLR, but would not fight for it or be upset if it was not accepted.\n\nThe ideas presented within the work are interesting and provide a way to frame PBT methods and could facilitate the design of many future algorithms. These ideas are supported with some empirical demonstrations; however, my concerns regarding: insufficient discussion of place in literature, absent discussion of how hyperparameters are arrived at, and lack of quantitative analysis on the later experiments; leave me with reservations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HQt2dRiQcrt", "reviewer": "Reviewer_kTPT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n1. The idea of representing an entire population of policies within a single conditional model to solve the shortcomings of existing population-based training algorithms is very elegant and novel.\n2. The proposed NeuPL framework is very general and can realize a lot of current mainstream population-based training algorithms.\n3. The ablation study is very detailed, which clearly demonstrate NeuPL’s effectiveness.\n\n**Weaknesses**:\n1. Although this submission is relatively well written, I think it may not be accessible to a wider range of audience since it contains too many concepts without proper explanation in the main text. For example, **a)** in the introduction section, the authors mentioned many concepts, such as game-of-skill, which should be difficult for readers who don't know much about this field to understand before reading the **game-of-skill hypothesis** proposed by [1]. **b)** In the preliminaries section, I think Figure 8 in the appendix is very helpful to understand some core concepts. I suggest putting this figure in the main text and using it as concrete examples to help readers understand these definitions. **c)** In section 1.2, the authors should introduce PSRO in more detail (perhaps in the appendix), because it is difficult for readers to understand NeuPL without PSRO. **d)** In the fourth line of the fourth page, is $E_{\\sigma_i, \\sigma_ j}$ a typo? I think it should be $E_{\\Pi_\\theta (\\cdot| \\sigma_i), \\Pi_\\theta (\\cdot| \\sigma_j)}$. **e)** It is better to redraw all the figures in the experiment section, all the current figures look blurry.\n2. Some important related work is missing, such as [2]. Pipeline-PSRO [3] and [2] are also to accelerate the convergence speed of PSRO. It is best for the author to discuss some differences and connections between NeuPL and these methods. If possible, it is better to add some comparative experiments with [2] and [3] since the baseline PSRO-C is relatively weak.\n\n[1] Czarnecki, Wojciech M., Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafiei, David Balduzzi, and Max Jaderberg. \"Real World Games Look Like Spinning Tops.\" Advances in Neural Information Processing Systems 33 (2020).\n\n[2] Smith, Max, Thomas Anthony, and Michael Wellman. \"Iterative Empirical Game Solving via Single Policy Best Response.\" In International Conference on Learning Representations. 2020.\n\n[3] Mcaleer, Stephen, J. B. Lanier, Roy Fox, and Pierre Baldi. \"Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games.\" Advances in Neural Information Processing Systems 33 (2020): 20238-20248.", "summary_of_the_paper": "This submission provides an integrated and versatile NeuPL framework to improve the performance and convergence speed of population-based training algorithms by representing the entire population of policies within a single conditional policy network. NeuPL is very general, the commonly used population-based training algorithms, such as self-play, fictitious play, and PSRO can be regrade as its special cases by changing the meta-graph solver. The authors conduct extensive ablation experiments on some small games to validate the effectiveness of NeuPL from different perspectives. The results on the large-scale football environment also demonstrate NeuPL’s generalization ability.", "summary_of_the_review": "The authors provide a framework called NeuPL to improve the performance and convergence speed of population-based training algorithms. The authors also conduct extensive experiments to validate the effectiveness and generalization ability of NeuPL. Although some important related work is missing and the writing can be further improved, this paper does solve some key problems in population-based training algorithms. Therefore, I recommend its acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3r84CLl2yAz", "reviewer": "Reviewer_xb7h", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths** \n1. The paper is written well and easy to follow. \n2. NeuPL is interesting, and it also converges to an N-step best response. I also like the idea of using conditional network to represent the population of policies. It could enable skill transfer and speed up the learning. \n3. The empirical study on MuJoCo Football is appreciated especially for population learning. But I recommend the authors should include more details for other researchers to reproduce the results.    \n\n\n**Concerns/Questions**\n1. How efficient is the LP Nash solver in Algorithm 3? Will it be the compute bottleneck when $N$ is large?\n2. It is not clear to me that how the conditional policy can be used in *execution* since it makes $\\sigma$ as input. \n\n**Minor comments**\n1. Some figures are too blur to read.   \n2. Some terms are not clear to me, for example, \"tabula rasa\". \n3. In Figure 8 (the right column), the matrix and figure do not match?  \n", "summary_of_the_paper": "The paper proposes Neural Population Learning which I think extends PSRO in two aspects. First, it avoid the premature *good*-response. Second, it uses a conditional network to represent the population of policies, so as to enable skill transfer.  NeuPL also offers convergence guarantees under some assumptions. NeuPL is empirically verified in *rock-paper-scissors*, *running-with-scissors*, and MuJoCo Football.", "summary_of_the_review": "I think NeuPL is new and interesting. Also, NeuPL is also rigorously studied empirically. Although I have the question above mentioned, currently I feel the strengths of the paper outweigh the weaknesses. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new framework of population learning that optimizes a single conditional model to learn and represent multiple diverse policies in real-world games. All reviewers agree the ideas are interesting and the empirical results are strong. The meta reviewer agrees and recommends acceptance.
Decision: Accept (Poster)