Sociodynamics of Reinforcement Learning
Authors: Yann Bouteiller
Karthik Soma
Giovanni Beltrame
Venue: Accepted by TMLR ()

Abstract: Reinforcement Learning (RL) has emerged as a core algorithmic paradigm explicitly driving innovation in a growing number of industrial applications, including large language models and quantitative finance. Furthermore, computational neuroscience has long found evidence of natural forms of RL in biological brains. Therefore, it is crucial for the study of social dynamics to develop a scientific understanding of how RL shapes population behaviors. We leverage the framework of Evolutionary Game Theory (EGT) to provide building blocks and insights toward this objective. We propose a methodology that enables simulating large populations of RL agents in simple game theoretic interaction models. More specifically, we derive fast and parallelizable implementations of two fundamental revision protocols from multi-agent RL - Policy Gradient (PG) and Opponent-Learning Awareness (LOLA) - tailored for population simulations of random pairwise interactions in stateless normal-form games. Our methodology enables us to simulate large populations of 200,000 independent co-learning agents, yielding compelling insights into how non-stationarity-aware learners affect social dynamics.
In particular, we find that LOLA learners promote cooperation in the Stag Hunt model, delay cooperative outcomes in the Hawk-Dove model, and reduce strategy diversity in the Rock-Paper-Scissors model.

Published in Transactions on Machine Learning Research (02/2026)
Sociodynamics of Reinforcement Learning
Yann Bouteiller
yorn, www.bootteileer@pplynd..o
Deportment of Computer Science Polytechnique Montreal
Karthik Soma
Department of Computer Science Polgtechnique Montreal
Giovanni Beltrame
Department of Computer Science Polytechnique Montreal
Reviewed OpenReview: https: opppreveepeereveeuu net/ forum? id=
Abstract
Reinforcement Learning (RL) has emerged as a core algorithmic paradigm explicitly driving innovation in growing number of industrial applications, including large language models and quantitative finance. Furthermore, computational neuroscience has long found evidence of natural forms of RL in biological brains. Therefore, it is crucial for the study of social dynamics to develop scientific understanding of how RL shapes population behaviors. We leverage the framework of Evolutionary Game Theory (EGT) to provide building blocks and insights toward this objective. We propose methodology that enables simulating large populations RL agents in simple game theoretic interaction models. More specifically, we derive fast and parallelizable implementations of two fundamental revision protocols from multi-agent RL Policy Gradient (PG) and Opponent-Learning Awareness (LOLA)- tailored for population simulations of random pairwise interactions in stateless normal-form games. Our methodology enables to simulate large populations of 200,000 independent learners
co-learning agents, yielding compelling insights into how non-stationarity-awis affect social dynamics. In particular, we find that LOLA learners promote cooperation in the Stag Hunt model, delay cooperative outcomes in the Hawk-Dove model, and reduce strategy diversity in the Tock-Paper-Sciseeee model.
1 Introduction
Our universe is one of perpetual change, where countless agents co-exist and co-learn. From an individual agent's perspective, other learning agents are a fundamentally non-stationary part of the environment, es- pecially when incentives are in conflict (Papoudakis et al. 2019). In the realm of Reinforcement Learning (RL), the study of this possibly adversarial non-stationarity is called Multi-Agent Reinforcement Learning (MARL) (Tan 1993 Foerster 2018 Zhang et al., 2021). Yet, MARL largely constrained by the high complexity of multi-agent training. Its applications often resort to "self-play" (Silver al. 2016 [Bener et al. 2019 i.e., training single neural network against one few copies of itself, or to "centralized training (Lowe et al. 2020 Yu et al. 2022 ie., circumventing non-stationarity by using privileged global information. We are instead interested in dccentralized, biologically plausible MARL processcs that may help us undersland the macroscopic social dynamics stemming from individual agents pursuing their own in- centives in the real world. This paper is a step toward achieving this objective: we simulate large populations of RI agents pursuing individualistic incentives classic stateless interaction models. At cach population step, agents are paired by some assortment process (in our case, uniformly at random), interact according to their respective policies, and adapt their policies according to their respective payoffs and learning rules.

---

Published in Transactions on Machine Learning Research (02/2026)
Our work brings advanced MARL to the scale of population game simulations. In particular, we study the population-scale effects of Opponent-Tearning Awareness, an advanced MARL, paradigm able to take advantage of non-stationary dynamics (Foerster et al. 2017). Our contribution is threefold:
We propose methodology for simulating RI-driven social dynamics in statcless interaction models,
We derive tractable implementations of exact Policy Gradient (PG) and Learning with Opponent- Learning Awareness (LOLA) tailored for pairwise stateless population games;
We analyze how learning shapes collective behavior in simulated populations of 200,000 agents.
2
Related work
Our paper constitutes a new step toward modeling the last-paced macroscopic social dynamics that stem from individuals actively optimizing their own payoffs in real-world economies. We trace our line of work back to Macy & Flache (2002), who first substantiated the claim that the emphasis on the stochasticity- driven model of genetic evolution long used by evolutionary biologists to explain the evolution of cooper- ation (Axelrod & Hamilton 1981 h may need to shift. to cognitive-driven dynamics. Concurrently, Sato & Crutchfield (2003) derived coupled replicator equations to describe how naive RL shapes social dynamicc in a 2-player repcated Rock-Paper-Sore Paper-Scissors interaction model. However, later attempts have remained largely constrained to imitation-driven dynamies (Sandholm 2010 Xia et al., 2011 Presumably, this restriction driven population dynamiic and by the strong heritage
can be explained by the high complexity of of replication-drrien evolutionary genetics. The mathemattcal framework typically used to model population dynamics is Evolutionary Game Theory (EGT) (Smith & Price 1973) , which comes with a large body of imitation-based literature. Mertikopoulos & Sandholm (2018) have recently extended this framework by formalizing population dynamics (such as the Replicator Dynamic) under the more general class of Rieman- nian game dynamics. Further, Yang et al. (2020); Gu al. (2021): Hu & Zhang (2025) used mean-field theory in a massively multi-agent RL scenario to reduce the environment dimensionality. Typically, in these approaches, naive learners approximate neighboring agents as one single, 'mean-field" opponent. While mean-field methods are indeed relevant to studying large-scale population dynamics, they essentially trans- form n-agent MARL into pairwise MARL, thus reducing the environmental complexity from the point of view of individual agents. Alternatively, Christianos et al. (2021 reduced the complexity of MARL settings by selectively sharing parameters among agents, as a practical trade-off between self-play and independent learning. In contrast, our work explicitly tracks full populations of 200,000 independent RL policies by making simulation tractable in simple evolutionary game-theoretical interactions.
3 Background
This section briefly reviews the concepts from Evolutionary Game Theory (EGT) and Multi-Agent Rein- forcement Learning (MARL) used in this paper. Throughout the paper, we interchangeably use vocabulary from EGT and MARL, whose correspondence is provided in the Appendix (Table 2
3.1 Evolutionary Game Theory (EGT)
EGT (Smith & Price 1973h models evolution as a series of random pairwise interactions, where interactions are typically simple bi-matrix games (i.e., 2-player multi-armed bandits). Agents are sampled from a large population to be randomly paired and evaluated against their drawn opponent. The outcome of this in- teraction is payoff for each opponent, whose expectation is called the agent's fitness against the current population. An agent's fitness depends on both its policy and the current configuration of the population. The agent's policy is called its lype, which is one of n possible lypes. In gene-inspired revision prolocols, agents with a greater fitness replicate and thus tend to "invade" the population, whereas agents with a lower fitness tend to go "cxtinct".. In particular, EGT is interested in evolutionarily stable equilibria, which are configurations of the population where the different types are present stable proportions under replication dynamics. In evolutionarily stable equilibria, the population configuration is robust to rare mutations, where
2

---

Published in Transactions on Machine Learning Research (02/2026)
few agents randomly switch from one type to another. In this paper, study more complex population dynamics generated by learning-based revision protocols (as opposed to revision protocols modeling genetic replication or social imitation) in three classic symmetric interaction models from game theory: Stag Hunt, Hawk- Dove, and Rock-Paper- Paper-Scissors These interaction models are the following bi-matrices:
Rock
Paper
Scissors
Stag
Hare
Hawk f.f
Dove
Rock
0,0
-1,1
1,-1
Stag
8,8
0,1
Hawk
2,0 1,1
Paper
1,-1
0,0
-1,1
Hare
1,0
1,1
Dove
0,2
Scissors
-1,1
1,-1
0,0
(a) Stag Hunt
(b) Hawk- Dove
(c) Mock-Paper-Scissee
where rows represent the action chosen by the ego agent (bold) with corresponding payoffs in first position, and columus represent the action chosen by the other agent with corresponding payolfs in second position. and f are cost. parameters, whose influence is explored in Section 5 The significance of these interaction models (or games) is further described in Appendix B for the reader unfamiliar with EGT.
A population whose individuals are distributed amongst types can be represented a population vector P E IR" whose components 0 S 2; < 1 sum to 1 and represent the proportion of type i. Under the "imitation of the fittest" revision protocol (as well as several other revision protocols), large populations are known to follow a famous population dynamic over time (t), called the Replicator Dynamic:
dp; (u((() - e(P)) dt
(1)
where (P) is the fitness of type i in the population, and ū( P) is the average fitness of all agents in the population. Denoting the vector of g,'s 's as Q, the vector of all ones as 1 and the Hadamard product] as O Equation can be written in matrix form:
d P PO Q 17)
(2)
di
3.2 Multi-Agent Reinforcement Learning
We examine populations of persistent agents actively learning and optimizing their own policies the pursuit of their own individualistic incentives. Following classic modeling of large populations in EGT, we model social dynamics learning rules. In this paper, we will be specifically looking at two important such learning rules: policy gradient (PG), also referred to as "naive learning" in the MARL literature, and learning with opponent- learning awareness (LOLA) in its true form (i.c., using all terms from the first-order Taylor expansion).
series of random pairwise interactions. Therefore, we are principally interested in 2-agent
Policy gradient (PG) Willliams) 1992; Sutton et al. 1999 is a fundamental learning rule from single-agent RL. It follows the first-order gradient of the value function with respect to the ego agent's policy parameters. Let us consider a pair of agents. We denote the ego agent as agent 1, and the other agent as agent Let further denote their respective policies P1 and Pz, parameterized by vectors 0, and 022 of current values ii and 02 The naive policy gradient is:
The reason why following this gradient is considered naive in MARL that this does not take into account the non-slationarity introduced by the learning process of agent 2.
Learning with opponent learning awareness (LOLA) FFoerster et al. 2017 is an improved version of PG that takes into account the lcarning process of the other agent. More preciscly, LOLA models the learning process of agent as if agent 2 were a naive learner, and differentiates through its learning step:
V 0 0 + A02)
(4)
The matrix product takes precedence over the Hadamard product in all our notations.
3

---

Published in Transactions on Machine Learning Research (02/2026)
where A02 U2222 82) is the naive learning step of agent 1) being its learning rate. LOLA approxi- mates this gradient with the following first-order Taylor expansion (Taylor 1717
Va ui (01, + A02) Va, (v (A02) V V
Vo 7 Ve, V1) V V8, D2 + VV. Vo,02) PG
T Ve.01
(5)
apponent-learning awareness
Difforentiating through the learning step of the opponent has important advantage in our discussion: it is naturally plausible way of predicting non-stationarity (assuming maintain internal model of others) in order to adapt beforehand and actively steer this non-stationarity toward our own incentives.
3.3 Population-poiie policy equivalence
In population of agents playing only pure strategies, uniformly sampling agents is equivalent to sampling actions from the abstract stochastic policy defined by the probability vector /' = P. Thus, Equation 2 can be viewed as a learning process, albeit at the population level, where the evolving population of non-learning agents P itself a self-play learning agent (Bloembergen et al. 2015
4 Methods
To simulate how learning affects societies, we adopt the philosophy of EGT. Namely, we consider large populations of independent learning agents, which are paired randomly at each evolution iteration and interact in normal-form matrix games. Each agent has its own learning rule (i.c., either of the two presented in Section 3.2) that it applies to its own policy afler each pairwise interaction. Whereas MARL usually thinks about these rules in the context of persistent interactions between fixed pairs of agents, in the context of population games, they instcad get applicd after single interactions between random pairs of agents. other words, from the perspective of a learning agent, the opponent samples at each step is a stochastic sample of the population and, for T.OTA, of the current direction of its non-stationary dynamics.
4.1 Policy architecture
From an RL. perspective, the normal-form matrix games presented Section are 2-agent multi-armed bandits. As this is common assumption in multi-armed bandits (Sutton 2018 we consider the policy architecture parameterized by the preference vector R", where is the number of actions, projected to the probability simplex by simple softmax function Ø, which yields the probability vector P e R" of the policy selecting each of the n available actions:
P= Po)
(6)
This policy architecture has a useful property for our derivations: its gradient has symmetric analytical form, which
TPP=(Vol diag(P) PP
4.2 Analytical Policy Gradient
Let us consider a symmetric normal-form game with n actions, played by pair of agents denoted as agents and 2. Since the game is symmetric,
represent its bi-matrix single matrix A R=x=, valid from the perspective of both agents?] Let us further assume that the policies of both agents are parameterized by 022 E IR", with the simple policy architecture described in Equation 6
P1.2 =
(8)
The value functions of both agents are:
v = PT 2/2 = P. AP,
(9)
2 formed by the first entries of the corresponding bi-matrix in Section
3.1
4

---

Published in Transactions on Machine Learning Research (02/2026)
Thus, the "naive" Policy Gradient of agent I's value with respect to agent I's parameters is:
Vo, U1 Pi @ (Q1 - 1s1)
where Q1 E I" is the vector of action- values of the n available actions (derivation Appendix D
As a side note, this draws an interesting parallel with Equation 2 the PG update on the parameter vector 0 is the same as the Replicator update the probability vector Pi - o01)). In other words, each individual PG agent can itself be seen evolving population of abstract non non-learning agents playing pure strategies, similarly to the equivalence noted in Bloembergen et al. (015))
This formulation yields the following analytical PG formulation for symmetric normal-form games:
Ve, 01 = P1 @ -I 1P;)) V 0. P2 3 (I - )AP
(10)
AP
(11)
which only involves simple matrix operations, and therefore is a fast, easily parallelizable implementation. ion.
4.3 Analytical LOLA
Similarly, we derive the following analytical formulation of the LOLA gradient in the symmetric normal-fon game defined by matrix (the derivation is provided in Appendices and
Vo, vi (01, O2 + A02) ~ P1 @ X,AP2
+ (T" @ XIAX, (P2 @ X2AP))
(T XIA O
(12)
where X1 / IPT, X2 = T- 1P and T P2PTT. As for Policy Gradient, this formulation only involves simple matrix operations and is thus straightforward to parallelize.
4.4 Batched pairwise bandits
ego
shuftling
parling
minoring
populal on
2
2
2
Figure 1: Pairing and batching. The population is shuflled and split to produce random pairwise matching. The resulting tensor is then mirrored such that the entire population step can be performed in parallel.
Equations 10 and 12
fast, backprop-free implementations of exact PG and exact LOLA for symmetric normal-form games. Still, it would be prohibitively slow to apply these updates itcratively on single agent pairs in large-scale simulations. At each population slep, all agents go through their revision prolocol once. To make this process scalable, we batch learning updates across the entire population. More precisely, at each population step, we shuffie the entire population and randomly pair all agents two-by-two. Since all interactions are pairwise, this enables batching updates across agent pairs. To optimize the process even further, mirror all pairs to perform the entire population update in one single batched operation.

---

Published in Transactions on Machine Learning Research (02/2026)
0.030
population update duration (s)
0.030
population update duration
autograd iterative LOLA
0.025
0.025
autograd batched LOLA
0.020
0.020
matrix batched LOLA
0.015
0.015
0.010
autograd iterative LOLA
0.010
autogred batched LOLA
0.005
0.005
matrix batched LOLA
0.000
0.000
0.0
0.5
1.0
1.5 2.0
0.0
0.5
1.0
1.5
2.0
population size
1e5
population
(a) CPU
(b) GPU
Figure 2: Duration of a full population step (lower is better). We benchmark different implementations of analytical LOLA. Both batched approaches implement Section 14 and the matrix approach implements Section
4.3 The matrix batched approach's 'ss efficiency enables full simulations of very large populations.
We found batching populations in this manner to be extremely efficient. Combining this procedure with the analytical PG and LOLA implementations described in Section are able to simulate populations of 200,000 learning agents for thousands of population steps in matter of seconds on a consumer-grade GP To ensure reproducibility and foster future work, we open-source our our code at
This batching procedure is illustrated in Figure [
5 Experiments
We the normal-form matrices presented in Section 3.1 as three explored interactions: Stag hunt (SH), Hawk-Dowe (HD) and Bock-Paper-Seiiee (RPS). Each individual agent has its own persistent learning rule: either PG (gradient ascent on Equation [0) or LOLA (gradient ascent on Equation 12) We use unit learning rates in all experiments except when stated otherwise.
5.1 Scalability
Figure reports the computational performance of our approach (*matrix batched LOLA"), compared to two baselines. The "autograd iterative LOLA" baseline reproduces how LOLA updates are usually performed in classical MARL scenarios: using PyTorch'' 's autograd to compute the LOLA gradient, and updating the policies of all agents iteratively. This baseline is clearly not a viable implementation for large population simulations and is only provided for illustration. On the other hand, our "autograd batched T.OTA" " baseline is of more interest for future work. While the "matrix batched" approach is significantly faster, it is limited to single-shot multi-armed bandits In particular, the "matrix batched" approach does not allow episodic interactions. Therefore, have implemented the batched approach described in Section 4.4 along with autograd, which yiclds potentially more general implementation. Since our matrix-based implementation is the fastest for normal-form games, we use it in the remainder of the paper.
5.2 Empirical results
The methodology proposed in Section 4 limited to simple, single-shot bandit interactions between random pairs of learning agents. Remember how, under the population-policy equivalence described in Section 3.3 population of pure-strategy agents can equivalently be seen as stochastic policy over types. In our setting, agents have full stochastic policies assigning non-zero probabilities to all available actions, but they are still simple stateless multi-armed bandits. Uniformly sampling a random pair of agents from such a population and then sampling from their policies equivalent in expectation to sampling two actions from the average policy of the entire population. In other words, we expect our modeled population dynamics show some
3 All experiments this paper conducted with an 17- 17-22 CPU, an RTX 3080 Ti GPU, and 64G RAM. 4 Seccion 4 possible because the value function has straightforward formulation 2-agent multi-armed bandits.

---

Published in Transactions on Machine Learning Research (02/2026)
resemblance to self-play over the population's average policy (represented in Figure 3 In Figure report the results of our actual population experiments in Stag-Hunt, Hawk-Dove and KookkPPaper-Sone Scissors. Due to the large number of simulated agents, the reported population dynamics are -ear-deterministic..
Stag
Hawk
200
150
125
100
Hare
Dove
25
50
75
100
25
50
100
leaming step
learning step
(a) PG SH (s = 1.8)
(b) PG HD (ʃ = 2)
(c) PG RPS
Stag
Hawk
88
60
Hare
Dove
25
75
100
0
25
50
75
100
learning step
learning step
(d) LOLA in SH (s = 1.8)
(e) LOLA in HD (f =
(f) LOLA in RPS
Figure 3: Evolution of a self-play agent following Policy Gradient or LOLA (rows) in the classic games of Stag Hunt, Hawk-Dove and Bock-PPpperScissoor (columns). In Stag Hunt and Hawk-Dove, colors mark the agent's initial policy (yellow for deterministic Stag/ Hawwk and blue for deterministic Hare/Dove). In Stag Hunt, gray dashes mark the uniform policy and red dots highlight the critical initial policy.
Stag Hunt. Figures 3a and 3d show how a single sclf-play agent learns against itself in Stag Hunt, via naive Policy Gradient and LOLA, respectively. The vertical axis represents the agent's policy and the horizontal axis represents time expressed in learning steps. Policies are color-coded by their initial configuration: yellow represents policies that start close to the deterministic Stag policy and purple represents those that start close to the deterministic Hare policy. Notably, PG tends to converge to the individualistic Nash equilibrium (i.e., deterministic Hare) for most initial configurations, whereas T.OTA tends to converge to the pro-social Nash equilibrium (i.e., deterministic Stag). Notice that the forks (red dots) are on either sides of the uniform random policy (middle tick), which is important because we will initialize all our population experiments with a Gaussian distribution around this neutral policy. We expect the final population dynamic to follow similar pattern. Figures 4a and 4d report the results our first population game simulation, featuring 200,000 learning agents in the Stag Hunt interaction model. Dark shades of blue represent high concentrations of agents. lighter shades represent lower concentrations. A Population steps correspond to one learning step performed per agent in the population. In Figure 4a the population is exclusively formed of naive learners, and quickly converges to the individualistic policy, as predicted by Figure 3a. In Figure 4d the population is formed of only LOLA agents. Contrary to the naive population, a population of non-stationarily aware learners such as LOLA evolves to unanimously adopt the superior pro-social equilibrium (i.e., deterministic Stag). This effect is modulated by the payoff of the pro-social stratogy, as shown in Figure 5a] In Appendix G we further show that, when enough opponent-learning aware learners are present in a mixed population, they become able to pull naive learners toward the pro-social strategy.

---

Published in Transactions on Machine Learning Research (02/2026)
Stag
Hawk
1400
1200
1000
800
400
200
Hare
50
100
150
200
Dove
50
100
150
200
population step
population step
(a) PG in SH (s = 1.8)
(b) PG in HD (f = 2)
(c) PG in RPS
Stag
Hawk
100
incusstton step
63
40
20
Hare
50 100
150
200
Dove
50 100
150
200
population step
population step
(d) LOLA in SH (s 1.8)
(e) LOLA in HD = 22)
(L) LOLA in RPS
Populations of 200,000 RL agents in the Stag Hunt, Lawk-Dove and Rock-Paper-SSissorr in- teraction models (columns), individually learning via Policy Gradient and LOLA (rows). Each agent is a stochastic policy, represented as linear coordinates between pure strategies. Populations are randomly ini- tialized with a Gaussian around the uniform policy. Dark shades of blue indicate high concentrations of agents, and population steps correspond to one learning step performed per agent. Hawk-Dove, black dots indicate the average policy over the entire population.
Figure 4:
Bawk-Dovee Figures 3b and 3e show that, in Have-Dove, self-play converges to definite policies regardless of where training starts from. Naive learning (Figure 3b) converges to the mixed Nash equilibrium. However, LOLA (Figure Bel converges to another, inferior policy, where selects Hawk 70% of the time (which yields a smaller payoff for both players, and is not Nash equilibrium). similar behavior has been described as "arrogance" in Letcher al. (2018), where both LOLA learners make wrong assumptions about the response of their opponent and thus pull away from the equilibrium.
From these observations, one could imagine that all learning agents in the population would converge to these policies, similar to what we observed for SH, but this is not at all what happens in practice. In HD, whether naive learning (Figure 4b) or LOLA (Figure 4ep is used as the learning rule of the entire population, evolves into a mix of Hawks and Doves, most of them with close-to-deterministto policies, and in both cases with an average population policy that corresponds to the mixed Nash equilibrium.
Notice that the convergence deterministic strategies is however much slower than what observed for SH, and it is in fact not clear whether this will eventually happen entirely, even after 10,000 steps (Appendix H.. Nonetheloss, what can be observed from Figures 4b and Sal is that LOLA learners converge laster to deterministic policies during early steps (shades of blue) but the population takes longer to stabilize (dotted black). In additional experiments, we mixed LOLA and PG learners to see whether LOLA learners would be more inclined toward the Hawk strategy (as suggested by Figure 3cb. However, these experiments invalidated this hypothesis: half LOLA learners and half PG learners were present amongst both final sub-populations of Hawk-inclined and Dove-inclined individuals.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Hawk
Stag
LOLA
PG
LOLA
PG
Hare
1.50
1.75
2,00
2.25
2.50
110.0
-7.5
-5.0
-2.5
0.0
5
(a) Stag Hunt
(b) Hawk Dove
Figure 5: Final average policy over the population, depending on cost values. In Stag Hunt, populations of LOLA agents deterministically cooperate more often compared to populations of PG agents, whereas in Hawk-Dove the final average policy depends only on f.
Rock- Paperr-Ss----sssssss. Our final population experiment explores the 3-action Mock-Paper-Scissoon interac- tion model, used in EGT to explain the coexistence of competitively unbalanced species (Allesina & Levine 2011). We show that, while populations of naive learners agree with this explanation of sustainable di- versity, populations of opponent-learning aware learners yield the opposite result. Similarly previous sections, Figures 3d and [f] report how 2-agent self-play behaves for both PG and LOLA. Only one policy is displayed for readability (other initial conditions yield similar effects). The triangle on the bottom of each plot represents the policy, and the vertical axis represents the number of learning steps. It can be seen that PG slowly spirals outward from the mixed Nash equilibrium (due to performing straight policy updates with a non-zero learning rate following a circular vector field), whereas LOLA quickly spirals inward until it reaches the mixed Nash equilibrium. Figures 4c and present the results of our population simulations in the RPS interaction. Color-coding follows the same principle as our previous population plots, with the bottom triangle being the policy simplex, and population steps being the vertical dimension. Similarly to Figure 4b Figure Ic shows that populations of naive learners evolve into 3 equally distributed groups of close-toddetermnnstii agents always playing Rock, Paper and Scissors respectively. The reason why this happens is of interest, and is more clearly understood from Figure 10 in Appendix
| In short, this dynamic results from the loss of plasticity introduced by the softmax function I Initially, naive learners are erratically moving around as they encounter all types of strategies. However, after some time, agents get "trapped" near the border of the policy simplex, where gradients toward the opposite action are near-zero. After long time, three groups of near-deterministic agents emerge, and small number of them continuously escape toward the strategy that counters the majority group, which eventually creates a new majority, and so on, yielding cyclic social dynamic. In other words, diversity emerges from populations of naive learners in the RPS model. On the other hand, Figure 4i tells the opposite story about populations of LOLA agents, which instead quickly and unanimously converge to the mixed Nash equilibrium.
5.3 Influence of learning rates
When ascending the gradients described in Sections 4.2 and 4.3 RL agents must select learning rates to scale policy updates. PG agents require single, first-order learning rate and LOLA agents require additional, second-order learning rate 17. In our work, both learning rates are constant and shared across the population. So far, we have only considered unit learning rates = 1. However, a puzzling effect emerges in the Stag Hunt interaction model when varying a and independently. Figures bd] and Gb report the strategy selected on average by populations of LOLA agents in Stag Hunt and Hawk Dove, depending on a and 17. Similar to Figure 4 populations are initialized with a Gaussian around the uniformly random policy. Figures [a and Ge report the policy to which self-play LOLA agent converges under the same conditions, when initialized at the uniformly random policy. Intuitively, in LOLAA, the first-order learning rate scales the ego policy update whereas the second-order learning rate n scales the simulated policy
5 A model similar to the "cost of motion" described by Mertikoponlos Sandholm 2018

---

Published in Transactions on Machine Learning Research (02/2026)
opponent-leariing awareness vanishes and LOLA becomes PG. This is consistent. with what we observe for Hawk-Dove: Figure 6c shows the gradual effect of arrogance as increases, while Figure 6bbshows that this effect vanishes in large populations, as we previously observed in Figure Ae However, Figures 6a and nd unveil a much more surprising phenomenon: whereas only ?) has significant effect on the final policy of self-play agent in Stag Hunt, the first-order learning rate a has a significant combined effect with rj in large populations. We do not yet fully understand this phenomenon, and therefore leave its explanation an avenue for future work.
update of the opponent. When 1) is close
0.92
Stag
Stag
0.5558
0.90
final average policy
0.5556
final policy
0.88
0.86
0.5554
0.84
0.5552
0.82
0.5550
Hare
Hare
2
(a) LOLA play in SH (s 1.8)
(b)
LOLA population in SH (s 1.8)
Hawk
Hawk
final average policy
2.0
2.0
final policy
1.5
1.5
1.0
1.0
0.5
0.5
0.0
0.0
Dove
Dove
2
(c) LOLA self-play in HD (f 2)
(d) LOLA population in HD -2)
Figure 6: Influence of the first and second-order learning rates and in Stag Hunt and Hawk-Dove, Surprisingly, in Stag Hunt, a influences the strategy adopted by populations of LOLA agents, even though this influence does not exist in self-play. In Hawk-Dove, the average strategy adopted within the population is unaffocted by learning rates, unlike that of self-play which is directly affected by r due to "arrogance".
5.4 Limitations and future work
Random versus structured assortment. Our simulations can mislead the reader into concluding that the mean policy averaged over the entire population always converges near a Nash equilibrium of the game, even when the learning rule itself does not converge to this equilibrium in the conventional 2-agent MARL setting (see Figures 3e and Bcb. This property however merely a consequence of the uniform random opponent matching scheme that we chose to implement in this paper. For instance, let consider an extreme opposite scheme, where all pairs would instead interact persistently. All individual pairs would then converge to the pure Nash equilibrium in the Hawk-Dove game (that is, exactly one deterministic Hawk and one deterministic Dove per pair): this would average to a uniform random policy, as opposed to Figures 4b and 4e (dotted black lines). In reality, partner selection is more structured (Anastassacos el. al. 2020 and can lead to different outcomes. Extending our study to structured assortment an avenue for future work. Statcless environments. Our proposed approach is limited symmetric normal-form matrix games (i.e., stateless multi-armed bandit interactions). Exploring stateful episodic interactions (like the Iterated Prisoner's 's Dilemma) through more compute-intensive approaches is another clear avenue for future work.
10

---

Published in Transactions on Machine Learning Research (02/2026)
6 Conclusion
We have presented a methodology enabling large-scale population simulations of independent learning agents, for both naive (Policy Gradient) and advanced non-stationarity-awaa (LOLA) learning rules. We have demonstrated the scalability of our approach by performing very-large-scale simulations of 200,000 indepen- dent Icarning agents, interacting in the classic games of Stag Hunt, Hawk-Dove and Rock-Paper-Seisoonn work essentially explores the effect of Multi-Agent Reinforcement Learning on classic evolutionary game the- oretical models of social dynamics, and demonstrates compelling dynamics originating from both naive and non-stationarity-awaa learners. For instance, depending the nature of the interaction, opponent-aware learners foster cooperation, delay cooperative outcomes, or inhibit diversity in the population.
Our
Broader Impact Statement
This work paves the way toward understanding and possibly exploiting the macroscopic effects of individual learning contexts such as biology, sociology, economy and finance.
References
Stefano Allesina and Jonathan M Levine. A competitive network theory of species diversity. Proceedings of the National Academy of Sciences, 108(14):5638- 5642, 2011.
Nicolas Anastassacos, Stephen Hailes, and Mirco Musolesi. Partner selection for the emergence of cooperation multi-agent systems using reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34:7047-7054, 04 2020. doi: 10.1609/aaai v3000// v3-100.6199.
Robert Axelrod and William D Hamilton. The evolution of cooperation. science, 211(4499) 1981.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arxiv: 1at.::912.0000. 2019.
Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. Evolutionary dynamics of multi-agent learning: A survey. Journal of Artificial Intelligence Research, 53:659 016599-67,-0 2015.
Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht. Scaling multi- agent reinforcement learning with seleetive paramcter sharing. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of PMLR, 18 24 Jul 2021. URL
Machine Learning Research, pp.
mlr.
J Foerster. Deep multi-agent reinforcement learning. PhD thesis, University of Oxford, 2018.
Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mor- datch. Learning with apponent-learning awareness. arXiv preprint arXiv: 04336,, 2017.
Haotian Gu, Xin Guo, Xiaoli Wei, and Renyuan Xu. Mcan-field controls with lelannnnn for cooperative marl: Convergence and complexity analysis. STAM Journal on Mathemalics of Data Science, 344) 1196, 2021. doi: 10.1137/20M1360700 URL http::://doi.rrgeee 1tt.:1111/220000771
Daniel Hennes, Dustin Morrill, Shayegan Omidshafici, Rémi Munos, Julien Perolat, Marc Lanctot, Audrunas Gruslys, Lean-Baptiste Lespiau, Paavo Parmas, Edgar Guzmán, et al. Neural replicator dynamics: Multiagent learning via hedging policy gradients. In Proceedings of the 19th international conference on autonomous agents and multiagent systems, pp. 492-501, 2020.
Anran Hu and Junzi Zhang. Mf-oml: Online mean-field reinforcement learning with occupation measures for large population games. 2025. doi: 10.1007/500245--05- 10328-5. URL https:///aoiorrg/1.... 1007/ 200245-025-10228--
11

---

Published in Transactions on Machine Learning Research (02/2026)
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. arXiv preprinl Xin: 181 1.08460.. 2018.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments, 2020. URL
Michael W Macy and Andreas Flache. Learning dynamics in social dilemmas. Proceedings of the National Academy of Sciences, (9(suppl_3) (3::7229-7233333 2002.
Panayotis Mertikopoulos and William H. Sandholm. Riemannian game dynamics, 2018. URL https:
maxxiv.org/bb//100.0022
Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V. Albrecht. Dealing with non- stationarity multi-agent deep reinforcement learning, 2019. URL httpp:://arxv...og//bbbbaeeeeeee 04737
William Sandholm. Population games and evolutionary dynomics. MIT press, 2010.
Yuzuru Sato and James P Crutchfield. Coupled replicator equations for the dynamics of learning multia- gent systems. Physical Review E, 67(1):015206, 2003.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwicscr, Ioannis Antonoglou, Veda Pannocrshelvam, Mare Lanctot, al. Mastoring the game of go with deep neural networks and tree search. nalure, 529(7588)):444 4499, 2016.
Maynard Smith and George R Price. The logic of animal conflict. Nature, 246(5427):15 18, 1973.
Richard S Sutton. Reinforcement learning: an introduction. A Bradford Book, 2018.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 3330-37, 1993.
Brook Taylor. Methodus incrementorum direcla & inversa. Inny, 1717.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning, 8(3):229-255) 1992.
Haoxiang Xia, Huili Wang, and Zhaoguo Xuan. Opinion dynamics: A multidisciplinary review and perspec- tive on future research. International Journal of Knowledge and Systems Science (IJKSS), 2(4):22 91, 2011.
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning, 2020. URL https://arxiv wwwp////////.roooo///////////o http:://aaxvvorr//bb//B82..000 05438
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising offoctiveness ppo in cooperative, multi-agent games, 2022. URL httpps//aaxxv.oorr//////000
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321 384, 2021.
12

---

Published in Transactions on Machine Learning Research (02/2026)
A Vocabulary
RL
EGT
return
payoff
value
fitness
policy action
type
pure strategy
learning rule
revision protocol
Table Correspondence of the vocabulary from RL and EGT
B
Games
Rock
Paper
Scissors
Stag
Hare
Hawk
Dove
Rock
0,
-1,1
1,-1
Stag
8,
0,1
Hawk f.f
2,0
Paper
1, 1,-1
0,0
-1,1
Hare
1,0
1,1
Dove
0,2
1,1
Scissors
-1,
1,-1
0,0
(a) Stag Hunt
(b) Hawk-Dove
(c) Rock- Mock-PPapr-Sciisoors
Stag hunt (SH) 2-action game modcling cooperative dilemma. In this game, agents need to hunt for food and choose to either go for Stag, or go for Hare. Hunting Hare is easy: any agent choosing this option successfully receives payoff of 1. Hunting Stag is harder: both agents need to cooperate, otherwise the agent choosing to go for a Stag fails to catch anything and receives payoff of O. However, if both agents cooperate, they succeed and each receives a payoff > 1, which better than going for Hares. The game of Stag hunt has two distinct. pure strategy Nash equilibria [ (1) both agents always playing Stag, and (2)
both agents always playing Hare.
Hawk Dove (HD) is 2-action game modeling confiict over shareable resources. Agents either choose to act "Hawk" or as a "Dove". When a Dove encounters another Dove, they share the available food (each receives a payoff of 1). When Dove encounters a Hawk, it yiclds and gets no food (payoff of 0) while the Hawk gets all of it (payoff of 2). But when a Hawk encounters another Hawk, they fight and both get injured (payoff of f < 0). The game of Hawk-Dove has two pure strategy Nash equilibria: (1) Hawk- Dove and (2) Dove-Hawk. But note that in these equilibria, Doves have a smaller payoff than Hawks. In the context of evolutionary genetics, this means that when Doves encounter almost only Hawks, they move toward extinction as Hawks invade. However, when Hawks encounter almost always Hawks, their expected payoff is even less than Doves encountering Hawks, and thus Hawks move toward extinction Doves invade. In other words, there are population confiuuratioss in which it is not more relevant to be Hawk than a Dove in terms of fitness, and replication dynamics naturally drive the population there.
(RPS) is a 3-action zero-sum game. illustrates more complex situations where there are cycles in the preferences over actions. Scissors beats Paper, Paper beats Rock, and Rock beats Scissors. RPS has a single mixed-Nash equilibrium, where both players choose their actions uniformly random. Similarly to the HD game, all populations whose average behavior is this equilibrium neutrally stable under replication dynamics and drifting due to random mutations.
Boock-Paper-Sciseee
C Parallels between EGT and single-agent RL
The population-pon policy equivalence described by Bloembergen et al. (2015) yields interesting parallels between single-agent Reinforcement Learning and the Replicator Dynamic. In particular, the resemblance of Policy
Gradient with the Replicator Dynamic noted in Section 4.2 was further studied by Hennes et al. 2020 From
62 Pl-plaeer equilibria where each agent always selects the action.
The sum of the agents' payaffs always O.
13

---

Published in Transactions on Machine Learning Research (02/2026)
this observation, they derived single-agent algorithm that bypasses the loss of plasticity introduced by the softmax architecture of Equation 6 However, beyond the fact that their line of work uses concepts from RL and EGT, it is unrelated to ours and we cite it here to clear a confusion made by early reviewers of our work: whereas Hennes et al. (2020) are interested in finding high-performance singlo-agent RL algorithms, we are interested in characterizing the population dynamics that stern from aggregated MARL.
Policy gradient
We derive an analytical formulation of the PG update symmetric normal-form games:
Ve, 21 = Vo, P. AP2
(diag( P1) PIP )AP
diag(Pi)AP3 - Piu
Pi @ (AP2 - -211)
Pi @ (AP; - 1P, AP2)
Pi @ (Q1 - 1e))
E LOLA
We now derive an analytical formulation of the LOLA update in symmetric normal-form games, similar what we found for PG in Section 4.2 We missing three terms from Equation 5
V V1 U1
Vo, V 02 U1
Ve, Vo,
To compute the first lerm, note that eı PTT AP2 is scalar and thus can also be written v1 Ps AT P.
We can then compute this term similarly to PG:
Vagei Vo, AT P1
= (diag( P2) - )
P2 (A P1 -v1))
P (ATP, -PPTATR)
= = (1 - DARR
(13)
Computing the two remaining terms is also possible. Let us start with Vo, U2:
V Vo, 02 = Vo, Vo, PI P AP
(14)
Ve, (diag (P2) P2P.7 APP
(diag(Pz) P2) P.P, A(diag( PPPT)
(15)
While it would already possible to implement this formulation, we further derive a more efficient implemen- tation in Appendix
VA, - P(1
(16)
where T P.P.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Computing Vo Volu1 is fairly straightforward:
Vo, Vo Vo, Vo P2 P
= Vo, Vo P2 BP
(B = A
= =T - 1P2 B - )
(c.f. 14 16
(17)
T 1Pg A A (
(18)
Substituting Equations
10
13
16 and 18
Equation yields the following analytical formulation of the
LOLA gradient in the symmetric normal-form game defined by matrix A:
Va, 1i (0 + A02)
PI XIAP
+ (T @ X,AX, (P2 O X2AR)) + tn @ AAA x, (P, X2A P1)
(19)
where X1 = I- 1PJ, X2 = I 122, 1 and T P2P.
F Second-order policy gradients
In this Section, we show that:
V Vo,22 =o = 1P, - R11T)
where T PTPT is agent 2's transition matrix.
Proof.
Ve, Vo, U2 diag(P)) P2P (
= diag(P2 P2) diag(P) P1)
- P2P._ diag + P2P.J API
(20)
Note that, for X,Y e RR:
XY =1 TIY
since:
T1V1
113/
119n
11
21
y1
1/2
3n
T2/11
F2l/2
322 D2
D2
Y1
1/2
3n
Fn
9/
1/2
Also, for M E R"." note that:
diag(X)M = X1 + M
since:
1111111
M1
11
21
M1,1
M1.2
M1.1
1212211
02
22
42
32
12,11
122,2
m2,
o
En Mm,1
2n2
Mnm.n
En
Fn
Mn,1
2n.2
Mn.n
And similarly:
(X) (M
So, taking closer look at each term in Equation 20
15

---

Published in Transactions on Machine Learning Research (02/2026)
diag(Ps)A diag(P:) ToA
(P)((P))PPP Pa1 @ A
P211 AP11 3 IPT
Pa1 O 1P @ APP
= =T AP1
PoP. diag(Pi) = P22 A IPT
=Pa1 O 012 O IPT
=P1 9 1P @ IP
= =T 1P
PaP ARP, P202P = =T 0,11
This enables writing the LOLA second-order gradient as:
VØ, To AA-AR1T - 1P, + +0111)
The term between parentheses can be factorized:
A A 1P A 11 = AP11T 1P, 11,1 T
= A - AP1 -
A
(I 1P
(A-
(1 - 1P ((11P)))(L - PP1)
So:
VA V 02 = = (I ) - PP1T)
16

---

Published in Transactions on Machine Learning Research (02/2026)
G
Stag Hunt
Stag
Stag
Hare
Hare
100 200
300
400
500
100
200
300
400
500
population step
population step
(a) 83% LOLA
(b) 86% LOLA
Figure 7: Mixed PG and LOLA in Stag Hunt (s = 1.8). When more than 86% of the population is made of LOLA agents, opponent-aware learners bring the entire population to the pro-social equilibrium (NB: the higher is, the lower this threshold becomes; reaches 0% when = 2).
H
Hawk Dove
Hawk
Hawk
Dove 0
Dove
2500
5000
7500
10000
2500
5000
7500
10000
population step
population step
(a) PG late evolution
(b) LOLA late evolution
Figure 8: Late evolution in Hawk-Dove (f -2)
17

---

Published in Transactions on Machine Learning Research (02/2026)
Peak-PPper-Scsssooe
1000
3000
800
population step
2500
Popp 1o0ouattoo step
600
1500
400
1000
200
500
0
0
(a) 20% LOLA
(b) 30% LOLA
Figure 9: Mixed PG and LOLA in Rock-Paper-SSiisunn
S
S
R
(a) step
(b) slep 50
(c) step 1000
S
s
R
P
(d) cycle, attractor
(e) cycle, attractor
(f) cycle, R attractor
Figure 10: Naive learning RPS, late evolution. Shades of blue indicate the concentration of individuals, while red arrows indicate their average measured movement. After about 4000 population steps, random drift slightly unbalances the 3 groups of near-deterministic individuals, which generates cyclic attractor pattern in the population. In the RPS model, naive learning sustains diversity.
18

---

Published in Transactions on Machine Learning Research (02/2026)
S
S
R
P
R
P
R
P
(a) step O
(b) step 10
(c) step 20
S
S
R
P
R
P
R
P
(d) step 30
(e) step 60
(f) step 100
Figure 11: LOLA in RPS. play the Nash equilibrium of this game (even when 70% of the population is naive, as shown in Figure 9b In the RPS model, LOLA hinders diversity.
quickly brings the entire population to unanimously
19

---

Published in Transactions on Machine Learning Research (02/2026)
J Algorithmic complexity
As noted in Scction 5.1 the scalability of the proposed methodology stems mostly from batching operations and leveraging the parallel computation capabilities of machine learning frameworks. In terms of "simple" algorithmic complexity (i.e., ignoring parallelization and framework optimizations for, e-g., tensor multipli- cations and gradient computations), all approaches tested in Figure 2 are O(N where N the number of individuals the population and A is the number of actions in the game. When using views, the procodure described in Section 4.4 is ((N) because of shuffling, and the computation of each term in Equation 12 is O(NA)). This is because Pi and P2 are of size N x A when batched across all N individuals, and consid- ering the schoolbook algorithm of matrix multiplications. As for automatic differentiation, the backward procedure is ((4)) per individual in the population, ie.. ((V More details are available in our code.
20


Reviews:
[{"review_id": "ZcNQhEQsqM", "reviewer": "Reviewer_N5h7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Policy Gradient (PG) and Opponent-Learning Awareness (LOLA) based techniques to study the learning behaviors of large populations of agents. Experiments on three stateless matrix games with populations of 200K agents demonstrate the efficacy of the proposed algorithm. This is an interesting contribution that aims to push the boundaries of multi agent reinforcement learning (MARL) to large populations of learners. \n\nStrengths:\n•\tThe paper presents fast, vectorized implementations of PG and LOLA updates that scale to populations of 200,000 agents, which is technically impressive.\n•\tAuthors derive update rules for PG and LOLA, and provide mathematical approach for large-scale simulations.\n•\tThe experiments aim to illustrate how LOLA and PG learners influence cooperation and diversity in classic games (Stag Hunt, Hawk-Dove, RPS), which is conceptually interesting (promoting or delaying the cooperation, and reducing the strategy diversity).\n\nWeaknesses:\n•\tUnsupported claim on heterogeneity\n•\tMathematical notation issues\n•\tLimited baselines", "claims_and_evidence_yn": "No", "claims_explanation": "•\tThe abstract claims that the proposed algorithm can simulate heterogeneous agents. However, none of the experiments actually demonstrate heterogeneity. If the authors wish to support this claim, they should define what they mean by heterogeneity (e.g., learning rules, initialization distributions, policy parameters, payoff matrices, or population structures) and include corresponding empirical results.\n\n•\tThe introduction cites Foerster (2018) as the primary reference for MARL, but MARL is a much older field dating back at least to Tan 1993 [Tan, M., 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning (pp. 330-337).].\n\n•\tIn section 3.1, 3.2 PG, LOLA, and first-order Taylor expansion are mentioned without citation. Introduction of Taylor expansions also lacks a clear mathematical citation.\n\n•\tParameter “P” is defined as both population vector and probability vector which makes the notation confusing.\n\n•\tThe proposed method is benchmarked against only two baselines. The results presented for “autograd iterative LOLA” baseline appear to diverge or explode visually, making them incomparable to the proposed method. This raises concerns about both baseline selection and the clarity of presentation.\n\n•\tThe captions for the figures 1, 2, 3, 5, 7, 8 are very short and not informative for readers to interpret them without referring to the main text.\n\n•\tCaption for figure 3 mentions that “the color marks the initial policy but the specific color mapping is not explained.\n\n•\tThere are many interesting similar works in the literature for mean field games (see for instance Hu et. Al 2025 https://link.springer.com/article/10.1007/s00245-025-10328-5) which are more recent and can serve as meaningful baselines. However, the most recent citation in the paper is from 2022.\n\n•\tIn section 5.3 authors said “From our results, it looks like …” is overly informal and introduces doubt.\n\n•\tIt is not clear if the results are presented for more than 1 replicate which is crucial for probabilistic models.\n\n•\tFigures 3, 4, 6, 7, 8, 9, 10 requires a color bar or heatmap for explanation.\n\n•\tIt is mentioned that the learning rate does not impact the algorithm , but no ablation experiments are provided to support this claim.", "audience_yn": "Yes", "audience_explanation": "The intersection of evolutionary game theory, multi-agent reinforcement learning, and large-scale population simulations is of clear interest to the TMLR community. The paper attempts to connect well-established MARL ideas (PG, LOLA) with scalable population-level models, which is relevant for both theory and applications (e.g., collective behavior, societal modeling).", "requested_changes": "Critical changes (required for acceptance):\n\nThe claim of supporting heterogeneous agents must be demonstrated experimentally or removed; authors must clearly define what “heterogeneity” means (learning rules, initialization, payoff differences, etc.).\n\nAdd foundational MARL citations (e.g., Tan 1993) to correct the misleading historical context.\n\nAdd proper citations for PG, LOLA, and first-order Taylor expansions in Sections 3.1 and 3.2.\n\nResolve the ambiguous notation where P refers to both population vectors and policy probability vectors.\n\nImprove the baseline evaluation: the autograd LOLA baseline diverges visually and is not informative; include more meaningful baselines, especially recent mean-field MARL methods.\n\nProvide multiple replicates or statistical reporting, as single runs are insufficient for stochastic population dynamics.\n\nImprove figure clarity: add descriptive captions and include colorbars/heatmaps for Figures 3, 4, 6, 7, 8, 9, and 10.\n\nProvide ablation results supporting the claim that learning rate has no effect.\n\n\nRecommended (non-critical) improvements:\n\nClarify color coding in Figure 3 (“the color marks the initial policy”) with explicit mappings.\n\nUpdate the related work section to include more recent literature (post-2022).\n\nReplace informal phrasing such as “From our results, it looks like …” in Section 5.3 with precise statements.\n\nRestructure figure presentation to avoid extremely large or diverging baselines that distort comparisons.", "broader_impact_concerns": "I do not see major ethical concerns beyond standard considerations for research on learning dynamics and social simulations. The work does not appear to require a broader impact statement beyond what is already typical for MARL and game-theoretic modeling.", "additional_comments": "The overall direction of the work is promising, and the connection between EGT and scalable MARL is valuable. Strengthening the empirical evaluation and clarifying several presentation issues would substantially improve the clarity and impact of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "T6XPPEaLyN", "reviewer": "Reviewer_o5Te", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors state the main contribution of the paper is a methodology for using reinforcement learning to learn strategies that reach stable equilibria in multi-agent settings where all agents seek to maximize their individual rewards.\n\n**Weaknesses**\n- This paper was very difficult to read and I had a hard time going. Consequently, neither the novelty nor the significance of the paper's contributions are readily apparent. Maybe I missed something or maybe I am not the right audience for the paper, but I did not come away with any clear message.\n- The authors' justification for using reinforcement learning to learn stable equilibria strategies when considering the evolution of population dynamics i.e., the EGT framework, is tenuous. The strongest justification being that their is some work that show organisms use some degree of reinforcement learning to learn behavior.\n- I also do not understand why the authors choose to fit policy gradient and LOLA (I get why LOLA would be used over policy gradient, but why apply it to the EGT framework?) to EGT. Approaches like *regret matching* and *fictitious self-play* seem to be obvious for the problem. They also iteratively learn policies through repeated interactions so can one not argue that they are also RL approaches? They also seem to be more suitable for EGT. If not, why not? Why use PG/LOLA over them? What do we gain or learn by applying PG/LOLA instead?\n- The paper is also not well-written. It was difficult to understand most of what was written. Some examples of problems:\n    - The subsection on EGT never clearly explains what the goal is.\n    - Many sentences are really long and hard to follow.\n    - The authors do not explain the purpose of experiments or what they intend to show with the results.", "claims_and_evidence_yn": "No", "claims_explanation": "It is hard to identify what the claims actually are or which claims are novel to the paper.\n\n1) They claim to introduce a way to implement policy gradient and LOLA in a backprop-free manner that can be parallelized.\n    - They say Eq 7 is what allows them to do this for PG, for instance. Is that novel? If it is, then the authors need to prove or derive the equation. They simply state that this can be done. Is it because it has been shown in the literature? Then it isn't a contribution.\n2) I don't quite know what claim(s) the experiments are trying to support. They compare an approach that treats the entire population as a single agent and their approach that treats the population as multiple agents. The message from the results that I took away was that treating the population as a set of agents is better. Yes, that seems like an easy conclusion to come to given the nature of the problem. What new insight do I get from these results?", "audience_yn": "No", "audience_explanation": "The paper offers no significant contributions, in my opinion.", "requested_changes": "I do not think there are any changes that would make me change my mind on the paper. The paper needs to be re-written so that its significance is more explicitly stated and justified, and their contributions are clearer.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WAN8soKtyH", "reviewer": "Reviewer_sMC6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate an efficient implementation of a policy gradient (PG) algorithm that incorporates opponent-aware learning considering large population game. The main contribution lies in the proposed matrix-batched method, which randomly pairs agents that enables scaling to large population game. The claims are mainly supported by experimental results, including Stag Hunt, Hawk-Dove and RSP. The opponent-aware learning method shows different behavior form original policy gradient approach and the differences depend on the game class and whether training is in self-play.\n\n\n**Strength**\n\n1. The paper efficiently observes distinct behavioral patterns for opponent-aware learning versus  naive PG algorithm.\n2. It is empirically shown that the proposed method scales to large population dynamics of PG thanks to efficient computation of matrix calculation.\n3. The paper is well-written and provides sufficient background to understand the paper.\n\n**Weakness**\n\n1. Even though the authors compare the proposed algorithm in several well-known examples, it is unclear why such difference happens and how we should understand it. It seems that the behavior vary by game class and by whether training is in self-play. What is the main take-away from these experiments?\n\n2. The paper lacks theoretical analysis on the dynamics of LOLA in (12). For example, what are the conditions of its convergence and how fast does it converge?\n\n3. It is not clear how efficient the method is compared  to original method of the computation comes from. Qualitative analysis of the computational complexity could be provided to clarify this point.", "claims_and_evidence_yn": "No", "claims_explanation": "The claims are mainly supported by the experimental results. On the other hand, it lacks theoretical justification, e.g., simple computation complexity analysis on matrix batched method or convergence (rate) of proposed LOLA algorithm.", "audience_yn": "Yes", "audience_explanation": "The paper deals a problem of learning dynamics of evolutionary dynamics in particular, the policy gradient algorithm. Empirical results on standard benchmarks further underscore its relevance to both the game-theory and reinforcement-learning communities", "requested_changes": "1. As mentioned in the weakness, theoretical analysis of the convegence or convergence rate of the update of LOLA in (12) could strengthen the paper.\n\n\n2. Providing analysis of computational complexity of the proposed method (matrix batched method) could strength the understanding of the proposed method.\n\n3. Could the authors clarify whether Figures 3(a) and 3(d) are the same plot? They appear indistinguishable.", "broader_impact_concerns": "na", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)