Population-based Evaluation in Repeated Rock-Paper-Scissors as a Benchmark for Multiagent Reinforcement Learning
Authors: Marc Lanctot
John Schultz
Neil Burch
Max Olan Smith
Daniel Hennes
Thomas Anthony
Julien Perolat
Venue: Accepted by TMLR ()

Abstract: Progress in fields of machine learning and adversarial planning has benefited significantly from benchmark domains, from checkers and the classic UCI data sets to Go and Diplomacy. In sequential decision-making, agent evaluation has largely been restricted to few interactions against experts, with the aim to reach some desired level of performance (e.g. beating a human professional player). We propose a benchmark for multiagent learning based on repeated play of the simple game Rock, Paper, Scissors along with a population of forty-three tournament entries, some of which are intentionally sub-optimal. We describe metrics to measure the quality of agents based both on average returns and exploitability. We then show that several RL, online learning, and language model approaches can learn good counter-strategies and generalize well, but ultimately lose to the top-performing bots, creating an opportunity for research in multiagent learning.

Published in Transactions on Machine Learning Research (10/2023)
Population-based Evaluation in Repeated Rock-Paper- Scissors as a Benchmark for Multiagent Reinforcement Learning
Mare Lanctot Google DeepMind
lanctot com
John Schultz Google DeepMind
jhtschulzz&gooogle.com
Neil Burch
nburch@@ullertto.cu
Sony Al University of Alberta
Max Olan Smith University of Michigan
masmith.uuicchccmm edu
Daniel Hennes Google Mied
hennes Sernne@@googe..com
Thomas Anthony Google Deep
twa twagggooge.coo com
Julien Pérolat Google Deep
perolat@google.co com
Reviewed on OpenReview: https: // openreview. net/ forum? id- 900JAA
Abstract
Progress in fields of machine learning and adversarial planning has benefited significantly from benchmark domains, from checkers and the classic UCT data sets to Go and Diplo- macy. In sequential decision-making, agent evaluation has largely been restricted to few interactions against experts, with the aim to reach some desired level performance (e.g. beating a human professional player). We propose benchmark for multiagent learning based repeated play of the simple game Rock, Paper, Scissors along with population of forty-three tournament entries, some of which are intentionally sub-optimal. We describe metrics to measure the quality of agents based both on average returns and exploitability. We then show that several RL, online learning, and language model approaches can learn good coumter-strategics and generalize well, but ultimately lose to the top-performing bots, creating an opportunity for research multiagent learning.
1 Introduction
How should agents be evaluated when learning with other learning agents? One metric is simply the average return over an agent's lifetime. Another is the agent's robustness against polential nemesis whose goals are only to minimize the agent's return. The first the conventional metric used in the evaluation of reinforcement learning (RL) agents, while the second quite common among game-theoretic. AI techniques for imperfect information games. In this paper, we argue our position that neither of these is generally sufficient in isolation: good agents should both maximize return and be robust. to adversarial attacks.

---

Published in Transactions on Machine Learning Research (10/2023)
Player 1
R
5
R
(0.0)
(1,-1)
(-1,1)
Player 0
P
(1.-1)
(0,0)
(-1.1)
S
(-1,1) (1,-1)
(0.0)
Figure 1: Rock, Paper, Scissors. Player 0 chooses an action assignod to a row, and similarly player for a column. Each entry shows the reward for player 0, then player 1 respectively.
The classical method to demonstrate superior AT performance is head-to-head matches, or direct comparisons of average return, against the strongest known agents. This method has driven progress of the field since the beginning: from Samuel's checkers program, to chess, Go, poker, modern real-time games, and so on. On the other hand, game-theoretic approaches to learning result in agents that approximately respond to a population of opponents which are enumerated in hopes that the full strategic complexity of the game is captured among the set of opponents, and convergence to an approximate Nash equilibrium is obtained. The extent to which current AI systems are robust to adversarial attacks is unclear. Nevertheless, there is evidence that even expert level AT agents can be demonstrably susceptible adversarial behavior (Timbers) al. 2022 Wang al. 2022). While current evaluation methodologies over-emphasize the single metric of cunulalin reward or performance against experts, human AI, we argue that the more important problem is the lack of benchmarks that prioritize the evaluation of agents in a more general way, where multiple metrics could lead to a better understanding of an agent's capabilitics.
In this paper, we propose a benchmark based on the classical game of Rock, Paper, Scissors augmented two ways: first,
repeated game and hence a sequential decision problem; second, performance is measured against population of agents with varied skill. The simplicity of the stage game is of paramount importance: it is well-understood two-player zero-sum game whose name-theoretic optimal strategy is well- known, and by construction maximizing rewards against fallible opponents naturally leads to behavior that is potentially exploitable. For learning agents to find exploits in the opponents, they must correctly deduce their strategics from observations. We describe a population of forty-thrce openly-available hand-crafted agents that were submitted to competitions and characterize their head-to-head performance, exploitability, and the extent to which they are predictable (by supervised learning). We then train agents using several modern approaches with different capabilities, against the population and independently trained against copies of themselves. These approaches show promise in various ways: out-of-distribution generalization of exploitative behavior, clear lack of exploitable behavior, and a good balance between these two metrics. Ultimately, none of the agents are able to outperform the top two participants in head-to-head matches while being more robust to exploits, leading to a challenge and opportunity for novel multiagent reinforcement learning research.
2 Repeated Rock, Paper, Scissors
In this section, we describe the basic notations, the environment, competition and participants, and population-based evaluation. The environment and population are froely available within OpenSpicl (Lanctot
et al. 2019
2.1 Notation and Environment Description
A normal-form game has a discrete set of players N (1,2, ... .n.. A matrix game is a two-player game with a set of actions per player A and Ag, a joint action set A = A₁ x Az, and utility functions for each player i & AV, us A + R. zero-sum game is one where Va E A, E (a)..
Rock, Paper, Scissors (RPS), also called RoShamBo, two-player zero-sum matrix game described by the matrix depicted in Figure] Rock (R) beats Scissors (S); Paper (P) beats Rock (R), and Scissors (S) beats Paper (P).
2

---

Published in Transactions on Machine Learning Research (10/2023)
The sequential version is repeated: there are K identical plays of RPS. At state su, agents simultancously decide their actions and agent i receives intermediate reward re, by joint action d, composed of all agents' actions combined and payoff matrix in Figure ! A trajectory is a state and (joint) action sequence of experience: p = (Su, au, $1, 41,' ,5K-1, sk). In this environment, every episode has length K and the full (undiscounted) return is defined as Go.; Ei= Pt.i. We choose K = 1000 as default from the competitions described in Section 2.2
Similarly to previous work in this environment [Hernandzz et al. 2019 observations to the agent depend on the recall, R. With a R = 1, the observation at 8, includes the most recently exccuted joint action a;-1, encoded as 6-bit observation (two one-hot actions). With R = 2, the observation includes the two most recent. join actions, and so on, where R = K includes the full action sequence. For example, when R = 10 there are g s 3.5 billion unique observations; a tabular Q- Q-learning agent would table of 10.5 billion entries. Unless otherwise noted, use R = 1 as default value.
Finally, as is standard (Sutton & Barto 2017;; a policy my is a mapping from an observation to a distribution over actions used by agent i, and π (without subscripts) is the joint policy used by both agents. In RPS, there is a large incentive to use stochastic policies because any deterministic policy is fully exploitable (Shoham & Seyton-Browi 2009). For simplicity of notation, we denote Gt,,,r GG,.]
2.2 Competition and Participants (Bots)
In carly 2000s, Darse Billings ran two Repeated Rock, Papor, Scissors (RRPS) competitions (Billings 2000a b). In this subsection, we describe the participant entries that were released and still openly accessible, which have since been integrated into OpenSpiel (Lanctot et al. 2019). In each competition, participants were asked to submit bo to play RRPS, with K = 1000, all played within a one-second time limit. Each program had full recall, the entire action sequence in each episode, but nothing more that would identify the other bots. Participants were told in advance that the population would include some sub-optimal bots.
The majority of the entries in the competition were hand-crafted heuristic bots that were developed inde- pendently by different programmers. A few participants submitted two entries. The resulting population consists of 43 bots: 25 entrant bots and 18 seed bots from the first competition. Including the winner of the second competition Andrzej Nagorko's 's GREENBERG, made open-source seperately, and the first competition winner Dan Egnor's IOCAINEPOWDER.
We now summarize the approach taken by most bots. The simplest seed bots do not use their observation to inform their action. RANDBOT generates an action uniformly at random. ROCKBOT always plays rock. R2266BT plays 20% rock, 20% paper, 60% scissors. ROTATEBOT rotales between R. P, S in that order. PIBOT, DEBRUIJN81, TEXTBOT all play a fixed sequence of actions derived from the digits of pi, De Bruijn sequences, and the competition rules in base 3, respectively.
Other seed bots have recall R = i.e. they use only the current observation. SWITCHBOT never repeats its' previous action, and chooses uniformly between the two alternatives. SWITCHALOT repeats previous action with 12% probability; otherwise, chooses uniformly between the two alternatives. COPYBOT plays to beat the opponent's 's previous action. DRIFTBOT and ADDDRIFTBOT2 bias their action by the opponent's action or joint-action, respectively, with an increase, or "drift", bias over time. FOXTROTBOT alternates between playing randomly, and an offset of its' previous action.
The remaining seed bots used historical observations either directly through statistical summaries. FLAT- BOT3 plays a flat distribution. ADDSHIFTBOT3 biases decision by previous joint action, shifting the bias if losing. ANTIFLATBOT maximally exploits FLATBOT3. ANTIROTNBOT exploits rotations played by the opponent. FREQBOT2 plays to beat opponent's most frequent choice.
The entrant's bots also used historical observations. ROBERTOT uses a voting algorithm informed by ob- servation counts. PREDBOT, PIEDRA, and SWEETROCK predict play from action counts. MODlBOT models the opponent as PREDBOT. BIOPIC maintains four prediction models differing in available information.
In this paper, "bot" always refers previous competition participant, whereas "agent" refers an RRPS player more generally.
3

---

Published in Transactions on Machine Learning Research (10/2023)
MARKOV5, MARKOVBAILS, RUSSROCKER4, and HALBOT inform their prediction with Markov chain models. PHASENOOT,, PETERBOT, MULTIBOT, and MIXED SEPAASSTRTTGGYY all switch between fixed set of policies de- pending on which is currently the most profitable. INOCENCIO, -OME, MARBLE, CRANITE, BOOM, and SHOFAR also implement complex rulc-based decisions informed by summary statistics of the history.
Several bots took very innovative approaches. SUNNERVEBOT implemented "nervous" network reminiscent of deep neural network. ACTR LAG2 DECAY implemented the cognitive architecture ACT-R (Anderson 1993).
10CAINEBOT |Egnor 2000), which won the first competition, works by maintaining set of predictions about its opponent, and building a set of strategies from each predictor. Predictions included random guessing, frequeney analysis, and history matching across six different history sizes. From each prediction six stralegies are constructed based on recursive response computations (e.g., triple-guessing). IOCAINEBOT then plays the most historically successful strategy. GREENBERG, by Andrzej Nagorko, won the second competition by exlending 1OCAINEBOT to include additional predictors utilizing more advanced history matching algorithms.
2.3 Population- Based Evaluation
We propose several ways to use this population to evaluate agents. We define an agent's 's POPULATIONRE- TURN to be the average return per episode against a bot drawn uniformly at random at the start of the episode. Performance against specific bots can also be reported: we compute the cross-table between all bots in Figure 2 The exploitability of an agent is by how much their nemesis (best response) beats them. Let -i refer to agent i's opponent. Then,
EXPL(TT) Go.-i(m.MMT)) where b(T,)) E BR(m:), and
BR(m.) = (r- max ( is the set of best responses to n;. Notice that exploitability is expressed in the opponent's return; it is non-negative and its lowest value is zero when an agent is not exploitable. However, due to the maximization over the entire policy space, it can be too computationally expensive to compule exactly, so we can approximate algorithms and taking the maximum achievable value. Another measure of approximate exploitability uses the bots as exploiters, taking the maximum over the bots, where P is the population:
by running several learning
WiTHINPOPEXP((T))
max Ba (lGoo a :CP
Head-to-head performance of all bots in the population is visualized in Figure 2P Each cell represents an average over 1000 episodes. Figure 3 summarizes some properties of the population. First, the population returns of each bot range from -648.42 42222t22 to 288.15, achieved by GREENBERG. GREENBERG dominates (achieves higher value against all opponents) five bots, and 10CAINEBOT dominates one bot. Sccond, the within- population exploitabilities range from 1.2 (RANDBOT) to 1000, with several reaching this upper-bound, 316.1 on average, We then trained several RL algorithms until empirical converges (millions of episodes) against each bot independently: Q-lcarning and IMPALA (Espcholt et al. 2018) with R E (1,3, 5, 10), and defined the external-learned exploitability of that bot as the maximum value achieved among these exploitability
eight. These values range from 4.8 to 1000.0, with an average of 420.3. The within-populant achieves 75.2% of the external-learned exploitability on average, and varies between 50-100% of the external- learned exploitability most bots. Due to this consistency across bots and significantly less computation requirements, we mainly use within-population exploitability from here on.
One simple way to rank agents under both metrics is to assume they both matter equally: ACCREGATESCORE(T.) = POPULATIONRETUR(T] - (OTHHI)PPE This definition makes sense since the units are idantical; could naturally be extended with different. weights depending on the evalua- tion context. These two motries (population return and tithin-population exploitability) capture the essence of the RPS game in the repeated setting: the only way another agent can predict an agent's 's next action is
2The precise values in this table available from OpenSpiel Lanctot open_eppiel//rree/mssteer//// oppeeere///eeeeeeeeeeeeeeeeeeeee httpa/aaeeeeeeeeeeepeettttaaa//e sppaee/aataa/ppprrrrr http://appedddaaaeeeeeeae lal_dddaa_aarrppppe _rrp
2019: https://gåthub. com/ www.///ooogeeeeeeeiddddde
4

---

Published in Transactions on Machine Learning Research (10/2023)
1000
750
500
250
250
500
750
1000
greenberg
iocalnebot
phasenbott
halbot
modlbot
blopic
robertot
russrockeri
boom
predbot
granite
marble
lag?
sweetrock
piedra
sunNervebot
zo_m
antirotnbot
mixed strategy
markovballs
markov5
multibot
inacencio
debruijn81
pibot
randbot
adddrifboo2
driftbot
fextrothot
flatbot3
texthot
addshifbbot
switchalot
sunCrazybot
switchbot
peterbot
r226bot
freqbot2
copyhot
rotatebot
rockbot
antirlatbot
greenberg
jocainezed
inasenbel
halbot
mad lbt
biopic
robertar
russrockerd
boom
predbot
shofar
grante
marble
actr Jag2 decay
piedra
sweetroce
LunNervvete rvebet
24, move
antirotrizo
strategy
markovs
ma Novo moovballs
muitibol
inocenci
debruijnet
pibot
randbot
addenitbotz
drifteos
foxtrottot
Ilatbot3
texthot
addshifibolt
switchalot
sunCraryaa
peterbos
2226Dot
espyl
notatebot
rockbot
antiffatbot
sunNer
Figure 2: RoShamBo bots payoff table. Each cell shows the return for the row bot versus the column bot averaged across 1000 episodes.
by determining their learning rule from the history of choices made by both agents. Ultimately the goal is to maximize return, but finding best response to the other player's action choice is the entire game, so agent cannot be loo exploitable doing so without risking giving reward to its opponents from the population. Hence, the combined score acts as a summary of how well an agent is performing on both fronts within is population, allowing agent designers to compare a single number. Under this metric, we list the top bots in Table The complete ranked list is given in Appendix A.1 For reference, we also include the scores of the best learning algorithm in each category from Secton
3 Predictability of RPS Bots
In order to win at RRPS, the bots must attempt to predict the actions chosen by other bots, while not becoming predictable themselves by their opponent. In this section, we investigate to what extent. the bots are predictable by a neural network.
To assess how prodictable each bot was, we sampled games of RRPS between the bot and each other bot, including itself. We trained LSTM per bot to predict that bot's next action with recall R 20 (details
5

---

Published in Transactions on Machine Learning Research (10/2023)
Bot Names
Pop. Return
W.P. Expl.
Agg. Score
GREENBERG
288.15
3.65
284.50
1OCAINEBOT
255.00
5.00
250.00
BIOPIC
196.36
36.66
159.70
BOOM
169.11
27.93
141.19
SHOFAR
152.01
16.87
135.14
ROBERTOT
177.77
50.16
127.61
PHASENBOTT
232.25
111.71
120.54
MODIBOT
203.16
90.16
113.00
SWEETROCK
146.25
41.21
105.04
PIEDRA
146.08
41.44
104.64
Agorithm/Agen Names
Pop. Return
W.P. Expl.
Agg. Score
PopRL
258.00
10.98
247.02
LLM (Chinchilla 70B)
201.00
45.80
155.20
ContRM
164.77
16.27
148.51
QL (R = 10)
00.5
8.62
8.10
R-NaD
10, 5]
[20, 40]
|-50--
-25]
Table 1: Top 10 bots ranked by ACCREGATESCORE, and top learning algorithms in each category from subsections of Section 4 The bot results are computed on 1000 episodes per profile. The algorithm results are averaged over seeds.
antiflatbot
mckkot
randbot
Exterrai approximate exploitability
rotatebot
copybot
ocainebot
Wihin population expleitabiiit
freebot2
shofar
r226bct
markovbeis
peterbet
markovs
sw switthbet
bo
sunCaayybet
biepic
switchalet
sweetrock
addshirbbot
piedra
textbet
webat
fistbot3
robertat
foxtrotbot
drittbot
pibot
addirittbtt
jstrategyy
phasenbott
debruijn81
halbot
Inocanclo
twitbat
Agant
multibot
actr lag2 dlecay
markoes
Agent
arbie
marksobails
granite
str ayy
dritbbot
predbot
adcerfibe22
SunNervebot
debruinS.
lag. detay
addihittbol
"LLmove
toxtrotbot
shofar
flatbot3
switchbot
russrockerd
russrockerd
sun cunarray
inoceevio
paterbat
modibot
copybot
halbet
rotatebot
phasenbott
heebol2
locainebet
antfiebbot
reckbot
.000
-400
-200 population tion return episode
200
400
600
200
400
500
800
1000
episode
Figure 3: Left: Population Returns for each bot. Right: Approximate exploitabilities for each bots.

---

Published in Transactions on Machine Learning Research (10/2023)
0.0
0.2
0.4
0.6
0.8
1.0
copybot
rockbat
rotatebot
textbot
pibot
debruijn81
fregbot2
a antiflabbot
peterbot
29_ move
predbot
marble
addshifibot3
multibot
granite
antirotnbot
sunCrazybot
inocencio
modlbot
sweetrock
piedra
halbot
robertot
foxtrotbot
actr lag2 decay
adddriftbot2
greenberg
mixed_strategy
driftbot
biopic
r22bbot
sunNervebot
locainebot
shofar
flatbot3
phasenbott
switchbot
markov5
markovbails
switchalot
randbot
copybot
rockbot
rotatebet
textbot
debruijn@1
freqbatt)
antifattoo
peterbet
move
predbot
marble
addsniftoot3
multbot
granite
antirotnbot
BraCrazybot
inocencio
modibor
sweetrock
piedra
bat
robertot
foxtrobbot
actr lag? decay
addonitootz
greenberg
mixed strategy
drffboo
boom
biople
/226bot
russrockerd
sunNerveboo
locainebot
shafar
flatbot3
phasenSots
switchbot
markovs
rkovballs
switchalot
randbot
antiro
Figure 4: Action prediction accuracy. Each cell shows the action prediction accuracy for row bot versus the column bot averaged across 100 episodes.
We report the prediction accuracy, i.c. the proportion of the time that the predicted action matched the bot's action, shown in Fig,
in App. A.5
Some bots are deterministic and easy to predict, e.g. ROCKBOT was predicted correctly 100% of the time, Stochastic bots, such as RANDBOT, have low predictability, but this comes at the cost of their ability to exploit other bots. Prediction accuracy for the entrants was substantially greater than for the Nash equilibrium, but varied substantially from 48% for MARKOVBAILS 94% for PETERBOT.
Successful action prediction reveals the existence of structure within the bot population. In principle, RRPS is purely non-transitive game, and there is such thing 'better' strategy. Under a unique Nash equilibrium, agent's 's past actions are not predictive of their future actions. Still, we hypothesize that it is possible to learn action predictions from a sub-population that generalise to the whole population.
To test this, we sample 30 bots from the population randomly, and generate RPS games between these bots. We then train the same LSTM model as before to predict the bots' actions. To succeed at this task, the agent must identify the strategy bot employing predict the next action it no longer knows the bot identities priori. This also means that if held-out bots employ similar strategies, the agent should be able

---

Published in Transactions on Machine Learning Research (10/2023)
co-player
train
test
train 69.14% 57.80%
67.35%
predicted bot
test
55.65%
Table 2: Action prediction accuracy.
to predict their actions too. We repeated the experiment 10 times with different splits of training/testtng bots.
On games between the training bots, the neural network achieved average accuracy of 69.1%. In games between the held-out test population, the neural network achieved accuracy of 55.7%, which is significantly better than chance, and demonstrates there is learnable structure in the bot behaviours. In Table we break down accuracy by whether the bot being predicted or the co-player are in the training population. We show that prediction accuracy drops for either bot or co-player being from the held-out population, but the effect is larger when the bot being predicted is not in the training set.
4 Learning to Play Repeated RPS
Can an agent learn to earn high population return and not be very exploitable? Here, we show baselines and RL agent performance on this environment. We evaluate them using the population- evaluation (PBE) criteria in Section 2.3 The resulting best achievable score of cach learning approach is summarized in Table 1
4.1 Baseline Independent RL Results
In this section we report the performance of fixed policies and baseline RL agents. Each individual run reports the best achieved performance of an fixed agent one trained by playing against. another copy of an agent of the same type (independent RL). Note that diffcrentiate this training from "self-play" due to the agents using the same algorithm but separale net works We then evaluate the agents against the population after 700k 1M episodes of training, with the population return and returns of each bot against the agent being averaged over a sliding window of the 50 most recent evaluations. Each reported value represent the the average over five individual runs using different seeds Table 3
For Q-learning (QL), we report results over varying recall size E (1,3,5, 10). Interestingly, the within- population exploitability of uniform is greater than zero, which is possible to due to maximization over noisy estimates and the deterministie nature of the random number generators. In addition, we run DQN (Mnih et al. 2015g, A2C (Miih et al. 2016), and Boltzmann DQN (BDQN) (Cui & Koeppl 2021) with various temperatures E {0.1, 0.5, 1, Overall we found that the algorithms improve as I increases, achieving a population return at most 18, and can be particularly exploitable. The high exploitability is somewhat mitigated by a sufficiently large (R=10) table in Q-learning, higher temperature in Boltzman DQN, and entropy bonuses in A2C. The best achievable aggregate score across these basclines is 8.1. Appendix A.2 contains hyperparameter selection details for the RL agents.
We then assessed the quality of independent RL agents when the recall is set to much larger value (R e {100, 1000}). This leads to observations that are 10-100 times larger and too many states to fit in memory, so we use only DQN, BDQN, and A2C. We run independent RL for the same amount of wall-clock time as the shorter recall results (12 days), which leads to fewer episodes due to the added computational cost per episode. In all cases, we noticed variation among hyper-parameters, but that many dropped to a low aggregate score (-1600) and then never recovered higher score. The best values over all the runs are presented in Table 4 The highest achieved aggregate score among these
7.60.
8

---

Published in Transactions on Machine Learning Research (10/2023)
Name
Pop. Return
W.P. Expl.
Agg. Score
ROCK
610.20
1000.00
-1610.20
PAPER
-61350
999.20
-1612.70
SCISSORS
648.10
1000.00
16488.10
UNIFORM
0.00
9.31
-9.31
QL (R = 1)
531.28
994.54
1525.82
QL (R 3)
280.65
910.56
1191.21
QL (R = 5)
-89.67
405.89
495.56
QL (R = 10)
-0.52
8.62
8.10
DQN
-194.49
693.13
887.62
BDQN (17 = 0.1)
124.52
515.60
640.12
BDQN (17 0.5)
19.59
164.25
183.84
BDQN (1 1)
18.00
51.93
-33.93
BDQN (17 2)
12.75
11.20
1.55
A2C
0.18
9.84
-9.66
Table 3: Baseline bots and agent performance. Results are averaged over 5 seeds.
Name
R
Num. Episodes
Agg. Score
DQN
100
316300
-653.59
BDQN
100
306770
-15.50
A2C
100
161660
-13.9
DQN
1000
65520
-1143.5
BDQN
1000
75840
7.60
A2C
1000
17170
-78.45
Table 4: Baseline bots and agent performance with longer recalls (R E {100, 1000)).
Results are averaged
over seeds.
4.2 Language Model Agent
Large language models (LLMs) have achieved state-of-the-art performance across a wide variety of natural language processing tasks. This is accomplished by simple token-level training objectives, applied to massive amounts of lext data scraped from the web. LLMs can be further fine-tuned on specific tasks, and have been successfully utilized as components in game-playing systems, most notably Cicero which achieved human- level performance Diplomacy (Meta et al.) (2022). Even without fine-tuning, LLMs demonstrate some game-playing ability like finding legal chess moves, but exhibit poor performance at identifying checkmate- in-one moves (rivastavaa et al. 2022).
Here we benchmark four model sizes (400M, 1B, 7B, 70B) from the Chinchilla family of LLMs (Hoffmann et al. 2022) on the RRPS task. As the information gain in RRPS per step is relatively low, to excel at this game, a successful agent has anticipate the other agent's 's actions through the history. The language model agent particularly relevant for RRPS because transformers have the unique ability attend (relative) paiterns in the history that are important for prediction of the next token.
We utilize the LLM action the opponent will play next. The model is given zero-shot prompt that plainly states the task and provides the game history (see Appendix A.3 for full prompt). The model's prediction of the opponent's 's next action is determined by choosing the max over the logprobs of the tokens {R, P, S}. The LLM agent then deterministiaally plays the action that beats the opponent's prodicted action. The true actions played are appended to the prompt and the process is repeated. No parameters are fine-tuned at any point. Methodologies for prompting and fine-tuning LLMs and integrating them into larger systems areas of
game-playine agent by selecting actions based the model's prediction of what

---

Published in Transactions on Machine Learning Research (10/2023)
active research, and optimizing the LLM's performance on RRPS is beyond the scope of this paper. However, even in this simple zero-shot setting, and despite not having been trained on RRPS, LLMSs demonstrate a surprising ability to to predict opponent actions that improves with model size. The largest LLM model achioves an average population return of 201.0 and aggregate score of 155.2, placing it fourth behind only GREENBERG, JoCAINEBOT, and BIOPIC; though, we did notice that size of the model size had a significant effect on the performance: our smallest model achieves an aggregate score of 2212.9 comparison (sce Appendix [A.3] for full results). Domain-specific fine-tuning would likely yield improvements and offers a promising direction for progress this benchmark. Moreover, RRPS also offers measure of an LLM's capacity for identifying and adapting to members of a population it interacts with.
4.3 Regularized Nash Dynamics
To minimize the exploitability (iee thus converging to Nash equilibrium); solution that empirically scale well is to learn policy with the Regularized Nash Dynamics (R-NaD) algorithm (Perolat et al. 2022 In a mutshell, this method repeat a 3 step process: 1) building a reward transformation based on a regularisation policy, 2) a step where the process converges to a new fixed point of the game and 3) update the regularization policy with the fixed point found at step 2). With R = 1, R-NaD achieves Pop. Return in the set [-10,- W.P. Expl in the set [20, 40) and an Agg. Score the set [-50,- -25] which is not far away from what the random policy achieves. The implementation used to produce these results uses the OpenSpiel implementation of R-NaD (Lanctot et al.] 2019). We used the paramcters from the open- source implementation and did sweep over the following parameters (randomized over 5 seeds): 17 reward transform: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]. trajectiry max 10, 000, 000, batch size : [64, 128, 256, 512], entropy schedule size: : (20000,) ), finetune from : 1-1,300000 600000).
This algorithm achieves a strategy that is hard to exploit but it will not exploit the other players.
4.4 Contextual Regret Minimization
RRPS fits into the setting of online learning and adversarial bandits, which looks at maximising value over repeated interactions in an environment with a fixed set of actions and unknown, dynamic payoffs. From the perspective of either player in a RRPS episode, they are repeatedly choosing an action of R, P, or S. The payoffs for each action are unknown because the player does not know their opponent's strategy for the next action. So, one natural choice for making decisions in RRPS is using an adversarial bandit algorithm. Regret minimizing algorithms all have theoretical guarantees that their average expected online performance is close to some oplimal baseline, in hindsight. For example, is expected to do roughly as well any single static action if we looked back in time and asked how well we would have done if we had played a instead. In RRPS, an agent that has low external regret would not have done significantly better by playing one the always-R, always-P, or always-S baseline policies against the opponent's 's sequence of actions. Other regret measures consider richer sets of baseline policies.
algorithm which minimises external regret
We look at four different algorithms for bandits with full information feedback, with different regret guar- antees. Regret Matching (RM) is simple, parameter-free free algorithm which minimises external regret (Hart) & Mas-Colel] 2000). Regret Matching+ (RM+) is a modification of RM that often has better empirical performance (Tammelin et al., 2015). RM+ also has a weak guarantee with respect to k-switching re- gret, which compares performance to all possible k-piecewise policies. The strongly adaptive online learner (SAOL.) provides a strong guarantee for non-stationary environments, with a performance bound on any sub-interval (Danicly et al. 2015). SAOL is meta-algorithm operating on top of another regret minimizing algorithm, and we used RM+ for the base algorithm in our inplementation. Minimizing swap regret ensures that agent would not have wanted to play action any time in the past when they had played b, for any actions a and b. For swap regret, we used the meta-algorithn of Ito Tto 2020) on top of RM+
While these four algorithms depend on the history - the historical actions played determine the current policy they do not explicitly consider the current context (R = 0). One way to frame RRPS as a contextual regret minimization problem is to completely scparate each possible recalled history for R > 0 into separate contexts, with independent regret minimizing algorithms running in each context. An agent using this discrete set of contexts has 9, 81, and 729 independent instances for R R= 1, R = 2, and R = 3 respectively.
10

---

Published in Transactions on Machine Learning Research (10/2023)
Context
Agent
Pop. Return
W.P. Expl.
Agg. Score
RM
48.45
27.39
21.06
SAOL
67.30
34.73
32.57
R 0
RM+
59.66
26.36
33.30
SWAP-RM+
62.73
21.96
40.77
SAOL
178.08
91.32
86.76
RM
164.75
76.30
88.44
R 2=1
RM+
169.88
63.99
105.89
SWAP-RMH
167.99
48.41
119.58
SAOL
171.46
155.39
16.07
RM
175.43
148.89
26.54
R =
RM+
174.44
121.79
52.65
SWAP-RM+
173.52
99.93
73.59
RM
157.55
23.92
133.62
R=1
RM+
157.99
17.51
140.48
History Experts
SWAP-RM+
156.93
15.69
141.24
SAOL
164.77
16.27
148.51
Table 5: Performance regret minimizing agents across very historical contexts. Results are avoraged over seeds.
Another way to add context is to instead augment the environment actions with context experts that suggest enviromment actions: "history experts". For R = 1, we added six history experts suggesting the opponent's last action o, our last action u, the actions that beat o and u, and the actions that lose to and u.
The full results from this experiment are included in Table 5 We see that, generally, SAOL performs best among all the choices of bandit rules, all contexts. Overall, this context regret-minimizing agent is able to increase its population return R increases, but its exploitability also increases well. In terms of aggregate score, the more dynamic definition of experts that are functions of the most recent actions and u as well as their counler-strategies (history experts) perform significantly better than using the raw observation histories as context, which were also used by the baselines in Section 4.1 The best-performing version of this agent achieves an aggregate score of 148.51, which places between third and fourth place among the bots in the population.
4.5 IMPALA and Generalization
Tn this subsection, we try more modern implementation of policy gradient algorithm that allows for boot- strapping and recurrent neural networks: Important-Ve -eirhhtt Actor-Learner Architectures (IMPALA) (Es- peholt et al. 2018). IMPALA is a synchronous variant of (batched) A2C which uses importance-weigg weighttd corrections for its value function estimates, and has been show to work on visual environments such as the Atari suite (Bellemare al. 2013) and scale.
Specifically, adapt the implementation provided in Haiku (Hennigan et al. 2020 to online (batched) agent consistent with the other agent implementations in OpenSpiel. We rum two IMPALA agents against each other, similarly to the basclines in Section 4.1 sweeping over hyper-parameters policy learning weight E {0.001, 0.0004, 0.0001}, entropy cost. E (0.01, 0.003, 0.001}, unroll length E (20, 50, 100), and It E (1,3, 5). For IMPALA we use a basic recurrent network with two hidden layers of size (256, 128) followed by an LSTM layer of size 256. After 600k episodes of training, the best population return and within-population exploitability achieved by this agent was 16.43 and 9.3, respectively (in both cases when 1) for an aggregate score of 7.13.
11

---

Published in Transactions on Machine Learning Research (10/2023)
Algorithm 1: Population Reinforcement Learning (PopRI.) for Two Players
Input: Bot population B, batch size B, prediction weight p, probability of self-play p Input: RL learning rule & (e.g. IMPALA)
for balch number 2, do
Reset data sets Dj = D2 = 0
for episode t E (1..... B) do
for i E [1 do
Place PopRL agent in player slot i Sample UNIFORM([O, 1])
if < p then
Place PopRI. agent 3 i in player slot j Set opponent identification label to [B]
else
Sample bot b ~ (NIFORM(B) with index binder, where 0 S binder < [B] Set opponent identification label o to bindez
end
Generate episode using agents (i, 3 --- i)
Add data from episode to D, with combined loss: Loss = (1 p))Los(()) + pLossaux end end
Perform separate learning steps on data sets D1, D2
end
4.5.1 IMPALA as General Bot Exploiter Agent
Since IMPALA was designed to be a single-agent algorithm and was unable to significantly improve over the baseline algorithms, we verify its ability to act approximate best response ("exploiter") agent when playing against the population. In this sctup, new opponent bot is uniformly sampled at the start of each episode to play against the IMPALA agent. By using similar hyper-parameter sweeps as before, we find a small set of good hyper-parameteed (learning rate 0.0004, entropy cost 0.003, and vary only the unroll length E (20, 50)).. In this case, we find IMPALA can consistently reach a population return of 220 after 200k episodes, which is significantly higher than the independent RL setting.
One benefit of PBE the ability assess the capacity of an agent generalize. In particular, we evaluate the ability of an IMPALA exploiter agent against bots that it has not trained to exploit. We apply cross validation over bot opponents: IMPALA trains against 33 agents, and evaluates only against the left-out set of 10 agents. We average the performance over 50 distinct sets of 10 left-out opponents. IMPALA consistently reaches an average of 120-130 per episode against the left-out bots, a significant drop compared to when training and testing opponent distribution are identical.
To investigate whether the generalization ability can be improved, inspired by UNREAL (Jaderberg et al. 2017), we augment the network and training procedure with an auxiliary task of opponent prediction. A output head is added that predicts which specific opponent bot the agent is facing, and standard classification loss is added to the combined RL loss with some prediction weight p E {0.001, 0.01, 0.1, 0.5). The results are shown in Figure 0 (in Appendix A.4 We observe that opponent identification helps, and improvements get are better with higher p. We also measure the average difference of the area-under-the- curve (interpreted as population return advantage per episode) between & = 0.5 and the baseline = 0), achieving 12.94, 15.81, 13.00, and 11.05 training episodes 25k, 50k, 100k, and 175k, respectively. The advantage diminishes slightly over time but maintains significant positive advantage well into the training run.
12

---

Published in Transactions on Machine Learning Research (10/2023)
Aggregate Score for PopRL
250
200
Aggregate aegate Score
150
100
PopRL (rho 0.01, p 0.1)
PopRL (rho 0.01, p 0.25)
PopRL (rho = 0.1, p 0.1)
50
PopRL (rho 0.1, p 0.25)
PopRL (rho 0.5, p 0.1)
0
PopRL (rho = 0.5, p 0.25)
0
100000
200000 Training episodes
300000
400000
Figure 5: PopRL's aggregate score across hyperparameters. Each setting of hyperparameters was averaged
across 5 seeds.
4.5.2 PopRL: A Hybrid Population-Bass Based Training Algorithm
We now propose new general training algorithm ("Population RL" in constrast to "IndRL") based on IMPALA with opponent identification. Inspired by Restricted Nash Response (Johanson et al., 2008) and game-theorotic population-based approaches (Lanctot et al. 2017 Hernandez 2022 Strouse et al. (0211)). PopRL mixes between best respouding to itself and to population members. Rather than train against the bot population only, a PopRI. agent trains against an augmented population containing the bots and an identical copy of another PopRL agent that is also independently training (concurrently or alternately). At the start of each episode, with probability p the opponent is set be the other PopRL agent, (with probability identification auxiliary task, but unlike before the number of classes is one greater to include identifying the other PopRL learning agent (44 instead of 43). The motivation is to leverage the population to train a generalist agent, while still guarding against being exploited by a similar learning agent. Pseudo-code is presented Algorithm
p) it is set to uniformly sampled bot. In both cases, the agent uses opponent
Results are shown in Figure The best combination of hyper-parameters is able achieve an aggregate score of 247.02, placing PopRL just behind IOCAINEBOT and far above BIOPIC, between second and third ranks. In addition, we show how the best PopRL agent scores against individual bots compared to GREENBERG in Figure 7 (Appendix A.4 while they score similarly on many of the agents in the population, they differ significantly against several bots.
We believe that PopRL combines several strengths in one approach: first, as was shown Scction 4.5.1 IMPALA equipped with a recurrent neural network and an opponent identification auxiliary task learns a very good bot exploitation strategy that generalizes across bots.
13

---

Published in Transactions on Machine Learning Research (10/2023)
To ensure that it doesn't become exploitabile, another copy of the agent tries to find weaknesses (100p% of the time) while its learning its bot responses. In essence, this combination balances maximizing return and minimizing exploitability.
5 Discussion: Limitations, Related Work, and Future Directions
The purpose of this paper is to propose new challenge for multiagent RL algorithms. While there has been impressive progress in MARL research producing agent that indicate human-level win rates in Go (Silver et al. 2016), Dota. 2 (Berner et al. 2019), and Starcraft (Vinyals et al. 2019) humans (and some cases, AI bots (Timbers et al. 2022: Wang et al. 2022 could find counter stuntersstrategies that consistently exploit these agents. Our arguments this paper are aligned with the ones in Player Games (Schrid et al., 2021): that win rate alone is insufficient to determine human-level ability, and that game-theorets reasoning is important to demonstrate robustness against exploits. However, unlike Player of Games which includes a complex search procedure, this domain focuses on repeated interactions with population in the purely RL domain.
We highlight the proporty that RRPS is simple, has a low barrier to entry and yet complex dynamics when playing against a populations akin to Axelrod's tournaments in iterated prisoner's dilemma (Axelrod 1984): additionally, RPPS allows the challenge to focus mainly on the tenuous balance between maximizing reward while not being exploitable. To the best of our knowledge, this a unique addition to the community: there is currently no challenge benchmark for learning agents that highlights these two axes in single domain coupled with population of human-crafted expert bots for finer-grained evaluation.
5.1 Limitations
The main limitation is that employing PBE requires bot population and that the approximation quality of hetthh-pppuuttioo-oplltaaalllll depends on there being bots that can exploit the various population mistakes within the repeated game. As such, the specific benchmark we are proposing and bot population we are characterizing are necessarily restricted to the domain of RRPS.
The population-based evaluation could be extended to other domains. To do so would require another set of hand-crafted bots. The aggregate score we are proposing in this paper only addresses the combination of maximizing reward and minimizing explottability; a different domain may nccessitate new metrics.
Finally, mentioned above, this benchmark focuses RRPS and the interactive between reward and exploitability. As such, RL algorithms more generally, such as in (Zwwadzki et al. 2014). The values achieved by the learning agents we ran should be interpreted as yardsticks to challenge the community in a specific way, rather than providing an evaluation methodology for the broader problem of general MARL.
not proposing PBE
evaluation methodology for comparing multiagent
5.2 Related Work
RRPS has served as dyadic test bed for understanding strategic interactions in both human and agents. The plurality of this research has focused on either learning opponent models or understanding learning dynamics. These two topics can be seen in the implementation of many of the competition bots -note, the majority of the related work occurs after the competitions. On opponent models, Cook et al. (20TT) showed that imitation can be a powcrful tool for learning strategies and understanding your opponent. Concurrent to this work, Lebiere & Anderson (2011) investigated human decision-making models, and notably discussed both that benefits of opponent models and their complications due to many models corresponding to the same behavior. [Brockbank & Vul (2021) quantified human play by quantifying information gain regarding the behavior of your opponent.
The other plurality of work focuses on the evolving dynamics of players. Evolutionary algorithms have been the primary mechanism for understanding how learners develop over time (Ali et al., 2000 Bédard-Couture & Kharma 2019). Waan et al.] (2014) found that strategies often follow a cyclic pattern in the actions that are employed throughout gameplay. However, studies have also discussed the possibility of chaos
14

---

Published in Transactions on Machine Learning Research (10/2023)
accounting for the inability to learn in RRPS (Sato et al. 2002). Despite the apparent simplicity of RRPS, it is clear that there is still much we have to understand about learning dynamics within it.
5.3 Future Directions
There are several avenues of potential future work. Firstly, we one could try different. populations in RRPS. Specifically, there are larger populations are openly available that could be easily adapted to fit within the PBE framework (Knoll et al. 2011
Secondly, a more complex extension would be to introduce a continual version of RRPS with a dynamic population that can introduce or remove agents over time. This would test the general capability of an RI. agent in a more dynamic way: could the agent also guard against anticipated new strategies, or learn now ways to counter these new strategies?
Finally, it could be interesting to population-based evaluation methods applied to larger extensíve-form gainss. There are several games being adopted by the community with hand-crafted strategies being used as benchmark. Poker is a popular challenge domain; poker competitions were organized and run regularly as well as several man-machine poker competitions against human experts. Population- could offer an altornative evaluation methodology, especially for games that current AI techniques have not mastered, such as variants beyond Texas Hold'em. em. There are several examples of domains in the cooperative Al (Dafoe et al.) 2020), such as Hanabi [Bard et al., 20200] Hu et al., 2020) and Overcooked (Carroll et al,, 2019 Strouse al. 2021a), that might benefit from a replicable population-based evaluation rather than evaluation in self-play or one-time human/ AI evaluations. And finally, population-baser evaluation may be the only satisfying way to evaluate the quality of agents in general-sum n-player games, such as Diplomacy, where there is no clear solution concept nor definition of "optimal strategy".
evaluation
6 Conclusion and Future Extensions
We propose repeated Rock, Paper, Scissors, a population of previous tournament bots, and population-basod based evaluation as new challenge in sequential decision-making with multiple agents. The bots range widely in terms of population return, exploitability, and predictability. Several standard Deep RL baseline algorithms, that have attained human-level performance on various challenge domains, fail to achieve both high reward and to be robust to a population of RRPS bots.
We show that an LLM agent able to achieve an aggregate score of 155.2, significantly higher than most baseline RL algorithms. The best agent trained via self-play (a contextual regret minimizer using SAOL) achieves aggregate score of 148.51. When training against the population, IMPALA is able to to leverage opponent identification to learn general responses, and when combined with population-based training, achieves a high aggregate score of 247.02; but, even with the added information, it was unable to defcat the top two bots.
Several modern approaches are unable to defeat the top two hand-crafted bots, IOCANBoT and GREENBERG, that were submitted over 20 years ago. We invite the community to try their own approaches to achieve this result and contribute their findings in an effort to understand how learning agents could both maximize reward and reduce exploitability simultaneously.
References
F. F. Ali, Z. Nakao, and Y.W. Chen. Playing the rock-paper-scissoon game with a genetic algorithm. In Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No. 00TH8512), volume 1, pp. 741 745, 2000.
John R. Anderson. Rules of the Mind. Lawrence Erlbaum Associates, Inc., 1993.
Robert Axelrod. The Evolution of Cooperation. 1984.
15

---

Published in Transactions on Machine Learning Research (10/2023)
Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moïtra, Edward Hughes, Tain Dunning, Shibl Mourad, Hugo Larochelle, Mare G. Bellemare, and Michael Bowling. The Hanabi challenge: A new frontier for ai re- search. Artificial Intelligence, 280, 2020. URL
50004370219300116
Bédard-Couture and N. Kharma. Playing iterated rock-paper-sccisooss with an evolutionary algorithm. In Proceedings of the 11th International Joint Conference on Computational Intelligence (IJCCI), pp. 205 212, 2019.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253- 279, jun 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.06680, 2019. URL attp://arxiv org/ abaaabaaaaa 1912.9912999
Darse Billings. The first international roshambo competition. ICGA Journal, 23(1):42-50) 50, 2000a.
Darse Billings. The Second International RoShamBo Programming Competition, 2000b. httpp:://group: google. commg//omp.ppo ai c/3g// Retrieved Nov 30th, 2022.
Erik Brockbank and Edward Vul. Formalizing opponent modeling with the rock, paper, scissors game. Gomes, 12(3))70, 2021.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of lcarning about humans for human-ai coordination. In H. Wallach, H. Larochelle, A. Beygelz- imer, d' Alcché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http::ttt//ooceedinngge wewpoddiinnnneuripppp neurips www.ppp....ppppprrrrrooc.c filesspaper/ 00s9://tee5b10001127771128822210
pdf
R. Cook, G. Bird, G. Lünser Unser, S. Huck, and C. Heyes. Automatic imitation in strategic context: players of woock-pppr-sscisoon imitate opponents' gestures. In Proc. the Royal Sociely B: Biological Sciences. The Royal Society, 2011.
Kai Cui and Heinz Koeppl. Approximately solving mean field games via entropy-regularized deep rein- forcement learning. In Twenty-Fous Fourth International Conference on Artificial Intelligence and Statistics, 2021.
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative Al. CoRR, abs/2012.08630, 2020. URL
https://arxiv. www.wbs.//////// http//bbs/012..... 20120000333
Amit Daniely, Alon Gonen, and Shai Shalev-Shwart Strongly adaptive online learning. In International Conference on Machine Learning, pp. 1405-1411, 2015.
Dan Egnor. Tocaine powder. International Computer Games Association Journal, 23(1):33 35,
((15533-35,11 2000.
Lasse Espcholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In Jennifer Dy and An- dreas Krause (ods.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1407-1416. PMLR, 10-15 Jul 2018. URL https: 7/proceedings... mlr. ppress/v80/espehol111 html hatprceedig.......prrss////////p
16

---

Published in Transactions on Machine Learning Research (10/2023)
S. Hart and A. Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica, 68())1112711150, 2000.
Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020. URL
Daniel Hernandez. Oppoment awareness at all levels of the multiagent reinforcement learning stack. PhD
thosis, University of York, 2022.
Daniel Hernandez, Kevin Denamganai, Yuan Gao, Peter York, Sam Devlin, Spyridon Samothrakis, and James Alfred Walker. generalized framework for self-play training. In 2019 IEEE Conference on Games (CoG), pp. 1-8 1-8, 2019. doi: 10.1109/CICC (L109//II..2119.0 201..099.884
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. In Thitty-sizth International Conference on Neural Information Processing Systems, 2022.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-play" for zero-shot coordination. In Hal Daumé III and Aarti Singh (cds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4399 4410. PMLR, 13-18 Jul 2020. URL htttps://proceedlngg mlr. httpp//oeeeeiaaaa....rrrreresa//
Shinji lto. A tight lower bound and efficient reduction for swap regret. Advances in Neural Information Processing Systems, 33:18550-18559, 2020.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. 5th International Conference on Learning Representations (ICLR), 2017.
Michael Johanson, Martin Zinkevich, and Michael Bowling. Computing robust counter-strateeiess In J.C. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems 20 (NIPS), pp. 721-728, Cambridge, MA, 2008. MIT Press.
Byron Knoll, Daniel Lu, and Jonathan Burdge. Rpscontest, 2011. http://waw.rpacoteest com/
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Thirtieth International Conference on Newral Information Processing Systems, 2017.
Marc Lanctot, Edward Lockhart, Bean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Péro- lat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafici, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Brad- bury, David Ding, Schastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. OpenSpiel: A framework for reinforcement learning in games. CoRR, Abs/1908.009453, 2019. URL http://arxiv http://aaxiv.or//bb///eeee 0aabs//998800940 09453
C. Lebiere and Anderson. Cognitive constraints on decision making under uncertainty. Fronliers in psychology, 2:305, 2011.
FAIR Meta, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komcili, Karthik Konath, Minac Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human- level play the game of Diplomacy by combining language models with strategic reasoning. Science, 378 (6624):1067-1074 2022. doi: 1..122//sciece.adeoon URL https::/ h/www. science. wtt//dr/ggss////eoooooooo 1126/science ade9097
17

---

Published in Transactions on Machine Learning Research (10/2023)
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 533, 2015. doi: 10. (038/naturre) 14266. URL bettps://doi.orge. wrt.://orrr/r....///aaataaiiii 10t.///ttaaaee123
Volodymyr Muih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Thirty- Third International Conference on Machine Learning, pp. 1928-1937, 2016.
Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T. Connor, Neil Burch, Thomas Anthony, Stephen McAleer, Romuald Elie, Sarah H. Cen, Zhe Wang, Audrunas Gruslys, Alcksandra Malyshova, Mina Khan, Sherjil Ozair, Finbarr Timbers, Toby Pohlen, Tom Eccles, Mark Rowland, Mare Lanctot, Jean- Mann-Bapiiste Lespiau, Bilal Piot, Shayegan Omidshafici, Edward Lockhart, Laurent Sifre, Nathalie Beauguerlange, Remi Munos, David Silver, Satin- der Singh, Demis Hassabis, and Karl Tuyls. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623):990 :9900996, 2022. doi: 10.1126/ 10.1126/science.addee /dd4
Y. Sato, E. Akiyama, and J. D. Farmer. Chaos in learning simple two-person game. In Proceedings of the Nutional Academy of Sciences of the Uniled States of America, volume 99, pp. 1748-4751. National Academy of Sciences, 2002.
Martin Schmid, Matej Moraveik, Neil Burch, Rudolf Kadlec, Josh Davidson, Kevin Waugh, Nolan Bard, Finbarr Timbers, Marc Lanctot, Zach Holland, Elnaz Davoodi, Alden Christianson, and Michael Bowling. Player of games, 2021.
Y. Shoham and K. Leyton-Brown. Multiagent Systems: Algorithmic, Game- Came-Theoretic, and Logical Founda- tions. Cambridge University Press, 2009.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Dricssche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanetot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, llya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukeuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484-4988 489, 2016.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnicszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, and over 300 additional authors not shown. Be- yond the imitation game: Quantifying and extrapolating the capabilities of language models. 2022. URL
http:://xxxvv..r.gooo
04615
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Thirly-Fifth Neural Information Processing Systems, volume 34, pp. 14502 14515, 2021a.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 14502- 14515. Curran Associates, Inc., 2021b. URL
797 98778466422111149744666022222220 Paper .dff
R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2017.
Oskari Tammelin, Neil Burch, Michael Johanson, and Michael Bowling. Solving heads-up limit texas hold'em In Twenth-Fourth International Joint Conference on Artificial Intelligence, 2015.
Finbarr Timbers, Nolan Bard, Edward Lockhart, Mare Lanctot, Martin Schmid, Neil Burch, Julian Schrit- twieser, Thomas Hubert, and Michael Bowling. Approximate exploitability: Learning best response in large games. In Thirly- Pirst International Conference on Artificial Intelligence, 2022.
18

---

Published in Transactions on Machine Learning Research (10/2023)
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Fwalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L.. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama,, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcnoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782) :330-3344 2019. doi: 10. 10.1038/s415866 /115//11116661 019-15566----1160 1724-z.
URL https://doi. org//0.
Tony Tong Wang, Adam Gleave, Nora Belrose, Tom Tseng, Joseph Miller, Michael Dennis, Yawen Duan, Viktor Pogrebniak, Sergey Levine, and Stuart Russell. Adversarial policies beat professional-leee go ais, 2022. URL https://arxiv..rrt http:/ab/2221..0000 00241
Zhijian Wang, Bin Xu, and Hai-Jun Zhou, Social cycling and conditional responses in the rock-paper-scisee game. Scienlifie Reports, 4:5830, 2014.
Erik Zawadzki, Asher Lipson, and Kevin Leyton- Brown. Empirically evaluating multiagent learning algo- rithms. CoRR, abs/1401.800 2014. URL http: //at:///axxiv... wttp://////....000000000 8077
19

---

Published in Transactions on Machine Learning Research (10/2023)
A
Additional Results
In this appendix, we give supplemental results referred the main text.
A.1
Full Ranking Bots
The performance and full ranking of bots in the population is given in Table
A.2 Hyperparameter Search
For Q-learning, swept over learning rates {0.001, 0.02, 0.01} and R (1,3,5, 10). We observed that while different learning rates had differently-shaped curves, that ultimately the differences were small (with a = 0.02 working best); on the other hand, the amount of recall made significant difference.
For DQN and BDQN we swept over syper-parameters batch size E {32, 128), R E {1,3,5) 5), learn- ing rate E {0.02, 0.01, 0.001]: replay bulfer capacity {10*, 109). For A2C we swept over hyper- parameters
(1,3, 5): E (0.99, 0.9, 0.75), entropy cost. E {0.01, 0.003, 0.001), policy learning rate E {0.0002, 0.0001, 0.00005), critic learning rate {0.0001, 0.0002, 0.0005). In all cases, networks were two- layer MLPs with layers of size (256, 128) and ReLU activations except the final output layer.
A.3 Language Model Agent
Language model prompt after two rounds of RRPS:
A repeated game of rock, paper, scissors is being played. Guess the next move based the game history. Game history (player1, player2) P,P P,S
Minor variations in the prompt did not significantly impact performance. The scores are shown in Table 7
A.4
IMPALA Agent
A.5
Behavioral Cloning
To access the extent to which the bots are predictable, we train action-prediction models that predict the bot's next action based on the full game history. We investigate three types of action-prediction models:
Individual: model trained to clone single agent's behavior against. the full population.
Population: a model trained to the full population's behavior against the full population.
k-Fold: model trained to clone a fold (n,, =30) of the population, and is also evaluated for gener- alization on the held-out population (nout 13).
Hercafter, the sub-population being modelled is referred to as the demonstrator population/indiiie /individual (c.g., in the case of the Individual model, it is the singleton bot). Common to all of the models is that the identity of the bots are never revealed.
Figure shows results for the case which a separate LSTM trained per bot. In Figure wC compare average action prediction accuracy of individual LSTM models a single LSTM model trained predict next actions for randomly sampled bot from the full population of 43 bots.
20

---

Published in Transactions on Machine Learning Research (10/2023)
Rank
Bot Name
Pop. Return
W.P. Expl.
Agg. Score
GREENBERG
288.153
3.648
284.505
2
10CAINEROT
255.003
5.006
249.997
3
BIOPIC
196.365
36.665
159.700
BOOM
169.119
27.928
141.191
SHOFAR
152.008
16.865
135.143
ROBERTOT
177.767
50.154
127.613
PHASSNNBTT
232.245
111.708
120.537
MODIBOT
203.162
90.158
113.004
9
SWEETROCK
146.250
41.207
105.043
10
PIEDRA
146.080
41.441
104.639
11
MARKOVBAILS
111.192
17.601
93.591
12
SUNNERVEBOT
138.054
45.490
92.564
MARKOV5
111.186
18.720
92.466
14
ANTIROTNBOT
121.387
58.616
62.771
15
HALBOT
212.429
176.229
36.200
16
MIXED STRATEGY
114.131
83.488
30.643
17
RANDBOT
0.234
1.197
-0963
18
PIBOT
4.516
81.000
-76.484
19
ACTR.
LAG2 LCG2 DECAY
146.319
236.865
-90.546
MARBLE
148.661
240.988
-92.327
GRANITE
149.252
241.840
-92.588
22
PREDBOT
167.112
267.687
100.575
23
ZQ MOVE
124.799
368. 744
243.945
24
MULTIBOT
56.057
307.065
251.008
25
TEXTBOT
-73.394
185.000
258.394
26
DEBRULIN81
10.250
301.679
-911.229
DRIFTBOT
-49.499
263.493
-312.992
28
ADDDRIFTBOT2
-41.855
283.910
325.765
29
RUSSROCKER4
172.334
529.751
357.417
30
SWITCHALOT
-82.877
315.612
398.489
31
ADDSHIPTBO13
-78.117
342.420
420.537
32
FOXTROTBOT
-51.019
407.418
458.437
33
FLATBOT3
-71.952
416.524
488.476
34
INOCENCIO
17.616
579.868
562.252
R226BOT
212.619
399.845
612.464
36
SUNCRAZYBOT
-83.609
578.089
661.698
37
SWITCHBOT
-173.178
497.182
670.360
38
PETERBOT
-174.238
927.986
1102.224
39
FREQBOT2
341.744
999.000
- 11340.744
40
COPYBOT
-475.327
997.000
- 1472.322
ROTATEBOT
602.641
998.121
1600.762
ROCKBOT
610.116
1000.000
11610.116
ANTIFLATBOT
648.420
999.002
-1647.422
Table 6: The full ranking of bots in the population.
21

---

Published in Transactions on Machine Learning Research (10/2023)
Chinchilla Parameters Size
Bot Name
400M
1B
7B
70B
AC L2 DECAY
123.3
11.5
28.0
13.1
ADDDRIFTRO12
27.5
52.4
82.4
89.7
ADDSHIFTBOT3
73.5
188.0
222.6
155.5
ANTIFLATBOT
995.0
995.6
991.4
992.6
ANTIROTNBOT
51.4
55.8
59.4
60.9
BIOPIC
193.5
63.4
-2288
20.0
BOOM
-65.7
10.4
-6.8
9.5
COPYBOT
981.0
981.0
983.0
979.0
DEBRUIJN8T
51.0
-11.0
30.0
20.0
DRIFTBOT
80.3
123.4
182.6
155.4
FLATBOT3
106.6
148.2
106.5
154.2
FOXTROTBOT
-1.7
57.3
44.8
33.9
FREQBOT2
598.0
774.0
871.0
919.0
GRANITE
-16.8
120.6
156.8
128.6
GREENBERG
305.9
121.2
108.6
39.8
HALBOT
300.9
145.6
134.9
-8.9
INOCENCIO
449.3
337.6
793.1
382.1
IOCAINEBOT'
323.0
144.4
148.7
28.6
MARBLE
20.6
141.7
146.1
123.0
MARKOV5
-78.6
3.5
14.4
19.3
MARKOVBAILS
80.4
2.4
10.9
21.1
MIXED STRAT
15544
31.2
34.0
57.4
MODIBOT
206.9
-8.77
766.2
25500
MULTIBOT
198.0
211.0
366.0
224.0
PETERROT
652.1
815.2
831.0
846.2
PHASENBOTT
315.3
174.7
165.4
45.8
PIBOT
-2.0
11..0
1.0
9.0
PIEDRA
42.4
42.8
44.4
44.7
PREDBOT
114.22
12.3
24.4
67.6
R226BOT
372.4
364.3
370.8
344.4
RANDBOT
1.1
3.4
6.3
-3.7
ROBERTOT
94.5
-8.6
3.0
-5.8
ROCKBOT
998.0
998.0
996.0
994.0
ROTATEBOT
983.0
992.0
995.0
1000.0
RUSSROCKER4
234.4
55.1
-5.88
14.5
SHOFAR
-00.2
34.5
23366
15.8
SUNCRAZYBOT
292.5
389.7
423.2
466.3
SUNNERVEROT
141.3
-4..7
37.0
16.5
SWEETROCK
43.9
49.6
30.8
45.3
SWITCHALOT
116.5
123.9
115.1
154.5
SWITCHBOT
200.1
230.7
225.1
276.6
TEXTBOT
144.0
113.0
129.0
31.0
ZQ MOVE
80.4
154.9
196.2
196.1
POP. RETURN
110.1
177.2
198.6
201.0
W.P. EXPL.
323.0
174.7
165.4
45.8
AGG SCORE
-212.9
2.5
33.2
155.2
Table 7: LLM agent performance against bot population (avg over 10 runs).
22

---

Published in Transactions on Machine Learning Research (10/2023)
Average Population Return over 10 Held-out Opponents
Population Return over Held-out Opponents
140
120
100
80
IMPALA (rho = 0)
60
IMPALA (rho = 0.001)
IMPALA (rho = 0.01)
IMPALA (rho 0.1)
40
IMPALA (rho 0.5)
25000
50000
100000 125000 150000 175000 Training episodes
75000
Figure 6: Population return over held-out opponents when IMPALA is trained exploiter agent. Each method is averaged over 50 held-out splits.
Training The models are trained with behavioral cloning objective that maximize the action pretiiiprediction model's likelihood of playing demonstration action (from the bot). Demonstration data is generated dynamically by uniformly sampling demonstrator and co-player. Note, that the co-player is sampled uniformly from the full population for bot the Individual and Population models, but is sampled only from the within-fold population for the k-Fold model. Data generated in parallel by 20 processes populating a temporary data buffer that uniformly sampled to prevent correlation in complete batches from the same strategy profile. The training batches contain 128 sub-trajectories of length 20 providing a limited recall during training, but during evaluation full recall can be maintained within the learned memory. Each model is trained for 1B frames corresponding to 1M episodes.
The trained models are fixed and their predictability is measured by their agreement with demonstrator playing 100 episode for each unique profile (across both demonstration- and co-player-bots). Agreement is measured by average action accuracy across all episodes.
Evaluation
Model Implementation The models are implemented with a 2-layer LSTM with sizes [64, 64). The output final layer of the LSTM projected into action space by an 3-layer fully-connected neural network with sizes [64, 3).
23

---

Published in Transactions on Machine Learning Research (10/2023)
Return versus opponent
dest drcay
Aigle
caprit
demtor
(arbe))
pronhhy
Dertur of the Poptii Areen berp un. marh Carestgge over 50 epploos)
natlor
munneal
mirt
strep
exttarr
muttlur
retertar
materbo
made
pedior
ander
coutor
amercior
Romberaau
Butber
Figure 7: Population return of PopRL agent against individual bots compared to GREENBERG.
24

---

Published in Transactions on Machine Learning Research (10/2023)
randbot
individual models single model
switchalot
markovbails
markov5
switchbot
phasenbott
flatbot3
shofar
iocainebot
sunNervebot
russrocker4
r226bot
biopic
boom
driftbot
mixed m strategy
greenberg
adddriftbot2
Predicted bot
actr lag2 decay
foxtrotbot
robertot
halbot
piedra
sweetrock
modlbot
inocencio
sunCrazybot
antirotnbot
granite
multibot
addshiftbot3
marble
predbot
zq momo
peterbot
antiflatbot
freqbot2
debruijn81
pibot
textbot
rotatebot
rockbot
copybot
0.0
0.2
0.4 Action prediction accuracy
0.6
0.8
1.0
Figure 8: Average action prediction accuracy comparison between individual LSTM models and a single LSTM model cloning all bots.
25


Reviews:
[{"review_id": "EZyd6o2sOr", "reviewer": "Reviewer_KsZo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very clear and precise in presentation, with smooth execution. I enjoyed the discussion as well as the number of analyses and the discussion overall. \n\nI have a few critical comments as well. \n\n1. Sec 4.1 is probably not very informative. I expect (please correct me if I am wrong) that the training algorithm, which does not maintain a population of agents, necessarily falls into 'local minima' strategies and, as such, cannot reliably provide much information. I'd probably keep it in the appendix only.\n2. The results in Fig 5 seem to be growing. I'd appreciate it if at least one run is taken to the 'asymptotic performance.'\n3. The results in Fig 5 how many seeds they are using. If only one (as nothing else is said), then the conclusions in this section are quite doubtful on the statistical end.\n4. It is somewhat unsatisfactory that only one network size is studied for the RL setting. It is quite \n\nApart from these specific comments above, I am unsure if the rock-paper-scissor game is challenging enough to be a meaningful benchmark for some not-too-short time.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the repeated rock-paper-scissors game as a benchmark for multi-agent learning. More specifically, the authors have collected $43$ bots of various strengths against which a new agent might be compared. Two principal metrics, which are used, are $\\texttt{PopulationReturn}$ and $WithinPopExpl$. The former measures how an agent performs 'on average', while the latter captures the worst-case scenario, i.e., how much it can be exploited by some other agent. $\\texttt{AggregateScore}$ is used to aggregate these two into one score.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "85nKNGrv6X", "reviewer": "Reviewer_RrDa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Challenging benchmarks for learning are important and has historically driven research in many areas of machine learning. Consequently, contribution in this area in the domain of multiagent learning is a promising endeavor.\n* The paper evaluates a range of well-known algorithms and uses a couple of known libraries, including Openspiel.\n* A new algorithm, PopRL, is proposed.\n* The paper asks some interesting questions: Is it possible to train a model (here LSTM) to predict bots' actions? Is it possible to reason about the whole population of bots by learning on replays from games played between agents in a subpopulation? Can RL agent learn to achieve good results in the population and not be exploitable? What are the capabilities of LLMs in RRMS or identifying the opponent? Can IMPALA agent perform good as an independent agent or as an exploiter, or do the auxiliary tasks help?\n\nThe paper lacks in the following aspects:\n* Focus:\n\t* What are clear objective of the benchmark and why is it relevant? What exact deficiencies of the current learning protocols can a RRPS highlight? Can the benchmark be used to reason about the algorithms weaknesses and point out towards future research directions.\n* Clarity of exposition:\n\t* A big concern is that, although the paper provides various experiments, it only marginally discusses the results. \n\t* What one would typically expect from a benchmark paper is an analysis of existing algorithms' behavior on the benchmarks, highlighting the limitations, and hypothesizing about underlying causes.\n\t* The reader would also expect from a paper like this, some guidance concerning future research or a list of open problems.\n * Structure:\n\t * There is no related work section (information in this respect is scattered across the text, most notably Section 4). Including such a section would help to organize the paper better, keep the experimental part cleaner and focused on discussion of the results, reassure the readers that the Authors did a proper literature overview, and help with future research. \n\t * There is no limitations section. Such a section is always useful: it not only shows that the Authors are aware of various constraints associated with the work, but allows researchers to avoid pitfalls and speed-up research.  \n\t * Technical discussion not directly relevant for the text (such as the hyperparameters grid search) can be moved to Appendix.\n\t * The paper would benefit from one central piece of information (e.g., a Table) where all the results of all the tested methods are summarized.\n\nOther:\n*  The paper could use some of the guidelines for RL evaluation from [1] (as to the number of seeds or uncertainty bounds).\n* Table 3 seems to have an error: the three metric values in row QL (R=10) do not reconcile. Also a comment regarding this number should be reviewed (last sentence in Section 4.1)\n* The last sentence on page 7 is not clear (\"averaged over a sliding window of 50 most recent evaluations \")\n* On page 4, it is written \"Figure 2 below\", where in fact the Fig 2 is above. Additionally, the a consistent naming should be used (see e.g., \"RoShamBo\" in caption of Figure 2).\n* The choice for the aggregated metric could be discussed more (e.g., that it has some desirable properties).\n* The paper only allocates one fourth of a page to discuss and describe a new algorithm it proposes.\n* In footnote 3, it is written \"The precise LLM used and number of parameters will be revealed in the camera-ready copy, if accepted for publication.\" The paper should contain all the necessary information to make its assessment possible.\n\nTo summarize, after a revision suggested above, the paper could be accepted, as some of the TMLR's audience would most likely be interested in its contents.\n\n[1] Agrawal, R., et al. Deep Reinforcement Learning at the Edge of the Statistical Precipice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a benchmark for multiagent learning that evaluates the agents on the repeated rock-paper-scissors (RRPS) game, against 43 bots submitted during two RRPS competitions held in the early 2000s. Three metrics are proposed and various well-known learning algorithms are evaluated. The paper also proposes a new training protocol, dubbed PopRL, that bases on IMPALA, UNREAL, and Restricted Nash Response.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the \"Strengths And Weaknesses\" section.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pwAI4OLFiw", "reviewer": "Reviewer_wm2S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The manuscript presents a straightforward idea executed in a good idea. The benchmark seems to be well packaged, and the paper presents a reasonable amount of information to the hypothetical reader looking to employ it in their research.\n\n- Particularly solid is the usage (and description) of  each agent that makes the benchmark, whose simplicity yet great emerging dynamics are definitely a big strength (since it gives the benchmark a lot of degrees of freedom in terms of evaluating future work).\n\n## Weaknesses\n\n- I found the problem that this benchmark is trying to solve -- i.e balancing (a) average return versus (b) best-case robustness against a worst-case opponent -- to be poorly justified. It is definitely true that MARL has an evaluation problem that single-agent RL does not, but IMO that stems primarily from MARL being not a single problem definition, but instead a (large) categories of problems. The manuscript presents regularizing (a) vs (b) as if a good combination of the two would be sufficient to solve _a_ general MARL problem, but there's a large set of issues that remain (say, e.g. zero-shot / one-shot / ad hoc multi-agent stuff). Given that (b) seems to stem primarily from game theory, I'd perhaps expect the manuscript to draw a better justification from that large literature of why this is a problem worth caring about (because it's not clear to me in general practical terms).\n\n- With all due respect, the baselines story (i.e. Ch. 4) from my perspective could use some work, and the comparisons against the bots feels particularly a bit unfair.\n  + Section 4.1 presents agents whose observability wrt. history is limited to max 10 joint-action steps, which seems a weird limitation given that (a) the state space is so simple, (b) the game dynamics are trivial but intrinsically stochastic, and (c) the amount of information gained per step is relatively low. \"Recall\" is bound to be a bottleneck against bots that can do long history matching, and seems like something that could be dealt with fairly trivially with some simple state abstraction or representation learning.\n  + It feels like a couple of the subsections were put there just to (superficially) cover relevant topics. Section 4.2 is very light on details, and doesn't seem to be that relevant (why should it matter *for this benchmark* that LLMs can play RRPS?). Section 4.4 also has no explained relevance to the benchmark (which, to be clear, I'm sure it obviously does -- but it's just not in the manuscript).\n\n- Given the wide range of bots available in the dataset, I was expecting the manuscript to use baselines to provide a large degree of insight about the kinds of bot-agent games that arise from using these benchmark. However, the evaluation often stopped at the level of average performance wrt. the metrics that the authors cared about. Seems like a missed opportunity to kickstart research on the benchmark appropriately.\n\nEDIT: cleaned up my wording on Section 4.1 -- apologies for the miscommunication, I didn't mean for the comment to be dismissive of the work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work argues that repeated rock-paper-scissors (RRPS), a well studied game in the game theoretical literature, can be used as a benchmark for multi-agent reinforcement learning. The authors primarily aim to provide a system for studying the problem of balancing average return versus best-case robustness against a worst-case opponent.\n\nTo build the environment, they employ bots previously made for other RRPS competitions, forming a population of bot players that agents can be compared against.\n\nThe manuscript focuses on analyzing the properties of this population of agents, as well as comparing the performance (wrt. the benchmark) of a few common RL baselines.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n- A better justification of the importance of the problem for folks working outside of game theory would be great to have. What are the (positive) consequences of finding learning algorithms that can effectively trade worst-case performance vs average returns in these settings? I would also be careful with over-claiming here, as noted in the above section.\n\n- I don't understand why the benchmark is usually testing on a sample of the bots rather than the entirety of the population. Isn't it kind of the point that one might wish to understand the distribution of performance against qualitatively different opponents? In that case, isn't it necessary for such a low number of opponents to test against all of them? Some clarity around the reason behind these choices would be good.\n\n- Tighten the story behind the baselines. What are the baselines trying to tell the reader? How far have they pushed the benchmark? (Suspicion: does the benchmark becomes \"solved\" if deep policies are given a much longer history window?)\n\n- It is IMO unacceptable to write something like this: \"The precise LLM used and number of parameters will be revealed in the camera-ready copy, if accepted for publication.\" -- the manuscript is already public, so why the secrecy? The lack of such information makes Section 4.2 largely pointless for the reader. Please add this information (as well as prompt employed, and other such relevant information to make it possible to replicate the result).\n\n### Typos\n\n- Page 2\n  + Identical (Note: they hopefully aren't?)\n  + Would a table\n  + Unless [...], use R = 1...\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a benchmark for multiagent reinforcement learning based on the repeated rock-paper-scissor game.
The reviewers found the paper well-written, addressing an interesting topic, and providing a solid benchmark.
On the other hand, they raised several concerns (lack of motivation and focus for the proposed benchmark, poor discussion of the results, related works, and limitations).
The authors addressed most of these concerns and uploaded a new, improved version of their paper.
However, not all the changes proposed in their rebuttals have been implemented (e.g., "Figure 2 below" and the values in Table 3).
To accept the paper, the authors have to address all the points discussed in their rebuttals.
Decision: Accept (poster)