Adversarial Diversity in Hanabi
Authors: Brandon Cui
Andrei Lupu
Samuel Sokota
Hengyuan Hu
David J Wu
Jakob Nicolaus Foerster
Venue: ICLR 2023 notable top 25% (2023)
Keywords: coordination
diversity
multi-agent reinforcement learning

Abstract: Many Dec-POMDPs admit a qualitatively diverse set of ''reasonable'' joint policies, where reasonableness is indicated by symmetry equivariance, non-sabotaging behaviour and the graceful degradation of performance when paired with ad-hoc partners. Some of the work in diversity literature is concerned with generating these policies. Unfortunately, existing methods fail to produce teams of agents that are simultaneously diverse, high performing, and reasonable. In this work, we propose a novel approach, adversarial diversity (ADVERSITY), which is designed for turn-based Dec-POMDPs with public actions. ADVERSITY relies on off-belief learning to encourage reasonableness and skill, and on ''repulsive'' fictitious transitions to encourage diversity. We use this approach to generate new agents with distinct but reasonable play styles for the card game Hanabi and open-source our agents to be used for future research on (ad-hoc) coordination.

Published as a conference paper at ICLR 2023
ADVERSARIAL DIVERSITY IN HANABI
Brandon Cui +t MosaicML.
Andrei Lupu"
Samuel Sokota
Meta AT & FLAIR, University of Oxford
Cameeie Mellon University
Hengyuan Hul
David Wu
Jakob N. Foerster FLAIR. University of Oxford
Stanford University
Meta AI
ABSTRACT
Many Dec-POMDPs admit qualitatively diverse set of "reasonable" joint policies, where reasonableness is indicated by symmetry equivariance, non- sabotaging behaviour and the graceful degradation of performance when paired with ad-hoc partners. Some of the work in diversity literature is concerned with generating these policies. Unfortunately, existing methods fail to produce teams of agents that are simultancously diverse, high performing, and reasonable. In this work, we propose a novel approach, adversarial diversity (ADVERSITY), which is designed for turn-based Dec-POMDPss with public actions. ADVERSITY relies on off-belief learning to encourage reasonableness and skill, and on "repulsive" fictitious transitions encourage diversity. We this approach to generate new agents with distinct but reasonable play styles for the card game Hanabi and open- source our agents be used for future research on (ad-hoc) coordination
INTRODUCTION
A key objective of cooperative multi-agent reinforcement learning (MARL) is to produce agents capable of coordinating with novel partners, including other artificial agents and ultimately humans. In order to make progress this objective, number of works have focused on the general chal- lenge of ad-hoc team play, which is to create autonomous agents able to "efficiently and robustly collaborate with previously unknown teanmates on tasks to which they are all individually capable of contributing team members" (Stone et al.) 2010). To evaluate such agents, many works on ad-hoe coordination rely on evaluation setups similar to the one proposed by [Stone al. (2010), pairing the agents at test time with partners sampled from a pre-determined pool.
The value of such evaluations depends on the size and quality of the pool of partners. A pool that is too small or too homogeneous may not be representative of all possible play-styles, and provide an inaccurate evaluation of the coordination capabilities of an agent. For this reason, previous works in coordination have relied on various approaches to generate diverse pool of partners.
A first approach to handcraft policies, either directly or by shaping the reward at train-time (AI- brecht 2015 Barrett et al. 2017 Zand et al., 2022), but it requires domain knowledge and scales poorly. Another is to train a population with varying hyperparameters or by deploying multiple RL algorithms on the same task (Nekoci et al. 2021 Zand et al. [2022] Albrechtt 2015). The diversity achieved this way is unclear, since it is a byproduct of the variability of the algorithms used rather than being actively optimized for. Yet other works augment training with diversity loss (Lupul jet al., (202)) or save multiple checkpoints (Strouse et al., (02T) but often do not report the level of diversity achieved. Measures of diversity based on policy similarity struggle in settings where not all different actions result in "meaningfully" different outcomes Furthermore, the number of possible trajectories often so large that becomes easy to maximize diversity objectives without learning qualitatively different policies imagine humanoid robot that wiggles finger at any sin-
'Equal contribution. Correspondence at brandon:@nossiimm.... and al aluuueetta.
Work done while Meta Al.
https: / / thub. com/
While "meaningfully different" is environment dependent, we claborate on what mean in Section

---

Published as a conference paper at ICLR 2023
11-1
[+1
yA (12))
Ti
Me-1) y² Val (Trz2)
2
Rea
H+2
142
(a) Standard OBL transition
(h) Repulsive OBI. transition
Figure I: Standard (a) and repulsive (b) OBL transitions when training Te for n steps. ADVERSITY
trains policy Ne on (b) with probability A, and on (a) otherwise. Differences between the two are in red.
gle time step in the episode rather than learning different walking styles. This particularly true in multi-agent settings, where the number of trajectories is exponential in the number of agents.
To avoid such pitfalls, another approach, followed by Charakorn et al., is to require distinet policies to be incompatible by training them to obtain a low score when paired in mixed teams. In Section 77] we show that this results in policies that simply identify whether they are playing in self-play (SP, with themselves) or in cross play (XP), with another agent. In the latter case, they purposely "sab- otage" the game by selecting actions that minimize return, such as playing unplayable cards in the card game Hanabi. Adapting such policies in an ad-hoc pool is non-goal, since they do not represent meaningfully different policies but rather actively poor and adversarial game play.
This is in line with previous findings that partners trained with SP rely on arbitrary conventions and symmetry breaking, making collaboration with them difficult (Hu et al. 2021). As such, producing strong and meaningfully diverse policies in Dec-PON PecPPS remains an important unsolved problem.
2020 2021b) Lupu et al.
We address this problem in turn-based settings with public actions by introducing adversarial diver- sity (ADVERSITY), a policy training method which, given a repulser agent, produces an "adver- sary" whose conventions are fundamentally incompatible with those of the repulser. The key insight of ADVERSITY is that it prevents the adversary from identifying whether it is currently in SP or playing with the repulser agent by randomizing between the two every time step. In other words, even if the adversary is in an action-observattion history (AOH) that is incompatible with the repulser agent, the adversary paired with the repulser agent with a fixed probability X, in which case the next reward is inverted. Likewise, with probability 1 - X the adversary is instead paired with itself.
Crucially, the choice of the current partner determines not only the sign of the reward (positive or negative) and the partners' action, but also how the entire AQH thus far is interpreted. Here, we build on top of the fictitious transition mechanism from off-belief learning (Hu et al., 2021b OBL) and use the belief model of the repulser policy the corresponding repulsive transition. In nutshell, the adversary is currently paired with the repulser policy, the transition is sampled from a belief distribution that assumes the repulser policy took all actions thus far.
When the adversary is paired with itself rather than the repulser, we must avoid the feedback loop between induced beliefs and future actions, which would allow the adversary form arbitrary conventions. Thus, we train in a hierarchy: we start with the grounded belief. like in OBI,, and at each level l we compute the vanilla transitions using the belief model of the level below, 1. The adversary is trained to maximize a "difference value function"""" which estimates the forward looking discounted difference between adversarial and vanilla transitions under their corresponding beliefs.
For the first time, ADVERSITY enables us to produce number of high performing, diverse, and symmetry-invann policies for the challenging collaborative card game Hanabi.
2 RELATED WORK
Stone et al. 2010) and Bowling & McCracken (0055) were among the first to formulate the ad-hoc
teamwork ("impromptu team play") setting, requiring autonomous agents to collaborate with novel teammates. Works in the literature have often taken a type- based approach, where potential partners are grouped in a number of possible types, which must be identified at test time. Different types (or classes) of polices have notably been generated through genetic algorithms (Albrecht et al. 2015) to
2

---

Published as a conference paper at ICLR 2023
induce diversity. However, these methods usually require hand-coded heuristics and are difficult to scale to complex, high dimensional environments that we consider in this paper.
Other works promote diversity in multi-agent RL (MARL) to robustify agent by having it train against a pool of partners (Lupu et al. 202T; Strouse et al. 202T), Lupu et al. uses an auxiliary loss to induce diversity, while Strouse al. selects older checkpoints of the model; both suffer from the arbitrariness and sabotage issues which we address in our work. Canaan et al. (2019) generates a diverse pool of agents through MAP-Elites (Mouret & Clune 2015), but the algorithm relies on low- dimensional rule based agents and fails to produce strong Hanabi agents (scores > 19 points). In contrast, our method scales high-dimensional settings and does not require manual hard-coding.
Hu et al. 2020) introduce setting where the goal is to maximize the cross-play (mixed team) performance between independently trained agents from the same training algorithm. For clarity, we refer to this metric as the intra-algorithm cross-play (intra-AXP) in this paper. Huu et al. (2020) highlighted the importance of "reasonable" policies, in particular focusing on symmetry breaking- Ma et al. (2022) also investigates producing reasonable policies through the inductive biases of various model architectures. They find that certain architectures achieve diversity in simple settings.
The work most similar to ours is that of Charakorn et al. (023). They learn incompatible policies by maximizing SP scores, while minimizing XP scores a pool, and optimizing the lower bound on variations between policies. As we show in Section 7 this approach leads to agents that obtain low XP scores by sabotaging the game rather than discovering fundamentally incompatible conventions.
BACKGROUND
3.1 TURN-BASED DEC-POMDPS WITH PUBLIC ACTIONS
In this work, we assume a turn-based Dec-POMDP (Oliehoek] 2012) which can be described by a tuple (n, A, P.r, 7), with number of agents n, state space S, action space A, transition function P S x A S + [o, 1] determining the probability over next states, reward function r: x A + R, observations function (ss) = o, and discount factor 7 E [0, 1]. We also define (On, ao.-........... ..... 1r) and the action-observation history (AOH)
the trajectory up to time as of agent as T (o), ao..... a,- of). For a given player i, a policy x' r'(a(T;) (T] is a function that maps AOH to distribution over actions. Importantly, this setting is turn based, meaning only one agent acts at any given time step. We also assume public actions, such that all agents observe the action selected by the acting agent-a limitation we inherit from OBL.
Multiple trajectories can produce the same AOH. The relationship between them is therefore prob- abilistic and depends on the policy that generated the trajectory up time L. We refer this distribution over trajectories as the belief, B, (Te(TT) = P((TlT(,,)) m).
3.2 SELF-PLAY TRAINING
Self-play (SP) is a general class of training methods for multi-agent RL. When training the (joint)
policy in SP, we unroll the policy up to time step f, producing the partial trajectory тr. Each agent f then observes of and samples an action from x ((-1r)). The team receives a reward re = r(T,, a.) and transitions trajectory Te41. In two-player turn-based games, the process is repeated for agent -i to obtain T and r/-2- Finally, SP computes the TD target
dsp = r. + TT+1 + + +
which assumes that actions will be selected according to policy и at all future time-steps.
During SP training, past, present and future actions are assumed to be sampled from the policy being trained. fully cooperative settings, this enables agents to communicate additional information about their private observation (and therefore the trajectory 7:) by selecting actions to sharpen their partner's 's belief B, (rl Arbitrary correlations arising during training (e.g. due to random initialization) are therefore reinforeed, they provide useful information to other agents future time-steps. While such correlations can improve the team's 's return, they are unlikely to reoccur on independent training runs, resulting in a policy that is brittle and difficult to cooperate with.
3

---

Published as a conference paper at ICLR 2023
3.3
LEARNING
Off-Belief Learning (Hu et al. b OBL) is training algorithm designed to address the short- comings of SP training in tum-baaed Dec-POMDPs with public actions. It prevents agents from learning arbitrary and brittle conventions peculiar to the random correlations of a particular run.
The general issue in applying SP training to cooperative MARL is that enables information feed- back loops that reinforce spurious correlations between actions and meanings. For example, con- sider in the cooperative card game Hanabi a policy лл where by chance n, often takes a particular action X when its partner has a playable card, and otherwise takes action Y. Upon observing X, the partner's belief over the card will be that playable. The partner therefore plays it, resulting in positive reward which reinforces the convention that "X means playable". Crucially, this occurs even if neither X nor Y reveal any extra information about the playable card.
The key insight of OBL is to break this loop by fixing belief model Bo, which is independent of TA- At each step, OBL then reinterprets the AOH based on this Bg by sampling a new fictitious trajectory Tf Bo. The correlation between the trajectory seen by A and its action are then restricted to what "survives" given this trajectory resampling. Thus, if πA takes action when a card is playable, but that same card isn't playable under T/, then reward is obtained if the partner plays the card. In particular, OBL uses B, = B.g where Tn is fully random. Thus, in expectation, "X means playable" will only be reinforced if X carries verifiable information about the playability of the card.
More formally: like SP, when training policy π, OBL unrolls æ to obtain the trajectory тr. AOH of and action of (--rr() for agent i. However, OBL breaks the information feedback loop by entirely ignoring the reward r(Tr,a)). Instead, it assumes access to the environment simulator, as well as a belief model Bx, of another policy mo, and samples fictitious trajectory 7/ B= (-17f).
In this fictitious trajectory, OBL. first applies the "real" action of and then fictitious action a++1 ~ T ((11(41). It thus receives fictitious rewards ri = r(T', at) and r'41 = r(((+1 a,41 '). and fictitious future trajectory r/42 and AOH T'42 ' The TD target used for training the policy is therefore
8OBL. = r + tr'41 + V
(2)
In essence, rather than training on the real rewards seen during the transition, OBL samples ficti- tious transition at every time step. This fictitious transition reinterprets the AOH of agent i as having been produced by policy To rather than As result, actions that sharpen the posterior distribution (x(r((r) over trajectories given и are no longer reinforced in general. Instead, agents must rely on actions which convey information about the trajectory in spite of this resampling.
As stated earlier, OBL assumes To to be a fully random policy. As a result, spurious correlations between the real trajectory and the observations do not propagate and agents can only rely on actions that convey verifiable information about the state. We refer to such agents as "grounded."
OBI. be iterated produce hierarchy of policies, with each level me being trained on fictitious transitions sampled from a belief model B = Ba (T)) This process was shown to reliably produce policies with similar conventions that are devoid of symmetry breaking and are altogether considered as more "reasonable" partners for coordination. practice, we use neural networks B, trained with supervised learning to approximate B, to enable fast inference and sampling.
4 PROBLEM SETTING AND MOTIVATION
Many works seek to train a large pool of diverse policies from scratch. Instead, given access to a fixed repulser policy Je, our goal is to train an adversary x-a new agent with a different play style. We predict that many methods addressing the latter task could theoretically be deployed at larger scale to produce a population of agents. However, this is not the primary focus of our paper.
We aim for the adversary to exhibit meaningful diversity from the repulser: to adopt conventions and strategies that differ drastically from those of the repulser, to the extent permitted by the environ- ment. This is in contrast to policies that merely exhibit small differences in action probabilities or state occupancy but otherwise converge the same high level strategy (Lupu et al. 2021). Further- more, we seek adversaries that achieve high return and that are reasonable teammates. We further discuss these desiderata in section 4.2
4

---

Published as a conference paper at ICLR 2023
4.1 MOTIVATION
The main motivation to work is to generate quality partners to serve as test suite for ad- hoc coordination. While it is often easy to produce a population by varying the training method or hyperparameters, there is no guarantee that such approaches will produce meaningfully diverse policies. Additionally, it is concerning if small training variations do produce diversity, as it suggests that the training algorithms used do not reliably output policies with consistent conventions. Indeed, such policies will fail in the context of intra-AXP, making them poor partners for ad-hoc evaluation.
Instead, with a method that is able to take a small number of reasonable partners and output highly skilled and very distinct adversaries, it becomes possible to generate a partner pool that both covers a more substantial portion of the policy set and is less likely to be populated by poor collaborators. With access to such a method, it will become easier to test the ability of an agent to generalize to unseen and meaningfully diverse partners.
A method for producing adversaries to skilled agents has other possible applications. Previous stud- ies (Hu et al. al. [021b] Lupu et al. (02T) have shown that biases in the training method or the environ- ment structure can result in RL agents repeatedly converging to the same equilibrium, Discovering new ways of solving a task is therefore an open problem. Were it solved, it would have applica- tions to software testing, for instance by discovering new ways of breaking feature simulating different user behaviours when using a program.
4.2 DESIDERATA FOR ADVERSARIES
Before proposing an approach to train adversaries, we first establish how to measure success. Whether two policies are meaningfully diverse is environment and task-specific. For instance, two humanoid robots moving their arms differently may considered irrelevant if the goal is to discover new gait patterns. Similarly, the feasibility of training strong and distinct policies is environment- dependent (c.g. a task may have a unique equilibrium). Therefore, we do not formally define the notion of "meaningful diversity", and instead determine the desiderata of such agents, each under- stood to be potentially limited by the environment itself.
Skill level: Past works indicate that diversity often comes the cost of performance. While the environment structure may in itself be at cause, this effect can also be attributed to the difficulty of simultaneously optimizing for return and diversity (Parker-Holder al. 2020). A good method would minimize this drop, and produce adversaries that are as strong as allowed by the setting.
Note that this is not claiming that ad-hoc partners of beginner intermediate levels are not useful. However, it is usually easy to obtain such partners by selecting earlier checkpoints of trained policy, Thus, our goal produce adversaries near expert level (i.e. as close to SOTA as possible).
"True" Diversity: Another core criteria to adversaries is that they adopt distinct strategies from those of their respective repulser policy. The first way evaluate whether the adversary π and its repulser adopted different strategies is by evaluating them in XP, as mixed team. Low XP can be indicative of distinct and incompatible conventions, but is not sufficient. Indeed, low XP can also be explained by having brittle policies that fail at the slightest deviation from SP. Even worse, since is a function of μu, it is possible that æ learns to identify when it is paired with μ,, and deliberately performs poorly, i.e. sabotages the game (see Section
While secondary, is also desirable that the adversary strategies differ in an interpretable way from those of the repulser policy. For instance, an adversary to a robot sports team that is particularly aggressive may instead play much more defensively.
We require adversaries to be "reasonable" or "well-behaved" in informal sense. First, this means policies that do not sabotage, as explained above. Secondly, we wish to avoid arbi- trary conventions symmetry breaking, since those are unlikely to be recovered even by subsequent runs of the same algorithm, making for very inflexible and uncooperative partners.
Reasonableness:
While we do not have a problem agnostic means of identifying the failure modes listed above, do have Hanabi specific metrics as explored in Section 6 Furthermore, as explained in Section [5] method avoids these failure modes by design, building on the desirable properties of OBI.
5

---

Published as a conference paper at ICLR 2023
5 ADVERSARIAL DIVERSITY
Given a repulser policy ps, how do we train an adversary policy л = Adv(p) satisfying the criteria above? A simple starting point is to train to maximize SP return while minimizing return when paired with μ.. However, as we show in section they playing in SP and sabotage if not. Thus, it fails to produce reasonable policies. We therefore introduce ADVERSITY, which overcomes these issues with two key insights.
7 this approach leads to agents that identify whether
Algorithm ADVERSITY training at level E for one data collection and training step. We present the two player case. At timestep t, the active player is i and the next player is -i.
_ belief model from previous ADVERSITY level.
b Te: new ADVERSITY policy being trained, Qs: the Q-network that constitutes the policy me-
& μc: repulser policy, B,: repulser belief model, A: repulsive probability.
procedure ADVERSITY(Q;: Tr. p. B,, D)
Initialize training episode & = EmptyList and sample initial environment To P(ro)
while Not Torminaal (T) do
Get observation 7/- = ('():) and action of ~ Te(TT)) for the active player i if x ~ ~(0, 1); < À then
B,,, puprtnrr = μs. w
else
B = BB Tpumer Te. w = =1
end if
Sample fictitious trajectory -/ B(-|7()
Apply active player's 's action on the fictitious trajectory T(41 = P(-I7), af) and collect reward r Partner observes and picks fictitious action a7.1 = T (0 (F(+a))
Apply partner's action on the fictitious trajectory T'+2 = P( a/71) and collect reward r'+11 Compute target Q. = wr + wrr'++1 + + (O'(((( Append observation, action and target to training episode E append((T)) at, Qt)) Apply active player's is action on the real trajectory and get to the next state 7441 ~ P(-17s, at) end while
Add training episode to the replay buffer D.add(E) Sample training episode from the replay buffer E' ~
gradient descent = 0 a 4 [()), where [(0) = (000 Q+I 26
end procedure
Firstly, we prevent sabotages by re- re-sampling the partner for every AOH, such that the adversary can- not condition on the partner's identity to choose its action. At every time step, the adversary's ' partner is the repulser with probability X, and itself otherwise. However, doing so directly perturbs the tra- jectory distribution and impacts learning. For that reason, we leave the trajectory being unrolled intact and perform this partner switching within the fictitious transitions of OBL. Thus, in "repul- sive" transitions (when π is paired with μ)) we not only sample the next action from the repulser, but also the repulser's 's belief model resample the current trajectory. This reduces the likelihood of being fif-iistribution and reinforces actions that are incompatible with μ.. Conceptually, on those transitions we pretend that all prior actions in the episode were taken by the repulser.
Secondly, we prevent arbitrary conventions by training a hierarchy of adversary policies, we. On vanilla transitions (when the adversary is paired with itself), each ग. uses a fixed belief model from the adversary policy at the level below, Tz-1- Like OBL, the lowest level belief is the grounded belief, i.c. the unique belief corresponding to a uniformly random policy. All belief models are trained using the supervised learning procedure described in Hu et al. (202aa).
we proceed as follows: at given level f, ADVERSITY follows a similar training pattern to OBL by unrolling policy we on the real trajectory and computing a fictitious transition given of ~ B and a,71 (-117771) However, with probability X, the algorithm utilizes a "repulsive" fictitious transition rather than the vanilla OBL one. In that case, we instead sample
Technically,
T! (, and the partner's action from μ. We then flip the fictitious rewards to obtain the repulsive target
dAd- = - 9r/ 1 + + 2 (TZ42') ).
(3)
This procedure is summarized in Algorithm [ aand each transition is illustrated in Figure 1

---

Published as a conference paper at ICLR 2023
Self-Play Worst Response
ADVERSITY repulser XP
repulser
SP
repulser XP
Intra-XXP
SP
Intra-XXP
Rank Bot
23.86 + 0.09
0.0 + 0.0
0.64 + 1.0
24.22 + 0.16
1.94 + 0.0
24.09 + 0.0
Color Bot
23.90 + 0.14
0.01 + 0.0
1.36 + 2.0
24.03 + 0.13
2.98 + 1.0
10.93 + 8.0
Clone Bot
23.90 + 0.13
0.0 0.0
6.22
2.0
23.94 0.16
7.48
2.0
21.38 + 1.0
OBL
23.82 + 0.1
0.0 + 0.0
3.39
+ 5.0
24.11 + 0.07
9.07 + 8.0
8.33 + 5.0
Table 1: Score table SPWR and ADVERSITY for 4 repulser policies. Each number averaged over independent adversaries. Both approaches produce policies with high SP and low repulser XP, but ADVER- SITY achieves higher intra-AXP, which demonstrates that the policies are a more principled and reproducible function of the repulser.
Accounting for both transitions, at every time step the policy learns difference Q-values that estimate the expected future discounted reward difference between vanilla and repulsive transitions:
Qui (rt, a²) = (1- - A) B-B 1(t)), ait "are fr's + Wh'41 + + ( ')3
+ AE B,B, ai 1' (-r - V (T242))
diff (T/) Σ me(alt:) (F).a)
Breaking it down, the first line corresponds to vanilla OBL. It implies reinterpreting the past as having been produced by given policy æ--1 and acting according n ever after, reinforcing only actions that lead to high expected return when interpreting the partner's actions according to πT-1- This prevents feedback loops in SP where the agent can learn spurious belief's from noise about which past actions correspond to what unobserved trajectories, and then use those belief's to signal information and reinforce them into arbitrary and brittle conventions.
The second line represents repulsive transitions. Resampling the current trajectory assuming the AOH was produced by ş results in af being reinforced if it incompatible with p's conventions and leads to low immediate rewards. At the next step, we assume our partner is μr, thus simulating XP between the repulser and the adversary. Here again, negating r'41 means we reinforce actions that misinterpreted by μ.
Finally, in both vanilla and repulsive transitions, we maximize expected future discounted difference reward, denoted by (+ Conceptually, these difference value functions assume that at every point in the future the repulser will intervene for time step with probability X, at which point rewards are inverted and the entire past re-interpreted according their belief model Ba,..
Overall, this procedure pushes Ne to select actions that achieve high return under Br,_,, and that are simultaneously incompatible with j's conventions.
This section describes training single policy me on top of B and μs. Because the initial belief By is restricted only rely on grounded information, the skill level of क is limited. Therefore, to improve skill we follow the procedure from Hu et al. (222b)) and iteratively learn higher levels Te. At each level, we decrease X, up Nonetheless, since each level uses the belief level of the previous level, in our settings the final play style incompatible to μ1, as we show in Section 7 This shows that we truly obtain novel equilibria.
at which point our training reverts vanilla OBL.
EXPERIMENTAL SETUP
Here describe the evaluation setting and bascline. Training details are included in the Appendix.
6.1 HANABI
We implement and test our method in Hanabi, a large scale cooperative card game proposed as a challenging benchmark for Dec-POMDP research (Bard et al. 2020). Hanabi is played with 8 hint tokens, life tokens and deck of 50 cards, each having a rank between and and one of five colors. It is game for 2-5 players, but restrict ourselves to the 2-player version.

---

Published as a conference paper at ICLR 2023
Rank Bot
Color Bot
Clone Bot
OBL
Non-repulser
SPWR
2.36 + 0.17
1.14 = 0.19
2.17 + 0.07
2.04 = 0.20
1.57 + 0.10
ADVERSITY
0.06 + 0.01
0.10 + 0.02
0.05 + 0.01
0.09 + 0.05
0.05 + 0.01
Table 2: Average number of sabotages per game by the row agents playing with the column agents they are trained be different from. Each pair is evaluated on 1000 games, averaged over seeds of the row agent. The "Non-repulser" column is when SPWR/ADVERSITY of one agent plays with the other three agents. For reference, in self-play, each OBL agent does 0.057 pure sabotages per game mistakes, risky bets and almost never loses all 3 lives.
The goal is to form stacks of cards in rank order for each of the five colors. The final score, between O and 25, is given by the number of cards successfully stacked by the end of the game. At any given time, player's s have five cards in their hands, and can only see their partner's 's cards. The players must therefore communicate effectively with their partners that they act in an informed manner.
One their turn, a player has up to 20 different actions. They can either a) discard one of their cards, b) attempt play one of their cards, hint all the cards of chosen color in the partner's hand, or d) hint at all the cards of a given rank in the partner's is hand. Discarding replenishes one hint token. Playing a card results in being placed on top of the pile of its color if it's the next logical card in that pile. Otherwise, the card is lost and the team loses a life. Hinting provides limited information the partner and consumes a hint token. Hinting is not allowed if there are no hint tokens left, and discarding is not allowed if all 8 tokens are available. If the team loses all 3 lives, the game ends prematurely, with score of Finally, the game also ends when there are no cards left in the deck.
6.2 SELF-PLAY WORST RESPONSE
As baseline, we implement "Self-Play Worst Response" (SPWR) a PPO agent using the same neural network architecture and hyperparameters as above but trained on SP data with probability 1 À and in XP with the repulser with probability X. Our SWPR experiments set À 0.25. An entire game is either SP or XP, with no switching within a single game. When in XP, the rewards received are inverted. This a simpler version of LIPO from Charakorn al. (2023).
7 RESULTS
We evaluate the skill level, diversity, and reasonableness of our method against the SPWR baseline. For both our method and our baseline, we train adversary seeds for each of 4 repulser policies. In addition to vanilla OBL level 5, the repulsers are 3 of the baseline policies inspired by Hu et al. (2021b) namely Rank Bot, which an Other-Play (Hu al. (200)) (and therefore color equivariant) policy favouring rank hints, Color Bot, which is a reward-shappe policy favouring color, and Clone Bot, which a supervised learning bot trained on human data.
We first see in Table that both the SPWR and ADVERSITY agents achieve high SP, corresponding to high skill in Hanabi, and very low XP scores when paired with their respective repulser, showing that both methods produce policies that are incompatible with their repulser. However, ADVER- SITY shows clear advantage over the SPWR in terms of intra-AXP scores, computed between independent adversary seeds. SPWR produces different policies every run, resulting in very low Intra-AXP score. In contrast, ADVERSITY agents tend to be quite similar, as shown by the lower SP-XP gap. Adversaries to Color Bot and OBL are exceptions and have low Intra-AXP scores.
To evaluate whether the adversaries exhibit meaningful diversity from their repulser, we first look at conditional action matrices, presented in appendix A.4 These display the conditional probability of player's 's action di+1 given the previous action, a, We find that both ADVERSITY and the SPWR exhibit meaningful diversity from their repulsers and between adversaries to different repulsers. For example, when Rank Bot is the repulser policy, ADVERSITY consistently produces adversaries that use color to indicate playable cards. Similarly, while OBL tends to respond to discard actions by also discarding card, adversaries OBL learn instead to discard their last card to hint play.
8

---

Published as a conference paper at ICLR 2023
(a) matrix with self-play play wonst response agents
(h) XP matrix with ADVERSITY agents
Figure 2: XP matrices between Hanabi models: 4 repulser models (Rank Bot, Color Bot, Clone Bot, OBL) and their respective adversaries, trained either through SPWR (a) or ADVERSITY (b). Each entry is averaged over 2000 games. Red squares highlight repulser-SPWR or repulser-adversary pairs. Both SPWR and AD- VERSITY produce policies that have low XP scores with their repuisers, but ADVERSITY exhibits wider range of scores when paired with ad-hoc agents, indicating graceful degradation and more reasonable policies.
In terms of method consistency, the action matrices for different ADVERSITY seeds tend to be similar, reinforcing the idea that is reproducible. SPWR seeds, on the other hand, differ wildly, explaining the low inter-AXP score mentioned previously.
Finally, Figure [b] shows the XP matrices between bots including the repulsers and one of each adversary seed. Notice that SPWR adversaries have near-zero XP scores with virtually every partner; a red flag supporting the hypothesis that they learned to identify when not in SP and purposely throw the game. Meanwhile, ADVERSITY agents exhibit graceful degradation of ad-hoc performance depending on the similarity to the partner's ' policy, indicating much more reasonable policies.
We verify that unlike ADVERSITY, SPWR adversaries exhibit sabotaging behavior in Hanabi. We do this by measuring sabotages, the number of knowingly unplayable cards (based on revealed information) played by the agent. We measure the average number of sabotages per game when SPWR and ADVERSITY are paired with their respective repulser agents and report results in Table 2 ADVERSITY consistently has a low number of sabotages (< 0.1) per game, whereas SPWR has at least 1 sabotage per game and in many cases > 2. The sabotages are lower for SPWR(Color Bot) simply because a color hint does not immediately reveal whether a card is definitely unplayable. The SPWR(Color Bot) agent simply plays unhinted cards blindly, which is not necessarily a sabotage by our strict definition, but a poor move nonetheless. Moreover, SPWR agents sabotage all non-SP games, not just the ones with the repulser they were trained with. This indicates that the poor XP performance of SPWR comes not from playing a reasonable reward- maximizing strategy that happens to be meaningfully different and incompatible with other agents, but from "deliberately" playing bad actions upon identifying that its current partner is not itself. We also verified that the 0.1 mean incidence of sabotaging for ADVERSITY in line with vanilla OBL evaluated in SP, i.c. corresponds to a standard rate of mistakes or risky bets. On different metric, SPWR responsible for a dominant amount of 2.67 + 0.05 out of life losses per game while, ADVERSITY is only responsible for 1.42 + 0.05-rought, half of the total mistakes. This also indicates that SPWR tries hard to terminate games deliberately, while ADVERSITY policies fail due to meaningful incompatibility, without any party trying to explicitly sabotage.
Sabotaging:
8 CONCLUSION AND FUTURE WORK
In this paper, we introduce ADVERSITY, a method for producing highly skilled and reasonable policies for fully cooperative task that play according to meaningfully diverse conventions. While our results show that both ADVERSITY and our bascline produce agents that exhibit high skill and meaningful diversity from their repulser, only ADVERSITY agents are also reproducible on independent runs and reasonable, as indicated by the graceful degradation of their performance with different ad-hoe partners.
The main limitation of our method is the high computational cost, making it difficult to scale the method to a large number of adversaries. Were this issue solved, ADVERSITY could theoretically be used to produce large pool of diverse agents by iteratively computing adversaries past models.

---

Published as a conference paper at ICLR 2023
REFERENCES
Stefano Albrecht, Jacob Crandall, and Subramanian Ramamoorthy, An empirical study on the prac- tical impact of prior belief's over policy types. Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Stefano Vittorino Albrecht. Utilising policy types for effective ad hoc coordination in multiagent systems. 2015.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020.
Samuel Barrett, Avi Rosenfeld, Sarit Kraus, and Peter Stone. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence, 242:132-171, 2017
Michael Bowling and Peter McCracken. Coordination and adaptation in impromptu teams. In AAAI, volume 5, pp. 53-58, 2005.
Rodrigo Canaan, Julian Togelius, Andy Nealen, and Stefan Menzel. Diverse agents for ad-hoc cooperation in hanabi. In 2019 IEEE Conference on Games (CoG), pp. 1-8, 2019. doi: 10.1 CIG.2019.8847944,
Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. Generating diverse coopera- tive agents by learning incompatible policies. In The Eleventh International Conference Learn- ing Representations, 2023. URL
Brandon Cui, Hengyuan Hu, Luis Pineda, and Jakob Foerster. K-level reasoning for (human-ai) zero-shot coordination in hanabi. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 4190 4203. Curran Associates, Inc., 2021.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. arXiv preprint arXiv:2003.02977 2020.
Hengyuan Hu, Adam Lerer, Noam Brown, and Jakob Foerster. Learned belief search: Efficiently improving policies in partially observable settings. arXiv preprint arXiv:2 09086, 2021a.
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, David Wu, Noam Brown, and Jakob N. Fo- erster. Off-belief learning. ICMI., 2021b.
//aaaii///raiai arxiv org/ abs 21 2103. 04000
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning, pp. 7204-7213. PMLR, 2021.
Mingwei Ma, Jizhou Liu, Samuel Sokota, Max Keiman-Weinee and Jakob N. Foerster. Learning to coordinate with humans using action features. CoRR, abs/2201.12655. 2022. URL https:
waxxiv.orq/abs.... 2200 12658
Jean-Baptiste Mouret and Jeff Clune. Illmiiaating search spaces by mapping elites. CoRR, 0bb//154400999, 2015. URL
wwwpa/wwww////////////////////// 04909
Hadi Nekoei, Akilesh Badrinaaraayanan, Aaron Courville, and Sarath Chandar. Continuous coor- dination as a realistic scenario for lifelong learning. In International Conference on Machine Learning, pp. 8016-8024. PMLR, 2021.
Frans A Oliehoek. Decentralized pomdps. In Reinforcement Learning, pp. 471-503. Springer, 2012.
Jack Parker-Holder Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:18050--188 2020.
10

---

Published as a conference paper at ICLR 2023
Peter Stone. Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. Twenty-Fourd AAAI Conference on Artificial Intelligence, 2010.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Evereu. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14515, 2021.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Fre- itas. Dueling network architectures for deep reinforcement learning. In Maria-Florina Bal- can and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016. New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1995-2003. JMLR.org, 2016. URL http: /rrccedd ngs. ml www.mm..p...//aaa...mmm / https/////ww/agfff/ /wan///aggli. 6. ht.l
/
Jaleh Zand, Jack Parker-Holder, and Stephen Roberts. On-the-fly strategy adaptation for ad-hoc agent coordination. arXiv preprint 00X1v:2203.08015, 2022.

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 TRAINING DETAILS
Our implementation based on the open sourced OBL code with two main modifications. We first replace the recurrent Q-learning backbone with PPO as it runs faster and requires significantly less memory. Then, we implement a synchronous method that trains all OBL levels simultancously.
Otherwise, we use distributed training set up similar to (Hu et al. 2021b), which we detail in the appendix. All bots are trained for 3000 epochs, each epoch consists of 1000 gradient steps. We select the checkpoint with the highest SP score.
The original OBL trains multiple levels of policies sequentially, using the output policy of the pre- vious level the imput policy of the new level. In ADVERSITY, we train all levels simultancously for faster wall-clock time. These policies are denoted as To, W1 7 and their corresponding belief models are denoted as , To warm up the belief model and avoid having too many invalid samples, we first train a belief model Bo on the uniform random base policy тo and initialize all B; = Bg. Then, L policy training tasks and L belief training tasks start at the same time. The belief task of Bi gets a latest copy of every 50 gradient steps and the policy task of gets a latest copy of Bi-1 every 50 gradient steps. The details of each individual belief follows the exact configurations of the original OBL paper and each policy task uses the PPO-OBL method described above.
For each adversary, we train a hierarchy of 7 levels, setting À 0.25 for / = 1 and decreasing by 0.08 every level (min. O). Levels S are trained simultancously, followed by levels 2 5, also trained simultaneously and with beliefs initialized at Bi = B1. This split was forced by limitations on the concurrent compute available to the authors, but anticipate no change in performance if all levels were trained simultaneously. The ADVERSITY numbers reported in Section a refer to the highest level of the hierarchy.
A.2 POLICY TRAINING DETAILS
We use large scale distributed training framework for policy training. To train a single policy, we 6400 games in parallel, each adding to centralized replay buffer. We achieve this by running 80 threads in parallel, with 80 games running per thread. All models are on GPUs and we dynamically batch all model calls in order increase inference speed. This schema also allows games on the same thread to forward environment calls while certain games wait for GPU calls. As done in (Wang al. 2016), when environment terminates, each game grabs all necessary objects: observations, actions, and targets, pads everything to a length of 80 and adds it to centralized replay buffer.
For every training step we apply the PPO update rule, but instead of using the real reward and advantage, we use the fictitious values. Every m = 10 training steps, we update the environment actors with the weights for the updated policy, As done in Cui al. (2021) synchronously train our hierarchy of beliefs and policies, querying for and updating all dependencies every p 50 training steps.
We utilize the same policy architecture as Hu et al. 2b22b). We utilize their public-private LSTM architecture. The public observation is encoded by a one-layer feedforwards neural network fol- lowed by LSTM. The private observation is encoded by three-layer neural network. We combine these encodings via element wise multiplication.
For all OBL experiments we compute the target with = 1 fictitious steps. We also sample the belief model 10 times and use the first sampled trajectory that doesn't 't violate card constraints to compute the fictitious targets. We then use simulator to produce transitions from the valid trajectory. Like Hu et al.) we discard the fictitious transition whenever the belief fails to produce a valid sample, which in practice happens less than 1% of transitions.
Implementation
The policy is represented by Public-LSTM network mo with value head and policy head. A large number of parallel workers generate data by sampling from a slightly outdated policy Ta and write that data into replay buffer D. One datapoint in D is entire trajectory T'. Although PPO
12

---

Published as a conference paper at ICLR 2023
normally does not need replay buffer, we still here to fully decouple inference and training for maximum speed. Its size is set to a small value of 1024 to minimize the instability caused by stale data. To is trained with the Adam optimizer (Kingma & Ba, 2014) minibatches of data uniformly sampled from the replay buffer. The value loss is E,, Li + >Vo(Ti+1) (T
The policy loss is EE E, min[rr(0)A))))))))))))) (l) + e)As] where rx(0) = xa(ailr1)
"gr(a:])() A.
StopGradient(rr + yup (7/+1) Vo (r:)]- We perform one gradient step per minibatch. We use 1-step bootstrapped value target instead of E, ra because it converges significantly faster and it fits well in the OBL fictitious target computation. Tgr synced with To every 10 gradient updates.
A.3 BELIEF TRAINING DETAILS
We utilize the same distributed training schema from policy training for belief training. This has also been done by (Hu et al.) 2021b). As done in policy training, we query and update dependencies every p = 50 training steps.
For belief training we store the true hand of the player along with the observation to train the be- lief. For training, we train an autoregressive belief model that predicts cards oldest to newest via supervised learning. More precisely, the belief model is trained to minimize the loss
C T:) = Σ
log p(hx (T] h1s--1)
(4)
k=1
where hg the kth card in the player's hand and n is the hand size (usually 5).
A.4 ADDITIONAL RESULTS
Rank bot
(b) Color
(c) Clone bot
(d) ORI.
Figure 3: Conditional action matrices showing p(ac+1las) for the 4 repulser policies
13

---

Published as a conference paper at ICLR 2023
ADVIOLSSED
010000 04
000400 W PA
02 0004
02
01
04
72
72
R
75
CL
C
0
C
M
M
M
SEIDO
DEDEDADEER PE CEEEE KEREMEMEN
mI #1 DE CRCaCIKI CO
01
01
00
02-
c
cx
C3
ARviclane: CETEA
AEVirione: SEERA
ADVYcooel SEE
04
Q
C2
G
RL
62
SEI06 M5 C3 MARS
SEED:
0100 DAEe 72 NS PS C2 M2
02
00
72
72
CL
0
C
G
G
G
BL
M
M
M
Figure 4: Action matrices for all SPWR agents.
14

---

Published as a conference paper at ICLR 2023
ADVIOLSSED
6
010000 04
0400 P
02 0004
02
01
04
72
72
R
CL
C
0
65
0
M
M
DEDEDADEER CR RE
mI DE EECACIKI CORRO
01
01
02-
02-
11
cx
C3
87
ARviclone: SETEa
AEVirion:: SEERA
ADNYcopel SEE
04
22
CL
Q
Q
G
8L
62
SEIDA
SE106
SEED:
010000 DEE
72 CE 12
PONMO 12
02
00
22
72
C
C
G
G
BL
M
M
Figure 5: Action matrices for all ADVERSITY agents.
15

---

Published as a conference paper at ICLR 2023
Comvergence
24:44
1.7
9:25
12:21
3:04
6.30
0:20
2:40
1.20
1-41
5:00
6:21
6:30
2:22
046
0.00
3.30
20-77
1.50
3:28
1.30
2.28
1.
0.79
5.00
0000
02
2.10
19
3.20
3.00
SWAM9)
3.20
L20
E.2L
3.50
00
0.
SPAR0n0 32570
3.50
5:00
XXY
R:00
7:7
00
6:22
SPINICRt SECDa
5.00
2.20
3:00
6:22
2.40
5.00
2224
20
83
6.57
5.20
6:25
00
2.22
2.05
5.00
SPARGC StED
0:30
3:20
2:00
6:35
3:00
3:20
6:30
5
00
5:30
1:40
3.20
3:00
3.00
0.00
3.00
6.30
2.00
5.23
22
6.3L
0.
SPerddmnd SE5T0
3.50
6:30
5:00
0:57
SPORG SIED
5:30
9:30
1:00
6:32
3:00
5:00
3:20
5:00
20
3:00
5:00
0000
SPWRLOE SEOTO
1:00
3.20
1.00
8:70
2:20
1.00
2.70
2.00
0.70
3:00
1:00
6:70
0000
8.70
10
3:00
orspncal Atatt
anwelco JEEB
Preeeriene HEEE
wwwftcctinnn Lud
FPOR SMD
Figure 6: XP matrix of the four repulser candidates and all the SPWR bots. Red rectangles indicate pairs of the form (X, SPWR())) Numbers below the diagonal
computed.
24.24
4.27
22:22
2.00
2.07
6:31
3:25
11.24
2155
640 34
3.10
2432
32.52
16.74
26.4
14:76
3.53
4.20
1.32
3.00
0.4
24/0
08
320
5.02
0.00
$4.20
20.23
25.05
23.7
62
11.57
2.40
6:22
36:45
34.50
167
2000
5.53
AENIRL AEMILSSEES SEE8S
3:00
5.02
0.00
0.03
10
24.0
23.00
4.76
5.44
3:62
4.18
2:48
0.43
252
06.23
3030
ADVIN 50E
5.03
8.00
3.20
2424
6.28
4.8
2.50
6.32
533
26.00
ADVISL SEID
5.03
8:00
3.00
24.36
21.23
1250
14:33
32.00
2.65
20.95
5.03
0.53
8.00
3.00
267
18:14
33.30
2.00
6:10
320
5.02
0.00
0.33
8.00
3.20
2.00
0.00
0.23
2.00
24:00
20:20
22.24
3.05
0.41
7.30
ATA SOECE
350
5.07
0.66
0.53
8:0
0:50
5:00
9:53
5.00
24.10
2.50
23.84
5.03
5.00
3.00
24.8
2.27
SE103
5.03
8.00
3.00
2.00
3.00
0.62
6.00
24.12
5.03
0.83
8.00
3.00
0.33
2.00
3,00
0.66
6.00
0.00
aDvesai S4000
Anvicteee StEre
Abv Anyrrinng WVED
40vilen stae
Figure 7: XP matrix of the four repulser candidates and all the ADVERSITY bots. Red rectangles indicate pairs of the form (X, Adv(X)). Numbers below the diagonal were not computed.
16


Reviews:
[{"review_id": "uFDRqJDc0mm", "reviewer": "Reviewer_p7TH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of computing \"diverse\" joint policies in Decentralized POMDPs (Dec-POMDPs) with public actions. The paper proposes a method based on off-belief learning augmented with \"repulsive\" fictitious transitions to encourage diversity. Finnally, the proposed algorithm (ADVERSITY) is run on the card game Hanabi to produce new playing agents with diverse play styles. ", "summary_of_the_review": "Overall, I think this is a good paper on an interesting topic. I only have some concerns on the theoretical groundings of the proposed method.", "strength_and_weaknesses": "STRENGTHS\n\n- The problem studied in the paper is interesting for the community focused on the designed of learning agents in games, as well as for the multi-agent reinforcement learning community.\n\n- The paper is well written and all the results are adequately commented.\n\nWEAKNESSES\n\n- While the paper has a mainly practical focus, I would have liked some theoretical analysis and/or formal claims explaining how the method is supposed to work. The experimental results seem promising in Hanabi, but further evidence is needed to conclude that the method could be applied to other domains.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Wsi6CWWjLw", "reviewer": "Reviewer_awGQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on producing strong and diverse policies in Dec-POMDPs. One important aspect of the proposed method is to prevent the adversary agent from identifying whether it is self-playing or cross-playing by randomizing between the two scenarios. It is also important to bear in mind that in the studied setting in turned based with public actions. The proposed method will lead to having several highly-skilled agents with different styles of play.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "Tackling the problem of MARL with diverse agents has given little attention in MARL. Authors try to fill this gap in MARL by introducing ADVERSITY which relies on off-belief learning to put more weights on actions with meaningful information about the trajectory.\n\nQuestions:  \n1- How was the SPWR baseline optimized in terms of hyperparameters or lambda? Using the same hyperparameters as ADVERSITY won’t be necessarily the optimal ones for SPWR. And for lambda, why 0.25?\n\n2- According to table 1, the difference between SPWR and ADVERSITY in SP is not statistically significant. Could you discuss this?\n\n3- Results are averaged over 3 seeds which is low. Increasing the number of seeds could be helpful in terms of trusting the results and see how the proposed method is robust against various scenarios.\n\n4- What does “reasonable” mean in the abstract?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0xHxAYKuLh", "reviewer": "Reviewer_wota", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A natural way to train qualitatively different policies for cooperative tasks is to penalize a new joint policy for performing well when individual components of this policy are paired with existing policies.  While this approach can identify multiple \"incompatible\" solutions to a cooperative task, this incompatibility may be achieved by learning to identify the behavior of existing policies, and then intentionally \"sabotage\" cooperation.  Experiments conducted in this work demonstrate just this type of behavior in the Hanabi environment.  To address this issue, this work presents the ADVERSITY algorithm, which modifies the existing \"off-belief learning\" algorithm to train joint policies that are incompatible with an existing policy, but cannot reliably identify that policy in order to sabotage play.\n\nThe ADVERSITY algorithm proceeds by rolling out the joint policy being learned.  At each step within this \"real\" trajectory, rather than computing the reward based on the true trajectory, ADVERSITY (like OBL) samples a \"fictitious\" trajectory that is consistent with an individual agent's observation history, but is generated by the existing policy that the new policy should fail to cooperate with.  In this way, during training, the new policy for an individual agent can never be sure whether it is interacting with the new joint policy, or whether it is interacting with the existing policy.  It therefore has no incentive to intentionally sabotage cooperation.\n\nExperimental results demonstrate that compared to directly penalizing self-play agents for performing well when paired with existing policies, policies learned with ADVERSITY are far less likely to engage in such sabotage, and are more effective in cross play with other ADVERSITY policies trained with different random seeds.", "summary_of_the_review": "The main justification for acceptance is the fact that this work is the first to address the problem of \"sabotage\" behavior when training diverse sets of policies for cooperative tasks.  They clearly demonstrate that their method is able to largely solve the sabotage problem in the Hanabi environment.  That said, the empirical results are limited, and are not sufficient to demonstrate that ADVERSITY can be used to learn \"good\" strategies for Hanabi in ad hoc play.  Additionally, the ADVERSITY method itself is a relatively straightforward combination of two existing approaches.\n", "strength_and_weaknesses": "The key contribution of this work is the exposition of the issue of partner sabotage when attempting to train mutually incompatible policies as a starting point for ad hoc cooperation.  They also contribute the first method that attempts to address this issue, by leveraging off-belief learning.\n\nThe main weakness of this work is the relatively limited experimental evaluation of the proposed method.  While the experiments clearly demonstrate that the ADVERSITY algorithm is able to avoid learning sabotage behavior, they do not evaluate the algorithm as part of a larger stack for learning policies capable of playing Hanabi with a variety of partners.  A more thorough evaluation, where ADVERSITY is used to train a population of different agents, which are then used to train ad hoc policies, might indicate whether the diversity encouraged by the ADVERSITY algorithm is actually helpful for ad hoc play.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%