Designing Skill-Compatible AI: Methodologies and Frameworks in Chess
Authors: Karim Hamade
Reid McIlroy-Young
Siddhartha Sen
Jon Kleinberg
Ashton Anderson
Venue: ICLR 2024 poster (2024)
Keywords: Skill-AI compatibility
Agent Systems
Decision-making
Chess
Deep RL
Primary area: general machine learning (i.e., none of the above)

Abstract: Powerful artificial intelligence systems are often used in settings where they must interact with agents that are computationally much weaker, for example when they work alongside humans or operate in complex environments where some tasks are handled by algorithms, heuristics, or other entities of varying computational power. For AI agents to successfully interact in these settings, however, achieving superhuman performance alone is not sufficient; they also need to account for suboptimal actions or idiosyncratic style from their less-skilled counterparts. We propose a formal evaluation framework for assessing the compatibility of near-optimal AI with interaction partners who may have much lower levels of skill; we use popular collaborative chess variants as model systems to study and develop AI agents that can successfully interact with lower-skill entities. Traditional chess engines designed to output near-optimal moves prove to be inadequate partners when paired with engines of various lower skill levels in this domain, as they are not designed to consider the presence of other agents. We contribute three methodologies to explicitly create skill-compatible AI agents in complex decision-making settings, and two chess game frameworks designed to foster collaboration between powerful AI agents and less-skilled partners. On these frameworks, our agents outperform state-of-the-art chess AI (based on AlphaZero) despite being weaker in conventional chess, demonstrating that skill-compatibility is a tangible trait that is qualitatively and measurably distinct from raw performance. Our evaluations further explore and clarify the mechanisms by which our agents achieve skill-compatibility.

Published as conference paper at ICLR 2024
DESIGNING SKILI-COMPATIBLA AI: METHODOLOGIES AND FRAMEWORKS IN CHESS
Karim Hamade kar@cs,toronto twwonto..diiiiooooo edu University of Toronto
Reid Sellro--Young reidmcy Gcs. www.stooootoooooooooo edu University of Toronto
Siddhartha Sen sidsen wwwirommottcccoooommmmmm Microsoft Rescarch
Jon Kleinberg
Ashton Anderson ashton Bechton&es.to toronto.ed edu University of Toronto
Cornell University
ABSTRACT
Powerful artificial intelligence systems are often used in settings where they must interact with agents that are computationally much weaker, for example when they work alongside humans or operate in complex environments where some tasks are handled by algorithms, heuristics, or other entities of varying computational power. For AT agents to successfully interact in these settings, however, achieving superhuman performance alone is not sufficient; they also need to account for suboptimal actions idiosyneratic style from their less-skilled counterparts. We propose a formal evaluation framework for assessing the compatibility of near-optimal AI with interaction partners who may have much lower levels of skill; we use popular collaborative chess variants as model systems lo study and develop AI agents that can successfully interact with lower-skill entities. Traditional chess engines designed to output near-optimal moves prove be inadequate partners when paired with engines of various lower skill levels in this domain, as they are not designed to consider the presence of other agents. We contribute three methodologies to explicitly create skill-compatible AI agents in complex maaiiign-making scttings, and two chess game frameworks designed to foster collaboration between powerful AI agents and less-skilled partners. On these frameworks, our agents outperform stalle-fftthearl chess Al (based on AlphaZero) despite being weaker in conventional chess, demonstrating that skill-compatibility is a tangible trait that is qualitatively and measurably distinct from raw performance. Our evaluations further explore and clarify the mechanisms by which our agents achieve skill-compatibility.
INTRODUCTION
As AT achieves superhuman performance in an increasing number of areas, a recurring theme that its behavior can be incomprehensible with their behavior. Game-playing is a familiar instance of this; it is well-understood, for example, that modern chess engines play in a style that is often alien even the best human players, to the extent that calling out "engine moves" (actions only computers would take) staple of professional commentary. It is also standard practice for human players use moves ted by chess engines as recommendations for their own play, only to find that they cannot successfully follow them up. Tn a way, chess AT (which is "near-optimal" human players (as examples of less-skilled agents) "speak" very different dialects that are often not mutually intelligible. This lack of compatibility leads to failures when less skilled agents interact with optimal ones, in settings where the less-skilled partics may be humans interacting with powerful AI systems, or simple heuristics intoracting with much stronger ones.
agents of lower skill levels, or incompatible
and
Given this state of affairs, important open question in any given domain is how to achieve Al performance that is both high-level and compatible with agents of lower skill. How might we accomplish this, and how would we know when we' ve succeeded?

---

Published as a conference paper at ICLR 2024
In this work, we propose training paradigm to create AI agents that combine strong performance with skill-oompatibility we use our paradigm to train several skill-compatible agents, and we illustrate their effectiveness two novel chess tasks. Our paradigm is based on the following idea: skill-coppatblle agents should still achieve a very high level of performance, but in such way that if they are interrupted any point. in time and replaced with much weaker agent, the weaker agent should be able to take over from the current state and still perform well. We enforce high level of performance by testing in an adversarial environment where the opponent might have superhuman abilitios, and we encourage robustness against interruption and replacement by a weaker agent in chess (skill- Shklllcoopattbilltty in two different ways: independently al random after each move; and by decomposing individual moves into a selection of piece lype and subsequent selection of a move using that piece type, following the popular chess variant known as hannd and brain."
This interruption framework thus provides computationally powerful agents with an objective function that balances two distinct goals: (i) it dissuades them from playing incompatible "engine moves," since such moves may strand a weaker agent in a state where it can't find good next action even when one exists, but (ii) it still promotes high performance since the goal is to win games despite interruptions from weaker counterparts. Our paradigm thus suggests strong and measurable notion of the interpretability of powerful agent's 's actions: the actions are interpretable (and skill-compatible in our sense if and only i weaker agent can find an effective way to follow up on them. This grounding in complex settings such as our motivating domain of chess which contains actions where there might be no succinct "explanation" " (we know this both in practice, and also theoretically from the fact that general game-trec evaluation is believed to be outside the complexity class NP.)
particularly useful
2 BACKGROUND AND RELATED WORK
Chess and AI. Chess has a long history in Al research Shannon 1950 with milestones including Deep Blue Campbell 1999)), followed by superhuman performance on commodity hardware, and more recently AlphaZero and its follow-ups Silver et al. (2016) Schrittwieser
et al. (2020). More recent work on the relation of algorit hnically--eneeattee chess moves to
human behavior play an important role in our work McGrath et al. (2022) McIlroy- Young
et al. (2020) Anderson et al. (201)); Maharaj et al. (2022).
Human-Al AI collaboration. Recent work has studied human-AI collaboration in
multi-agent scenario, where an AI agent and a weaker human agent work alongside each other to complete a lask Carroll el al. (2019); (Strouse el al. (2021); Yang el al. (2022). One distinction from our work is their notion of compatibility, where the tocus has been on agents working simultaneously on related tasks; in contrast, a central feature of our framework is that the compatibility is inter-temporel, with the design goal that a less-skilled agent should be able take over at any point from the partially completed state of the AT agent's 's progress.
Opponent modeling. Agents that interact with humans have made great strides in per- formance by modeling other human actors Bard el al. (2019), whether by modeling opponents as an optimal player (Perolat el. al.) 2022 Gray el al.) 2020 Brown & Sandholm 2018), or
by building agents that communicate and col collaborate with other human players in multiplayer
games (Vinyals et al.) 2019 Yu et al. 2021
(FAIR); et al. 2022). Several prior works explore from a strict adversarial, non-collaborative perspective the use of MCTS to exploit suboptimal
play safely, in strong agents (Wang et al. 2022: Ganzfried & Sandholm 2015
2.1 CHESS ENGINES
We select chess agents and Al agents designed to emulate lower-skilled human players, the complexity of the decision-making task, and the need for more understandable game AI agents that others can interact with. We list the existing engines that we make use of in our work.
model system due to the ready availability of both superhuman Al
LEELA. LEELA Lyashuk & et al 2022) is an open source version of AlphaZero et al. (2018). a deep RL agent that consists of neural network that evaluates boards (value) and suggests moves (policy) (iiverr et al. (2016), both of which are used to guide Monte Carlo Tree Search
2

---

Published as conference paper at ICLR 2024
Team
Team 2
Team
Team 2
81
So
S1 Sa J2
S
2
S
S
x5
AAAWWRA
NG
X
e5
X
a5
de
Awc
J1
J2
J1
J2
Figure 1: Stochastic Tag Team Framework
Figure 2: Hand and Brain Framework
(MCTS) algorithmto select the next action Jacob et al. (002);; Grill et al, (2200) The network is trained using repeated iterations of self- play followed by poock-roppggttio. We use small version LEELA as the superhuman engine in our tests and as our bascline for comparison.
MAIA. The MAIA engines Mcllroy- Young et ol. 2020 2022) are set of human- -ilean-like chess engines that capture human style at targeted skill levels. They are trained as a classification task on tens of millions of human games
specified skill level, and as such MALA does not use MCTS during play. Most of our work uses the weakest version, MAIA 1100, which was trained on games from 1100-rated players those roughly in the 20th percentile of skill on the open-source online chess platform lichess.org MAIA serves as the instantiation of sub-optimal, lower-skilled agents.
METHODOLOGY
3.1
CHESS FRAMEWORKS
Our goal to develop chess agents that can behave in inherently skill-compatible ways, much like skilled coaches tailor their actions to suit their students. In this work, we approach this task via two proxy frameworks in which our engine "coaches", or seniors, interact collaboratively with weaker "students", or jumiors. There are two natural ways in which this collaboration could take place within a sequential game setup: the seniors and juniors could alternate seqquee
between who takes the current action, the semiors and juniors could collaboratively construct each action. Our two frameworks follow these two directions to enable cooperation within chess game, suiting our experimental purpose of designing and evaluating skill-compatible agents. The former can be likened to self-driving cars, where AT needs to be ready for a hand-off to humans a moment's 's notice, whereas the latter resembles automatic stock picking strategies where potential candidates are filtered by Al for humans to make final picks. Note that both frameworks incorporate a strong and weak engine jointly in control of a single color, and incorporate an clement of stochasticity to proclude perfect prediction strategics by agents that skirt the objective of achieving compatibility. We will generally refer to our agents as playing the Focal roles against team of opponent agents playing the Alter roles.
Stochastic Tag Team (STT). The STT setup consists of two teams, each in control of color on the chessboard. A team consists of two agents, the junior teammate, which is generally MAIA, and the senior teammate, which is generally a stronger engine (e.g. LEELA, or the engines design). Prior to each move, Nature flips a fair coin to determine whether the junior or scnior makes the move, with no consultation with any other party allowed. This setting introduces cooperative aspect to chess, since senior will need to be prepared for the possibility that a weaker junior might be making the next move. At the same time, the senior will need to play at a high level of chess, since the opponent senior will also be playing the
at a high level, and it will simultaneously need to attempt to exploit. the weaknesses opponent junior. The STT framework thus allows us to explore scenarios where teammates and opponents can be both high-skilled and low-skilled, and the high-skilled Al is required to both perform at high level and account for low-skill involvement in both teams. Soe
Figure We use the following tuple to denote game played under STT: {1 Ja where S is the senior engine on the white team, Jj the junior engine on the white team, S2 the senior engine the black team, and Jz the junior engine on the black team. Note that while the first.
(9) Su
3

---

Published as a conference paper at ICLR 2024
team is technically white and the second team black in this notation, we abuse the notation to indicate multiple games played between these two teams, alternating between black and white.
Hand and Brain (HB). This cooperative setup, which has witnessed massive increase in in- terest by grandmasters (GMs) and amateurs alike in recent years, also consists of two teams, each in control of color the chessboard. A team consists of two agents, the brain agent (always the stronger agent in our case), which selects the piece type to be moved (c.g., knight 4), and the hand (MAIA agent in our case), which then selects the specific piece and move to make given the brain's 's selection (e.g, move the knight on g1 to f3). See Figure 2 GM Hikaru Nakamura stated that as the brain playing with a weaker hand. he often picks sub-optimal moves he finds more suitable for his hand partner Nakamura (2021), which is in the spirit of our work. In contrast to the previous framework, this framework exemplifies scenarios where the stronger agent "nudges" the weaker agent, narrowing their action space, which then makes the action. We will use the fol- lowing tuple to denote a game played under HB:
{
By B2 J. where H₃ and B₁ are the hand and
Hr H2
brain agents on the white leam, and H2 and B2 are the hand and brain agents on the black team. In our work, the brain will always be the stronger agent, and the hand the weaker engine that chooses a move (stochastically from its distribution, induce some randomness and ensure the brains aren't 't able to perfectly predict their decision) conditioned on the brain's 's piece-type choice.
3.2 METHODOLOGIES TO CREATE SKILL-COMPATIBLE SENIORS
We contribute three methodologies to create agents that outperform superhuman chess engines (LEELA) in these two skill-compatibility frameworks. We emphasize that we are not aiming to improve upon state-of-tho-art chess engines. Instead, we are interested in designing skill-compatible chess Al that can productively interact with weaker agents.
Tree agent. Our first agent is the MAIA engine augmented with MCTS. By exploring future game states based solely on MAIA'S 's policies and values, the TREE agent inherently takes its junior's 's propensities into account when deciding what move to make, Notably, this agent only requires a MAIA model and does not rely and thus implemented identically for both frameworks. In H HB, the TREE agent's 's output is filtered to only convey the piece type, mentioned above.
superhuman agent. It is also framework- aggostii
Expector agent. We introduce a type of gold standard agent that conforms to the exact setting of the frameworks. The expector agent has access to models of juniors/ /hanos,,//d,,,, and maximizes its expected win probability w over a short time horizon given the identities of the seniors and juniors. For STT, it simulates all possible bitstrings 2 plies into the future, selecting the move m = argmax [ (00,01,10,11) (m,(m,)))): and for HB it selects the piece
m that maximizes its expected win probability over MAIA'S distribution of moves conditioned on that piece (Dp): p= argmax (EmennxxxEE,e,,eeee
In the former case it requires a model of the
other three agents in the game to perform the simulation, and in the latter case it also needs model of its own hand to obtain the distribution Dp. In both cases, it requires access to a strong agent to compute the win probabilities that the expectation is maximizing. Although it requires no training, playing moves expensive due to calls to multiple chess engines and evaluations of the current board state. The version of the expector designed for STT will be denoted as EXP,, and the one designed for HB will be denoted as EXP.-
Attuned agent. The attuned agent is a self-play RL agent that directly learns from playing in the two frameworks. In contrast to learning from self-play in conventional chess, as LEELA does, the attuned agents are created by generating games from self- play of LEELA and MAIA teams in both frameworks. With small training set, this method is essentially fine-tuning procedure for LEELA that takes into account MAIA'S 's interventions, rather than training a chess engine from scratch. The Supplement includes full training details. This methodology is the most practical of the three in its ability to be modified and generalize, and lies somewhere between the rigidity of TREE and the specificity of EXP. However, it is the only agent of the three we introduced that requires training. The version of the attuned agent trained on STT will be denoted as ATT,, and the one trained on HB will be denoted as ATT,, In H B, similar to the tree agent, we only take the piece of the outputted move.
4

---

Published as conference paper at ICLR 2024
Table 1: Game Results for all agents STT and HB
STT
HB
Game Setup
TREE
EXP,
ATT,
MAIA
TREE
EXPn
ATTH
MAIA
local MALA focal
LEBLA
56.5
66.5
55.0"
7.5"
60.0"
56.5
55.0
27.5"
MAJA
LEELA 0.5 **
14.0*
12.5
0.0
0.5 SM
N/A
4.5
0.0
<* =0.5, S +1.0, appendix for full ranges. Note that doean't output can't play leela directly.
EXPERIMENTS
4.1
AGENT STRENGTH IN EACH FRAMEWORK
Our foremost goal is to quantify the objective performance of each agent on each framework. Are they better at playing with weaker partners than sate--ofttheart chess Al that is, are they skill-compatibl?? Here, we use MAIAl as the junior and hand agent for all analyses, and the three methodologics will accordingly make use of MAIA 1 100 as a basc model to guide the scarch for TREE, as a junior to guide the look-ahead for EXP, and in the creation of the training datasel for ATT. Our evaluation metric is the win-share over games, which is defined as (W+D/2) where W is the number of wins, the number of draws, and 1000 S n 10000 the number of games, from the perspective of the focal team. Equally-matched agents will each score a win-share of 50%, and scoring a above 50% indicates a victory. We compute the standard error by treating the experiments as random samples from trinomial distribution, detailed in the Supplement. Tables shows the results for all three agents on both frameworks STT and H B respectively.
As shown in the first row, which documents the performance of our focal agents in matchups against the state-of-the-are chess Al LEELA in our frameworks, all of our methodologies achieve winning score (>50%) in both frameworks. In STT, the EXP, agents that perform short look-ahead (66%) tend to dominate, and in HB it is the TREE agent that scores the highost (60%). Our main result is that all three methodologies produce agents that play well and more intelligibly to weaker partners than state-of-the-art chess AI.
In order to validate that our focal agents are achieving their gains by explicitly accounting for the presence of the weaker partners, we eliminate the two most pressing potential confounders. The first hypothesis we rule out is that our focal agents are simply stronger than LEELA, which the second row in Tables 1 Tshows is evidently not the case. In fact, they are significantly weaker, losing most of their games to LEELA in head-to-head regular- chess matchups. TREE performs particularly poorly (<1%), likely because it is unrelated to LEELA'S 's weights, unlike EXP and ATT. It is striking that TREE outperforms ATT on the STT and HB despite being significantly weaker, indicating that. it must compensale with larger adaplation lo and synergy with its lower-skilled partner. The second hypothesis we rule out is whether LEELA is a particularly bad senior/ brain due to its strength, and our focals are better at adapting to their hands/ brains simply because they are weaker and more similar to them. Replacing our focals with MAIA as a senior /hand (the most similar, weakest possible senior /orioo//rrinnnn refutes this idea (see last column). We note that, while ATT and EXP are weaker than LEELA, they still achieve non-trivial scores (> 10%) in regular chess versus LEBLA, indicating they are still strong chess agents.
We have established our central result: Our focal agents are objectively weaker than standard state-of-the-art AI, but their compatibility with MAIA is more than sufficient to defeat LEELA in both collaborative frameworks. We now investigate the mechanisms of cillaiippaiibllity.
4.2 MECHANISMS OF ACHIEVING SKILL-COMPATIBILITY IN STT
How do our agents achieve saill-compatiiilitty In this section, we answer this question by analyzing agent behavior at the individual move level. We define the win probability loss of move, which measures the degree to which any given move is sub-optimal. Notice that any chess move is either optimal, meaning it preserves the win probability of the previous position (as evaluated by a strong engine such as LEELA), or is sub-optimal, meaning it degrades the agent's 's win probability by certain amount. We will define the win probability loss of a move, or simply
5

---

Published as conference paper at ICLR 2024
Table 2: Average losses for agents in STT.
Agent
G2
G.
Ga
Agent
G.
G.
Go
LEELA
1.15
1.16
1.17"
MAIAL
4.46
4.21
4.19" g*
focal
1.91
1.37
1.43"
MAIAp
3.57
3.37
3.77
Dg (LEELA, local,*)
-0.76"
-0.21" "
-0.26"
Doy (MAIAL, MAIAD,*)
0.89*
0.84
0.42
AG (teamr, teamp,*)
0.13"
0.63
0.16"
=0000, 40.04, appendix for full ranges
Table 3: Different effects of seniors juniors in STT. Effects that are stronger (p<0.05) than that the opposing senior are in bold.
Ge
Ge
G.
Agent
LEELA
TREE
LEELA
EXP
LEELA
ATT
Tricking:
-0.03* **
0.54 **
-0.27
1.38"
-0.01
0.25 **
Helping (Interceding Junior):
0.26
0.36
0.30
0.61
0.34
0.21
Helping (Interceding Scnior): 0.15
* 0.46*
0.12
1.88* **
0.20 **
0.25 **
Indirect:
0.56
-0.18
0.37
S =0.07, s $0.111, see appondix for full error ranges
the loss, as the difference in the win probability of the board following the move to the win probability of the board preceding It ranges from 0 100. Given an agent and a condition C' on moves played by A from set of games Gj, we define a mean value LG, (A,C) as follows: where S = {m E Gy satisfies C and m is played by A}. To
La, (A,C)= TST L compare losses between agents A₁ and A2, we define AG, A2,, LG, (A1,C) (A2,C), where setting C'= mcans taking all moves.
Leela MALA. where focal refers to of TREE, EXPr, or ATT,; LEELA denotes LEBLA playing as the alter; MAIA, refers to the Focal team's 's junior MAIA agent, and MATA, refers to the Alter team's junior MAIA agent. Note that MAIA, and MAIA, are both MAIA1100 agents. We will refer this tuple as Gy for simplicity, with f being the starting letter of the corresponding focal. All agents and games in this section in STT, so we omit the specifying subscript.
For this section, we will referring to agents as they appear in the tuple
focal
MALA,
4.2.1 GENERAL EFFECTS ON JUNIOR PERFORMANCE
Table2shows the average loss by the agents involved at every move from the games played in STT. For all focal agents, we have LG, (focal,*) > LG, (LEELA,*)) yet LG, (MAIA,,*) LG, (MAIAL.*). This more granular, move-level statement of our central result: our focal engines sacrifice some optimality for the ability to influence and be skill-compatible with the sub-optimal agents they are interacting with.
Note also that LG, (TREE,)) > LG, (ATT,*), yet Ac, MMAIA,,,AIA,,** Ac. (MAIA,, x)), implying TREE's results are more dependent on influencing the juniors present in the game. EXP combines low absolute Ac. (focal,LEEIA, with higher Ag, (MIAA,, to get the best overall team loss difference among the three agents, which explains its higher score in the main evaluations.
To compare how these findings vary with position strength, we plot Ac, (MAIA,, where i stipulates the moves originate from boards where the probability of winning is equal to i (Figure 4). For all focal engines, the gap is increasing in i: the closer the board to winning, the more MAIA, outperforms MAIA,, Winning boards thus more critical, where the junior's moves have a chance to throw the game, compared to losing situations where the junior cannot bring about large positive changes to the evaluation.

---

Published as conference paper at ICLR 2024
Ratio focal locia
esp
-
- att
exp
tree
Dr. (maiaumaia,
[0. 1]
[1. 5]
[5.
[10. 50)
[50. 100)
p0.
20
200 10j
[40. 60]
(60, 80)
[60. 100)
Range of loss magnitude
Board Win Probability (+)
Figure 3: Ratio of probability of excess loss induction over diffcrent loss magnitudes function of different board win probabilities
Figure 4: A (MAIA) MAIA,) as
4.2.2 TRICKING, HELPING, AND INDIRECT EFFECTS IN STT
Having established that our focal agents induce gap between MAIAp and MATA, performance, we investigate three possible mechanisms by which they can achieve this: an immediate "tricking" effect, which we define as an induction of loss in MAIA, over 1-ply horizon (the alter junior's 's next move), an immediate "helping" effect, which we define as reduction of loss in MAIA, over a 2-ply horizon (the focal junior's next move), and finally an indirect effeet that cannot be measured over these short horizons (for example, early choices that impact how the game unfolds many moves into the future).
The chief difficulty of comparing the focals' effect on the juniors to that of leela on the juniors in the games played is that direct comparison would be potentially confounded by the different board distributions the opposing teams faced. We deal with the different distributions using two distinct methods detailed in the appendix. We present in table 3 the main results derived from the first method. Bold values indicates that the senior exhibits the effect in question. All agents display some tricking effect, with the magnitude being largest for EXP (1.38) and smallest for ATT (0.25). We do not observe a helping effect for ATT, but do so for the other focals. In particular, the helping effect for EXP is dramatically higher the when opponent leela intercedes (1.88 vs 0.61), and we believe this is due to the agent's 's preference for tricking the opponent junior when intercedes instead of helping its own junior down the horizon. Finally, EXP has no beneficial indirect effect the juniors (meaning longer than its optimization horizon of 2 plies), whereas the TREE and ATT have a measurable indirect effect even when they do not precede the juniors. Additionally, extra analysis in the supplement suggests that this indirect effect is more important than the tricking effect for these engines. These results demonstrate that there are multiple ways to influence the juniors, and the rongettt agent, EXP, distinguishes itsell by a complete lack of a beneficial indirect elfect in favour of strong immediate effects, both helping and tricking.
4.3 MECHANISMS OF ACHIEVING SKILI-COMPATIBILITY IN HB
We now turn to the HB framework. Note that Gy now refers to the tuple focal LeblA MAIA MAIAL
and agents without subscript will be referring those created for the HB framework. We examine the effect of our focal agents (playing as brain) on the MAIA agents (playing as hand), and focus on two mechanisms: intra-team effects, where the brain picks a piece that causes MAIA to pick a better /worse move than it would have without interference, and inter-team effects, where the teams affect each other.
We first examine intra-team elfects by computing the savings, the difference between the loss of the team's 's actual move played to the loss of the move MATA would have selected without interference from the brain agent. Interestingly, Table 4 shows that EXP is the only brain exhibiting such an effect (0.3%), which is natural as it has been explicitly instructed to minimize the expected loss of its hand. TREE, which is the best performing agent in the framework, actually has nogative savings (-0.2%), mcaning its influence is causing its own hand to play worse moves. To analyze results more closely, inspect the interaction between hands and brains.
There are four key hand-brain interactions: agreement (same move chosen), blindsiding (different moves but same piece-type, allowing the hand's move), correction (hand resamples to
7

---

Published as conference paper at ICLR 2024
Table 4: Comparison of team loss hypothetical MAIA loss without brain HB. Better in bold.
G+
Ge
G
MAIA as focal
LEELA
TREE
LEBLA
EXP
LEELA
ATT
LEBLA MALA.
MALA
MAIA-
MAIA,
MAIA:
True loss
3. 3.76"*
3.55
3. 3.44
3.33
3.61
*
3.50
3.73"
4.25
MAIA loss
3.75
3.29
3.40
3.62
3.59
3.43
3.39
3.60
Savings -0.01
-0.22
-0.04"
0.3
-0.02
-0.07
-0.32
-0.65"
==0.03
S 500007 see appondix for full error ranges
match the brain's 's different picce-type move), and disagreement (hand selects different move after forced resampling). Detailed results on the proportions of cach are in the supplement.
For all brains, the correction case yields some savings (4% 6%), and the disagreement yields a drop the performance of the hands (1% 2%). Furthermore, the savings in the correction case are lower for TREE and ATT than they are for LEELA. TREE's effect does not appear to come from savings, but rather, we see that when TREE corrects its hand, it induces a high loss in the opponents" next move compared to LEELA's correction (4.8% vs 3.8%), showing a tricking action exerted by the TREE on the opponent team.
As TREE and ATT both increase their agreement with MAIA, test the strategy of maximizing agreement and eliminating disagreement (as well as correction and blindsiding) entirely by letting MAIA play its own against LEELA the alter senior and MAIA as the alter junior, and it loses to LEBLA with a 40% 1.5 score. This confirms that disagreement-indde eeneennnc-dddeded loss more than compensated for by the benefit of correction of a strong brain.
4.4 EXPLORING IMPERFECT PARTNER MODELLING
All our experiments thus far have explored the results senior agents designed to play with MAIA1100 (meaning generic maia trained many 1100 rated player games), tested MAIA1100. Note that the term "designed for junior X" means, for ATT: that it trains with junior X; for TREE: that it runs MCTS using X to guide the search tree; and for EXP: that it uses junior to simulate the next couple of moves. Now we investigate cross- coaascoomptibiilityy meaning whether semiors designed for these generic MAIA juniors are compat ibbee with juniors that they not explicitly designed for. We investigate three possible instantiations of cross-compatiiill cross-skill compatibility, specific player compatibility, and cross-style compatibility, and observe some success with the first two and an inability to generalize to radically different juniors.
4.4.1 CROSS-SKILL COMPATIBILITY
Here, MAIA1900, the strongest available version of MAIA trained on data of 1900 rated players, as an alternative junior/hane hand. We analogously create TREE, EXP, and ATT agents designed be compatible with MAIA1900. In this section, focalx denotes a focal designed for compatibility with MX0O Testing focal9 agents with MAIA1900 as the junior in 8 shows that they are able to beat LEELA in the frameworks, with the exception of TREED producing no gains in STT. We test pross-compatibllity of the focall and focal9 agents by partnering them with each other's juniors. We emphasize that focal9 agents are not exposed to MAIA prior to testing, and vice versa. As shown in Table g (appendix), focal9 agents are always compatible with MAIA 100 as a junior, irrespective of focal and framework, while the same is not always true of focall agents with MAIA1900. We hypothesize the assymetry is due to focall agents being more aggressive in playing suboptimally in order to exploit the junior, which backfires when MAIA 1900 does not fall for exploits, whereas focal9 agents do not get explicitly punished if they over-conservativel fail to setup a trap for MAIA 1100.
4.4.2 SPECIFIC PLAYER COMPATIBILITY IN STT
We turn to validating whether the focal agents created here using the generic MAIA models are compatible with individualized engines that are fine-tuned to mimic particular human players Mellros-eooooooooo el al. (2020). de so with the objective of exploring the applicability of
8

---

Published as a conference paper at ICLR 2024
our method in the case where complete model of the opponent /pantntt junior is not available (as is the case in most situations). Our initial experiment consists of individualized models of 2 players (rated 1650 and 1950), for which designed specialized seniors, and compared the performance of these specialized seniors to that of seniors designed for gemeric MAIA 1 1900. In these games, generally, generic focal agents win against. leela, with proportionally smaller margin than when seniors designed for generic/specific juniors are matched with the junior it was designed for. We now turn to more extensive player generalization testing.
Accordingly, we selected a random subset of 23 models, each trained on a particular human Lichess players rated between 1400 (43rd percentile skill) and 1900 (83rd percentile skill) as the junior for this experiment. We then use an EXP that internally uses the generic Maia agent with the nearest rating to the player in question, and have play in STT with that player's model. The median score of the different EXP agents over in this scenario with the 23 different juniors is 53.3% (+1%). While this is below EXP'S performance of 66.5% in table [ it nonetheless demonstrates that generic approximation is sufficient to encapsulate skill-compatibility with individuals. In fact, out of 23 different players tested, EXP was shown to be winning (p=50.05) in 18 of the cases after 3000 games played, with the experiments the remaining players not showing statistical significance to that level.
4.4.3 CROSS-STYLE COMPATIRILITY IN STT
We now investigate using completely different junior based on a non-neural architecture. To do that, we calibrate a low-depth version of stockfish (we call it SFW), to play at the skill-level of MAIA1100 and comparison in the appendix demonstrates that it is very different from any agent used so far. Note that there is no TREE agent. It is seen that EXP is able to obtain results against SFW, but ATT is unable to. Generalization is nonexistent, with engines trained with MAIA 1 100 losing when testing with SFW junior and vice versa. We attribute this to the lack of similarity between SFW and MAIA 100. This does indicate, that our agents' compatibility is not function of merely skill, but also style alignment.
5 LIMITATIONS AND DISCUSSION
Our work proposes a methodology for creating powerful agents that are compatblee with weaker partners. Our key finding is that, in complex decision making setup as chess, skill-compatiilitty is a qualitatively and quantitatively measurable attribute of agents distinct from raw ability on the underlying task. Our designed frameworks show that in situations where strong engines are required to collaborate with weak engines, playing strength alone is insufficient to achieve the best results; it is necessary to achieve compatibility, even at the cost of pure strength. Finally, our three methodologies, cach distinct in design and method of operation, demonstrate that there are multiple viable techniques to create agents that achieve this form of compatibility, with different agents using different strategies in-game. Indeed, some strategies center on helping the weak engine make better moves should it assume control, while others explicitly disrupt the compatibility of the adversary forcing weak opponent agents into errors, which even strong partner like LEELA is unable to mitigate.
Our work therefore is an empirical proof-of-concepy for skill-compatibility in chess, and provides a roadmap for the creation of human-com compatible agents in this domain and beyond. While our paper does not speak directly to the prospect of skill-comatibbiiit in other domains, we believe that number of the techniques here are relatively general in nature, with clear analogues to other seltings. For example, while the tree agent is very chess specific, and EXP is difficult to run in continuous environments, a number of ideas underlying these methodologies - EXX's short-term prediction, ATT'S tandem training with the targeted skill - can be easily modified to fit different tasks, and offer potential for cilll-oopatibilityy in these tasks.
The training frameworks we propose use human- like MAIA agents as weak partners, and a natural next direction for future work is the design of experiments to test these methods with human chess players. Our scope likewise did not include instantiating the stronger partner beyond using LEELA, which offers opportunity to test robustness to modifications of the environment.
9

---

Published as conference paper at ICLR 2024
REFERENCES
Anderson, Ashton, Kleinberg, Jon, & Mullainathan, Sendhil. 2017. Assessing human error against benchmark of perfection. ACM Transactions on Knowledge Discovery from Data (TKDD), 11(4), 25.
Bard, Nolan, Foerster, Jakob N., Chandar, A. P. Sarath, Burch, Neil, Lanctot, Mare, Song, H. Francis, Parisotto, Emilio, Dumoulin, Vincent, Moitra, Subhodeep, Hughes, Edward, Dunning, lain, Mourad, Shibl, Larochelle, H., Bellemare, Mare G., & Bowling, Michael H. 2019. The Hanabi Challenge: A New Frontier for Al Research. Artif. Intell., 280, 103216.
Brown, Noam, & Sandholm, Tuomas. 2018. Superhuman Al for heads-up up no-limit poker: Libratus beats top professionals. Science, 359(4)74), 418 424.
Campbell, Murray. 1999. Knowledge discovery in deep blue. Communications of the ACM, 42(11), 65 67.
Carroll, Micah, Shah, Rohin, Ho, Mark K, Griffiths, Tom, Seshia, Sanjit, Abbeel, Pieter, & Dragan, Anca. 2019. On the utility of learning about humans for human-ai coordination. Advances in nearal information processing systems, 32.
(FAIR)); Meta Fundamental AT Research Diplomacy Team, Bakhtin, Anton, Brown, Noam, Dinan, Emily, Farina, Gabriele, Flaherty, Colin, Fried, Daniel, Goff, Andrew, Gray, Jonathan, Hu, Hengyuan, et al. 2022. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378(6624), 1067-1074.
Ganzfried, Sam, & Sandholm, Tuomas. 2015. Safe opponent exploitation. ACM Transactions on Economics and Computation (TEAC), 3(2), 1-28.
Gray, Jonathan, Lerer, Adam, Bakhtin, Anton, & Brown, Noam. 2020. Human-level performance in no-press diplomacy via equilibrium search. arXiv preprint Xiv:2000 0..0020...9.....
Grill, Jean-Bastien Bastien, Altché, Florent, Tang, Yunhao, Hubert, Thomas, Valko, Michal, Antonoglou, Ioannis, & Munos, Rémi. 2020. Monte-Cart tree search regularized policy optimization. Pages 9769-3776 of: International Conference on Machine Learning. PMLR.
Jacob, Athul Paul, Wu, David J, Farina, Gabriele, Lerer, Adam, Hu, Hengyuan, Bakhtin, Anton, Andreas, Jacob, & Brown, Noam. 2022. Modeling stng and human-like gameplay with -T-regularized search. Pages 9728-9728-00 of: International Conference on Machine Learning. PMLR.
Lyashuk, Alexander, & et al. 2023. leela. https:/ /lzzero.rzr// Accessed: 2023-02-11.
Maharaj, Shiva, Polson, Nick, & Turk, Alex. 2022. Chess Al: competing paradigms for machine intelligence. Entropy. 24(4), 550.
McGrath, Thomas, Kapishnikov, Andrei, Tomašev, Nenad, Pearce, Adam, Wattenberg, Martin, Hassabis, Demis, Kim, Been, Paquet, Ulrich, & Kramnik, Vladimir. 2022. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(, e2206625119.
Mcllroy- Young. Reid, Sen, Siddhartha, Kleinberg, Jon, & Anderson, Ashton. 2020. Aligning superhuman ai with human behavior: Chess as model system. Pages 1677-1687 of: Proceedings of the 26th A CMM SIGKDD International Conference on Knowledge Discovery & Data Mining.
McIlroy- Kellroy-Young, Reid, Wang, Russell, Sen, Siddhartha, Kleinberg, Jon, & Auderson, Ashton. 2022. Learning Personalized Models of Human Behavior in Chess. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Dato Mining.
Nakamura, Hikaru. 2021. In Hand and Brain chess, is the stronger player generally preferred to wwtss.sstsceeeeeeeeee waapsssstaccecchnge.coom /3973.. Accessed: 2023-04-07, Original video source longer available.
be the hand or the brain? https:/
10

---

Published as conference paper at ICLR 2024
Perolat, Julien, De Vylder, Bart, Hennes, Daniel, Tarassov, Eugene, Strub, Florian, de Boer, Vincent, Muller, Paul, Connor, Jerome T, Burch, Neil, Anthony, Thomas, et al. 2022. Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, 378(6623), 990-996.
Schrittwieser, Julian, Anlonoglou, loannis, Hubert, Thomas, Simonyan, Karen, Sifre, Laurent, Schmitt, Simon, Guez, Arthur, Lockhart, Edward, Hassabis, Demis, Graepel, Thore, al. 2020. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839), 604 609.
Shannon, Claude E. 1950. XXII. Programming a compuler for playing chess. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 41(314), 256-275.
Silver, David, Huang, Aja, Maddison, Chris J, Guez, Arthur, Sifre, Laurent, Van Den Dricssche, George, Schrittwieser, Julian, Antonoglou, Ioannis, Panneershelvam,, Veda, Lanctot, Marc, el al. 2016. Mastering the game of Go with deep neural networks and tree search. nature, 529 (7587), 484-489.
Silver, David, Hubert, Thomas, Schrittwieser, Julian, Antonoglou, Ioannis, Lai, Matthew, Guez, Arthur, Lanctot, Mare, Sifre, Laurent, Kumaran, Dharshan, Graepel, Thore, et al. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140-1144.
Strouse, DJ, McKee, Kevin, Botvinick, Matt, Hughes, Edward, & Everctt, Richard. 2021. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34,
Vinyals, Oriol, Babuschkin, Igor, Czarnecki, Wojeiech M., Mathicu, Michaël, Dudzik, Andrew, Chung, Junyoung, Choi, David H., Powell, Richard, Ewalds, Timo, Georgiev, Petko, Oh, Junhyuk, Horgan, Dan, Kroiss, Manuel, Danihelka, Ivo, Huang, Aja, Sifre, L.., Cai, Trevor, Agapiou, John P., Jaderberg, Max, Vezhnevels, Alexander Sasha, Leblond, Rémú, Pohlen, Tobias, Dalibard, Valentin, Budden, David, Sulsky, Yury, Molloy, James, Paine, Tom Le, Gulcehre, Caglar, Wang, Ziyun, Pfaff, Tobias, Wu, Yuhuai, Ring, Roman, Yogatama, Dani, Wünsch, Dario, McKinney, Katrina, Smith, Oliver, Schaul, Tom, Lillicrap, Timothy P., Kavukcuoglu, Koray, Hassabis, Demis, Apps, Chris, & Silver, David. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 1-5.
Wang, Tony Tong, Gleave, Adam, Belrose, Nora, Tseng, Tom, Dennis, Michael D, Duan, Yawen, Pogrebniak, Viktor, Miller, Joseph, Levine, Sergey, & Russell, Stuart. 2022. Adversarial poli- cics beat professional-level go ais. In: Deep Reinforeement Learning Workshop NeurIPS 2022.
Yang, Mesul, Carroll, Micah, & Dragan, Anca. 2022. Optimal Behavior Prior: Data- Efficient Human Models for Improved Human- Al Collaboration. arXiv preprint or Xrv:v:2211.0 0.66002.
Yu, Chao, Velu, Akash, Vinitsky, Eugene, Wang, Yu, Bayen, Alexandre M., & Wu, Yi. 2021. The Surprising Evfectiveness of MAPPO in Cooperative, Multi-Agent Games. ArXio, abs abs/2203.01955.
11

---

Published as conference paper at ICLR 2024
6
SUPPLEMENT
6.1
CODE RELEASE
Our code is released at w http:////S..ata/e1e--ooaaaabaaa- several of our trained models.
We also include
6.2
METHODDLLOGY DETAILS
6.2.1 TESTING AND ERROR RANGE COMPUTATION
Forat LERII To lest a particular local STT, werun games of the form
For any bitstring 8, we play 2 games, with the focal and LEELA teams switching between black and white. This is done because some bitstrings are biased in favour of particular color, and we therefore eliminate this bias by having each team playing both sides of the bitstring. Additionally, unfair bitstrings are undesirable, as it is likely that the team less relevant than the color to achieve victory. Hence, they represent source of unbiased noise to the result, by adding 1 win for each team. It. is difficult to quantify fairness of a bitstring, (it is not sufficient to ensure equal number of senior and junior moves-the the order matters). Therefore, for consistency, all experiments test individual focal agents are sampling from the same set of bitstrings during testing, climinating the possibility that some agents sample more unfair (and hence, cossrcoonti)buting bitstrings during their testing.
MALA
MAIA
No analogous measures are applicable for HB, as stochasticity is internal to the teams rather than being a characteristic of game.
For both frameworks, testing consists of between 1000 and 10000 games, depending on our targeted significance. Here we detail computation of the standard error se for the win-share displayed in the main section of the paper.
we play n games, with W wins and losses, we can write the empirical win-share as
0.5+ 1 w-1
2
where v and are the empirical unbiased estimators of the true probabilities w, 4, computed as W/n and L/n respectively. Since w, L, and d (probability of draw) form trinomial distribution, we have
V(w i)= (w1-w) 1(1-1) 2wl
n
n
which simplifies to
w w+1
(w 1)2
n
Plugging in the variance to get the of the win-share, we have
w+1 (w 1)2
se sc=0.5
n
This is maximized with w=1=0.5, and we have
0.5 se < Vn
Our data is presented in percentages, so, this implics setting 10000 guarantec se 0.5%% and n= 1000 guarantecs se 1.5%. In practice we are often able to get lower errors with lower because w.
All other quantities in our paper are sample means, with large sample sizes allowing the central limit. theorem to be used to obtain their standard errors.
12

---

Published as conference paper at ICLR 2024
6.2.2 LEELA
We use the a 128x10-660-2-5300 LEELA network, obtained from Vier with 1500 node search. Against stockfish 13 (60k nodes), a strong classical engine that uses alpha-beta search, this version of LEELA obtains a score of 59 + 3. Melon benchmarks stockfish 13 to human elo, so we deduce that our version of LEELA plays at around 3050 elo. While LEELA can be made significantly stronger if more nodes are used in scarch, limiting it to 1500 allows us to generate more games and run more tests, while still retaining superhuman capability. We use 1500 nodes for all seniors, for consistency. To compute the win probabilities of boards, needed to conduct most of our loss analysis, we use separate instantiation of LEELA with the same parameters. In practice though, the LEELA used for evaluation is stronger than the LEELA playing as senior, because evaluations occur multiple times per move for different statisties, which, due to caching, is equivalent to working with more nodes,
6.2.3 ATT
To create ATT, a dataset of 10000 games (80% train, 10% validate, and 10% test) is generated of the following game <LEELA) LEELA > for STT or I (LEELA LEBLA } for HB. Then, starting MAIA with LEELA'S weights, and using learning rate of 10-5; and 10000 iterations, we run propagaopan to update LEELA's policy and value neural network. We use the version of MAIA for which we are attempting to achieve compatibility in this training scheme.
MALA
As this is tuning task, and we are starting with LEELA weights, we perform some parameter tuning, with the objective not necessarily to find optimal parameters, but rather to find a set thereof that compromises between cost and robustness. The main time cost comes from generating the datasets of games, and training the agents.
To do so, we train models with learning rates from 10-1 to 10-6 on dataset sizes of 1000, 10000, and 100000 games, and a short training time of 1000 iterations. We lest these models with a small nunber of matches that is sulficient. to determine whether the training procedure produced viable engines (viable simply means plausible, not necessarily successful). See table 5) Some learning rates catastrophically fail and exhibit nonsensical learning curves, or produce agents that lose a large majority of their games.
From the above, we decide to use 10000 games our dataset size, as it appears that a range of learning rates are viable on it, and it is not as costly to generate as 100000. We also settle on learning rate of 10-5, as this rale is more robust in small datasets than 10-1 and requires less time to convergence than 10-6.
We note that there appears to be connection between the quality of the policy and value accu- racy curves and actual performance, meaning, models that overfit or fail to converge also perform poorly when testing on the frameworks. Accordingly, we observe that 100000 iterations produces more complete convergence curves for this particular learning rate and dataset size choice.
After having selected papee-paraeeters which we deem plausible, we run our first full training run 8 times to ensure that performance on the framework following training is repeatable, rather than a product of chance. The worst model of the has a win-rate of 53.5 #1, and the best model a win-rate of 56.0 +1.
Note that our goal to find valid, stable hyper-paramotors, which does not proclude the existence of better sets of syper-parameters. For all other models, (meaning different frame- works, or different MAIA jumiors), we use these parameters, and if convergence issues arrive, we modily parameters heuristically. This lype of modification was not actually needed for any models in the main paper, but was required for a few additional models which we detail here.
The rest of the hyper- parameters can be found in the configuration files in the linked code.
/twk//wii/w/eeeee bestibbeeteettso--lbee Phttps:/ http://wwwwlwlmmmrrroooo /tttt//e////// 202222221222 /03/000 htt.ttffiiinnl-----eaaalffeeemmm
nodes/
13

---

Published as a conference paper at ICLR 2024
6.2.4 EXP
EXP more expensive run than the standard neural netework engines as makes multiple calls to engines as subroutines to compute the move that maximizes expectation. Accordingly, it is unfeasible to do a full search over all logal moves, as that may consume up to hundreds of times as much compute. Likewise, these engines use shallow LEELA engines for board evaluation lo deter- mine their move selection. For STT, we compute the expectat ion over the top moves, with eval- uation conducted al. 300 nodes, whereas in HB, for each piece, we compute the expectation over the top 3 MAIA moves (meaning 18 moves checked total), with evaluation conducted at 50 nodes.
6.2.5 DETAILING METHOD 1 USED 4.2.3
While involved, the technique used here comes with the added benefit that no engines play additional moves outside what has already played within evaluation games, and therefore the comparison is more pertinent to the games themselves. To study these three mechanisms, we analyze special sequences the board. Since in STT, agent selection is done via coin flipping, games can be represented as a bitstring, 8 where Is indicate senior moves and Os indicate junior moves. We now detail how we compute the values present in table 3 By computing the loss on moves that are preceded by specific substrings, we can isolate different effects. Accordingly, we define LG, (A,s) where s condition that stipulates that the moves must be preceded by a (partial) bitstring in G5. For example, La, (MAIA,,1) means that we are computing the loss of MAIA, only when its move comns immediately after LEELA'S 's whereas (MAIA 0) means the moves come after MAIA,'S. 's.
As mentioned earlier, in order to compare the tricking effect of senior S₁ to senior S2, we not directly compare Lc, (J2.1) (measuring how S₁ tricks J2) La, (J1,1) (measuring how Sa tricks J1). as J1 and J2 are playing on different board distributions (the distributions of the board the two leamn are not the same, the simplest example being that the focal leam will have more win- ning boards). We perform an indirect comparison of each semior's effect to its own junior's 's effect, as they share board distribution, and all necessary moves are already present. We do the same for both seniors and then compare these quantities. Formally, we define Ic, (4.s)=Lo, (A,1 L ( (A,00s) where @ is concatenation, the and denote the comparison of the loss induced by the senior being present in that position to that by the junior, and s is a string to standardize the agents that play in between should we be measuring an effect across more than ply.
The immediate tricking effect of a senior S under this definition can thus be computed as Ic,(Jop ()) where & the empty string and Jopp is the opposing junior.
To measure the immediate helping effect of senior S on its partner junior par, note that the opposing team must play a move prior to per, and there are two possible situations depending on which opponent plays. There are thus two helping effects to be measured, IG, (Jpar.1) and Ia,(J (Jpar;0), denoting the opponent senior and junior being the interceding agents, respectively.
An in order to examine the indirect effect, sosee whether the focal agents affect the performance of their juniors over longer time horizons. We compute this as Ac, (MAIA,,,AIAp,,00) where two zeros indicate the move must not be preceded by any senior for 2 plies.
6.2.6 DETAILING METHOD 2 USED IN 4.2.3
We perform this analysis with LEELA and each focal agent on two separate board distributions: the set of boards that LEELA'S 's team and that the local's leam encountered in-game. Table b shows these results for different focal agents, EXP induces similar loss in MAIA regardless of the board distribution
4.399. This is in line with EXP's myopic optimization objective. The gulf between EXP and LEELA remains large regardless if we compare the distributions as seen in game (4.39> 3.25) or if we standarde the board distribution (4.39 > 3.65 and 4.47 3.25). In contrast, there is a notable degradation in the loss induction abilities of TREE and ATT when eliminating distributional effects (3.89 < 4.85 and 3.95 < 4.46). Consequently, the standardized comparisons for these focals to LEELA shows a much closer result than the unstandardized in- game observations. Interestingly, comparing along the minor diagonal shows that LEELA induces more loss than these two agents (4.50 > 3.89 and 4.45 > 3.87), suggesting that the distributional effects induced by TREE and ATT are actually more important. than their direct effects.
14

---

Published as conference paper at ICLR 2024
6.3 HARDWARE
We made use of four Tesla K80 GPU's for the purpose of experimentation, each with a VRAM of 12 GB. We show in table 11 the times taken for the most important lasks of the paper. As an example, if we wish to generate the games for an ATT, agent, Irain it, Lest. it to se= = 0.5, and obtain analysis metrics, it would take 10 hours for generation, hours for training, 10 hours for testing, and 12 hours for analysis, a total of 35 hours. Alongside experiments that are not included in the paper, we approximate the total compute time used by the project to be approximately 1 year's worth of our GPUs.
6.4 EXTRA EXPERIMENTS
6.4.1 AGREEMENT RATE OF VARIOUS ENGINES
We compute the agreement rate of various engines used in our experiments in figure 5 Notice how the ATT agents from both frameworks and calibrated to both juniors all play similar moves to each other and to LEELA, on which they are based. MAIA and MAIA 1900 also exhibit similarity to each other and dissimilarity to ATT agents. TREE exhibits moderate similarity to both the MAIA agents and the LEELA-derived agents. SFW, a weakened version (25 nodes) of stockfish 8 used in later experiment, is entirely dissimilar any other agent, likely due to its architectural uniqueness.
6.4.2 MODIFICATION OF TRAINING TARGET FOR ATT IN STT
The training procedure for ATT includes back boock-roppggatig on moves that both LEELA and MAIA play in STT. Semantically, this updates the value-head to take into account MAIA'S 's interven- tions, however, it also updates the policy-head to learn MAIA'S moves, which we believed would weaken the engine. Therefore, created version of the engine that only learns LEELA'' 's moves, as to not affect the policy with MAIA's 's moves, and only to have the value-head change to adapt to MAIA'S interventions. We also use an increased training games of 40000 because convergence with smaller datasets was worse (exclusion of MAIA moves halves quantity of data, and reduces its diversity, making it prone to overfitting). Interestingly, this turns out to be less effective than the default version in STT. (5.% 55..0.0.555.00.-- 0.5) than including the MALA policy moves, although the engine turns out to be very strong in raw strength, achieving a 36.5% 14.0% It is expected that the engine is stronger, as it is not ingesting MAIA moves, and we suspect that STT, training the policy on MAIA moves actually beneficial, as it allows the agent to conduct search at least partially based MAIA's 's moves, mimicking TREE to an extent.
6.4.3 COMPARISON OF TRICKING VS HELPING STRATEGIES IN STT
In order to compare the effect of attempting purely to sabotage MAIA,, to that of purely aiding the MAIAf, we modify the algorithm of EXP m = argmax (E [w](m,)))), which
elfectively optimizes only one ply in the future, eliminating any helping effect which requires at least. 2 ply foresight and making this a pure tricking engine. In order to isolate the helping effect, we use m = argmax (E.e 11)
(1,1 [m][m, which (falsely) assumes an un-exploitable
m
LEELA is always playing the opponent move, thereby forcing the optimization to be solely to help MAIA,- Both versions are able to beat LEELA in the framework, however the tricking version achieves higher score of 62.5% as opposed to the helping version which achives a score of 57.0% 1, both lower than that achieved by the actual EXP. We suspect that pure tricking casier to conduct than pure helping, as there no interceding agent to account for, and a shorter time horizon, hence less branching.
15

---

Published as conference paper at ICLR 2024
6.5
EXTRA FIGURES AND TABLES
maial100
maia1900
sfw
treel
tree9
attl.
att9;
att
att9
leela
malal100
1.0
0.65
0.38
0.56
0.52
0.52
0.49
0.51
0.48
0.43
maia1900
0.65
1
0.4
0.58
0.66
0.56
0.58
0.56
0.57
0.49
0.9
sfw
0.38
0.4
1
0.39
0.4
0.42
0.42
0.4
0.41
0.41
treel
0.88
0.56
0.58
0.39
0.69
0.64
0.62
0.65
0.63
0.56
tree9
0.52
0.66
0.4
0.69
0.67
0.7
0.67
0.69
0.61
0.7
attl.
0.52
0.56
0.42
0.64
0.67
1
0.82
0.76
0.76
0.71
att9
0.49
0.58
0.42
0.62
0.7
0.82
1
0.74
0.79
0.73
0.66
attin
0.51
0.56
0.4
0.65
0.67
0.76
0.74
1
0.8
0.66
att9h
0.5
0.48
0.57
0.41
0.63
0.69
0.76
0.79
0.8
0.69
leela
0.43
0.49
0.41
0.56
0.61
0.71
0.73
0.66
0.69
1
0.4
Figure 5: Agreement of different engines with each other
Table 5: Viability of ATT, engines created according to learning rate and dataset size
Learning Rate 10-1 10-2 10-3 10-4 10-5 10
1000 Games
10000 Games
100000 Games
6.6
MAIN PAPER TABLES WITH STANDARD DEVIATION
Tables 12
15
show the standard deviation of each value from the main text.
16

---

Published as conference paper at ICLR 2024
Table 6:
MAIA loss induced by different seniors in distributions occurrigg to different teams in STT
Ge
Go
MAIA loss induced by
LEELA
TREE
LEELA
EXP
LEELA
ATT
Distribution of
LEELA
3.60:10.02 3.89-0.02
3.25:10.02 4.47 +0.02 3.87+0.02 3.95
MAIAL
Distribution of
FOCAL
4.50:10.03 4.85+0.03 3.65+0.02 4.3940.02 4.33:10.01 4.4610.01
MAJAr
Table 7: Metrics by interaction type for TREE and ATT in HB
Interaction
Agreement
Correction
Disagreement
Blindsiding
Team
LEELA
FOCAL
LEELA
FOCAL
LEELA
FOCAL
LEELA
POCAL
MALA
MAIA,
MALAL
MAIA,
MALAL
MAIA,
MALA:
MAIA, 10+1
Gr Distribution
4411
60
20-1
16
21+1
14+1
1521
Gr Savings
0
0
5.4+-0.1
4.3+-0.1
-2.0=0.11
-211-0.1
0
0
G Opponent loss
3.2::..1
3.4+0.11
3.8:00.1
4.8:10.1
4.0.50.1
4.4:10.1
30.1
3.8:t0.1
G. Distribution
44+1
52+1
20+1
20+11
20+1
16+1
16+1
11+1
GA Savings
0
0
5.2=0011
4.6=0..1
-1.8=0.1
-1.9-0.1
0
0
G Opponent loss
3.2+0.1
3.2+001
3.8+0.1
4.0+50.1
3.9+0.1
3.910.1
3.3
+0.1
3.7:10.1
Table 8: Generalization results with MAIA1100 and MAIA1900 as junior partners.
STT framework
TREE
EXP,
ATT,
Tested on junior Designed for MAIA1100
MAIA
M90A
MAIA1100
MAIA1900
MAIA1100 55.0 52.0.10.5
MAIA 1900 51.5+0.5
56.5 51.0:L0.5
41.5=:.55
6..5
53.0 #0.5
Designed for MAIA1900
50.0
55.0
65.5=1.0
53.0
HB
framework
TREE
RXP
ATT;
Tested on junior
MAIA1100
MAIA1900
MAIA1100
MAIA1900
MAIA 1100
MAIA1900
Designed for MAIA1100
60.0
52.5=1.55
55.0+0.5
+1.5
56.0 54.0++0.5
37.5+1.
Designed for MAIA1900
57.0++0.5
58.0+0.5
51.5+0.5
55.0=0.5
54.0+0.5
Table 9: Generalization results with MAIA1100 and SFW STT
EXP;
ATT,
Tested on Designed for MAIA1100
MAIA1100 66.510.5
sfw 44.0+1.0 55.0
1AIA1100 35.0 48.5
sfw 44.0=1.0 51.0+0.5
Designed for sfw
43.5+1.0
Table 10: Generalization results with Specific Players in STT
Player A (1650 rating)
Senior trained for
Att
Tree
Exp
1aia1900
51 +0.5 Not trained
52.5 +0.5
57 +2 68 +2
Player
51.5 =0.5
Player B (1950 rating)
Type of Senior
Att.
Tree
Exp
Maia1900 Player B
51 +0.5
00.5
53 +- 66.5 12
Not trained
17

---

Published as conference paper at ICLR 2024
Table 11: Approximate times main lasks involved in experimentation
Task
Approximate Time (h)
1000 STT games, no EXP
1
1000 STT games, with EXP
2-3
1000 HB games, no EXP
1000 HB games, with EXP
2-3
1000 games with metric collection
4
10000 training ilerations
3
Table 12: Table 1 with standard deviations
STT
HB
Game Sctup
TREE
EXP.
ATT,
MAIA
TREE
EXPn
ATT
MAIA
focal
56.5#0.5
66.5+0.5
55.0+0.5 7.5 +1.
60.0
56.5+0.55
55.0 27.5 #1.5
MAIA
MAIA
focal
LEELA 0.5+0.5
14.0 #1.5 12.5 #1.5 0.0=0
0.5 N/A
4.5 #1.5 0.0
Table 13:
Table 2 with standard deviations
Agent
G₁
Ge
G.
Agent
G.
G.
G.
LEELA
1.15 +0.01
1.16
#0.01
1.17 +0.01
MAIA,
4.46
+0.04
4.21
#0.04
4.19
local
1.91
+0.01
1.37
+0.01
1,43
+0.01
MAIA
3.57
10.03
3.37
40.02
+0.02
Ac, (LEELA, fo
-0.76
+0.02
-0.21
#0.02
-0.26 +0.02
AG, (MAIAL, MAIAp.*)
0.89
10.04
0.84
#0.04
0.42 #0.03
Ac, (teamy, t
0.13 +0.03
0.63
+0.04
0.16
+0.02
Table 14: Table 3 with standard deviations
G2 LEELA -0.03
Ge LEELA
G.
Agent Tricking: (Jopp:0)
TREE
EXP
LEELA
0.54+0.07
-0.27+0.004 0.30 0.12+0.09
1.38+0005
-0.01+0.006
0.25+10.04
Helping: T(Jaar.0)
0.26+:0.11 0.15+0.09
0.36++0.10 0.46-0.09
0.61+0.07 1.88+0.07
0.34+0.06 0.20
0.21+00.5
Helping: 11 [(Jpar.1)
0.25+0.05
Indirect: (MMAIA, MAIA,,00)
0.5610.10
-0.1810.07
0.37
Table 15: Table 4 with standard deviations
Ge
Ge LEELA
G LEELA
MAIA focal
LEBLA MATAL
TREE
IXP MAIA,
XTT
LEBLA
MALA
MAIA;
MATAL
MAIA
MAIA,
MAIA>
True loss MAIA loss Savings
0.76+-0.02
3.44-0.02 3.002 00.02
3.33 3.62.10.02
3.6120.02 3.59-E0.02 -0.02.0.03
3.50-L0.02 3.03 -0.07
1..05
4.25-10.05
3.00 -0.04
3.39 10.05 -0.32L0.07
3.60--0.05
-0.0
-0.22-L0.03
0.3 0.03
-0.65=0.07
18


Reviews:
[{"review_id": "s7yuXgYsxo", "reviewer": "Reviewer_SKaW", "summary": "This paper studies the problem of making super-human AI chess programs more \"skill compatible\" with humans, in that the moves they recommend are more likely to be moves that a human could successfully follow up and thus would actually have success playing in a game. This is studied by building on open-source engines like Leela.  The proposed methods either do not require training or only require fine-tuning versions of a powerful base model.", "questions": "What is the relationship between this work and that of the exploitability and adversarial attacks fields?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This is clearly an important problem and is, to my knowledge, quite understudied. This appears to be a natural set of approaches that should be tried, and the thorough empirical evaluation of these approaches is useful for the community.\n\nThey also appear to be the first to have formalized this problem, and they have set up about as good a methodology for evaluating their method as one could hope for short of human trials.", "weaknesses": "Given that these approaches do not train from scratch, there is some concern that the local improvements may not be representative of the improvements you would hope for if the models were retrained.  \n\nSection 4.4 should be emphasized more from the beginning. It seems the biggest challenge with this line of work is the model of their partner.  It's not enough just to match the \"skill level\" of the human; but it is also important to match the style of the human and adapt to the patterns in human mistakes. In the rest of the paper, it appears this aspect is ignored, but in Section 4.4, evaluation is done by fine-tuning to mimic specific human players and thus could hope to account for the exact sorts of errors those humans are more prone to. I believe this section greatly strengthens the work, and it should be emphasized to ensure the reader understands where much of the remaining difficulty is.\n\nMinor points:\nThe “expector” agent appears to be quite similar to prior work[1,2], though without fine-tuning against the fixed opponent. It is also closely related to the opponent-exploitation work in the AI-poker community related to work like [3]. I would expect there to be some more similar work to this to be in the exploitability literature in game theory and in the adversarial attacks literature, investigating slightly different questions than this work.\n\n[1] Timbers, F., Bard, N., Lockhart, E., Lanctot, M., Schmid, M., Burch, N., Schrittwieser, J., Hubert, T., and Bowling, M. Approximate exploitability: Learning a best response in large games.\n[2] Wang, Tony Tong, et al. \"Adversarial Policies Beat Superhuman Go AIs.\" (2023).\n[3] Ganzfried, Sam, and Tuomas Sandholm. \"Safe opponent exploitation.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "udv7i3m70h", "reviewer": "Reviewer_9rEp", "summary": "The paper discusses the challenge of enabling powerful AI agents to interact effectively with agents of lower computational abilities. Using collaborative chess variants as models, the study proposes a framework to evaluate the compatibility of high-performing AI with lower-skilled entities. Traditional chess engines, though nearly optimal, struggled in this domain, leading to the development of three methodologies to create skill-compatible AI agents. These agents demonstrated superior performance in the proposed collaborative frameworks than conventional chess AI systems, such as AlphaZero. The paper emphasized that achieving raw performance in AI is not enough; the AI should also be compatible with lower-skilled agents for successful interactions.", "questions": "1. How can the methodologies be adapted for more dynamic, less rule-bound environments outside of chess?\n2. Would the proposed techniques be effective in real-time scenarios where the lower-skilled entity is a human?\n3. How does the proposed skill-compatible AI adapt to the continuous learning and evolving capabilities of lower-skilled agents?\n4. In real-world applications, how can the balance between raw AI performance and skill-compatibility be optimized without compromising on crucial tasks?\n5. Could the techniques be integrated into current AI frameworks for instant benefits, or would they require a complete overhaul of the system?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Innovative Concept: The paper introduces a novel and timely concept of \"skill-compatibility\" that addresses the real-world challenge of AI and human collaboration.\n\n- Empirical Evidence: The use of collaborative chess variants as model systems offers practical insights and empirical proof-of-concept.\n\n- Multiple Methodologies: The paper presents three distinct methodologies, showcasing the versatility of approaches to achieving skill-compatibility.\n- Comparative Analysis: By comparing newly proposed agents with state-of-the-art chess AI, the research demonstrates the tangible benefits of their approach.", "weaknesses": "The main weakness is that the methods proposed don’t show a clear path toward broader human-AI collaboration. Since you use the weaker chess engine as a subroutine in search, it isn’t clear how you could make a version with humans since they can’t communicate at test time. Furthermore, these methods seem limited to chess. It would be more interesting to have methods that would work for a variety of cooperative or cooperative/competitive games such as Bridge or Hanabi. \nMinor: formatting is off", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qfIHhXl7k9", "reviewer": "Reviewer_D1Jx", "summary": "The paper considers the interaction between highly skilled agents with less-skilled counterparts. The paper uses chess as a benchmark due to the availability of both a variety of highly and less-skilled agents at a multitude of skill levels. The paper introduces three methodologies to compare a tree agent, an expector agent, and an attuned agent (finetuned self-play RL agent). Prior art has considered simultaneous teamwork between AI and humans at tasks but does not consider if they are inter-temporally compatible. The paper evaluates against state-of-the-art chess AI and finds that their combined method can outperform. The paper evaluates in two scenarios: random swapping between the high-skill and low-skill AI after each move and a variant of chess called “hand and brain”, which requires one teammate to select the piece to move while the other teammate chooses the move selection.", "questions": "1) Tables 1-3 could be explained more clearly. Please consider bolding best results\n\n2) Why is the EXP method N/A in table two?\n\n3) Do you have any intuition as to why EXP performs best overall? Why does the paper not claim this as the best method if it has the highest winrate?\n\n4) The results are explained well after reading through the experiments section many times. However, I believe that this could be clearer. Please consider using a chart that summarizes the best combinations (no score, just to guide the intuition).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "+The wide variety of chess bots available at different skill levels and variety of play styles at each skill level makes for a very interesting and realistic benchmark. I could see this having direct influence on human-AI teaming.\n\n+The motivation for this paper is very strong. It is very interesting to consider the advice mismatch between superhuman AIs and mere human players. This is an open problem.\n\n+Introduces a new version of interpretability: “interpretable iff a weaker agent can follow–up”\n\n+Clear discussion of limitations.", "weaknesses": "-Although there are many comparisons between the lower-skilled agents and humans, this does not guarantee that the results will be the same with humans. It would be interesting to have a small experiment to confirm the results in this setting.\n\n-The description of the main results do not claim that there is a best guidance for what another researcher should try as the main takeaway", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ICoKJzXWXk", "reviewer": "Reviewer_XZq8", "summary": "The paper studies the problem of AI agents acting in team settings, particularly where the AI agent is more capable at the task than it's partner(s). Chess with both sides played by two players is used as a study domain, with two formulations of team play considered:\n1) stochastically switching between which of a pair of agents on the team chooses for any given move\n2) having one agent choose the piece type and the other choose the specific piece and move to make\n\nThese formulations are meant to target the capabilities of the stronger agent to work with the weaker agent either due to temporal uncertainty (1) or only indirect/partial control on action selection (2).\n\nThree agents are evaluated to play the role of the strong agent:\n1) Tree agent: Uses Monte Carlo Tree Search (MCTS) to augment a baseline policy (agent)\n2) Expector agent: Maximize win probability over a short horizon given access to models of all agents\n3) Attuned agent: Fine-tunes the agent from traces of agent pairs playing\n\nIn all cases the models have full access to models of the other agents in the game. Evaluations demonstrate that all three agents proposed improve over a naive baseline where a strong and weak agent are paired. Comparisons are made in terms of win rate and change in win probability for moves. Detailed studies are made to examine the mechanisms for the improvements, along with a small study of the ability of the expector agent to generalize to imperfect partner models.", "questions": "- Tables 1 & 2\n\t- Minor suggestion: Combine these two tables into one with a row defining the task (STT vs HB).\n\t- If possible it would help to include the error ranges in the main text.\n\t- Is there any intuition for why TREE does so much better than EXP in HB? The improvement is the opposite of what is observed in STT. Is the problem that in HB EXP has too limited of a search depth to handle the relevant decision horizon?\n- Section 4.2.2\n\t- The final conclusion from this analysis was unclear. \n\t- What are the implications that the standardized and non-standardized board position distributions differ substantially for TREE and ATT, but not EXP? This seems to suggest tricking is the not the main effect (as opposed to some more general distributional difference), but that was not completely clear either.\n\t- It may be better to remove this analysis (to the appendix) and provide more detail in the main text on the normalization applied in 4.2.3.\n\t- Section 4.2 would benefit from stating the conclusions drawn on tricking, helping, and indirect effects at the start of the section. A similar comment applies to section 4.3.\n\t- I could not understand from the text: which agents demonstrate tricking (and under what analysis)?\n\t\t- For helping it is more clear (based on the more complex analysis) what is happening.\n- Section 4.4\n\t- Are there results when the MAIA agent is selected without a rating prior? This would be interesting to see to understand how well a focal can compensate when the \"amount of weakness\" of the partner is highly uncertain.\n\t- One approach would be randomizing the skill of the partner agents over a reasonable range of ratings.\n\t- Are there comparable results for using TREE? I understand ATT is more computationally demanding to add, but those results would also be interesting.\n\t- In general I consider these results to be some of the most important for the work and would trade less detailed analysis from sections 4.1, 4.2, and 4.3 for further elucidation of how well the approach generalizes when given imperfect partner information.\n- On the supplement\n\t- The results on using MAIA1900 and stockfish are both good information to add to the main body. These provide more color to the discussion about capabilities to partner with other agents and would be welcome in the main text.\n\t- I personally would trade these for the space used in sections 4.1, 4.2, and 4.3 as the final conclusions on the mechanisms at play remain unclear to me, while the conclusions about capacity to generalize (or not) seem more clear. \n\t- \"This does indicate, that our agents' compatibility is not a function of merely skill, but also style.\" An important point to make in the main text! Also very relevant for future work that would investigate other ways to enhance agent compatibility.\n\nTwo broad conceptual questions came to mind that I would appreciate further comments on:\n- Why is inter-temporal coordination the right framing for evaluating agent compatibility?\n\t- Perhaps sketch some specific applications where this is a natural interaction paradigm outside chess. The example that came to mind is autonomous vehicles (and robotics more generally), where humans may take over at arbitrary points from an AI driver.\n- How correlated is this notion of interpretable with prior efforts?\n\t- The term interpretable is quite loaded in the literature, so I am trying to understand why it is a good description of the method here.\n\t- Prior efforts normally aim to enable inspection of the behavior of a model, but that is not obviously what is demonstrated here. Weak agents can perform better when in scenarios prepared by the strong agents. But it's not obvious that is due to any ability to \"interpret\" these situations, as opposed to being given better initial states to use.\n\n---\n\n**Author discussion amendment**\n\nI have revised my score up to reflect the improved paper structure and clearer promising results on generalization with other types of agents. These reflect the methods generalize across some dimensions (human players and skills) but not others (whatever makes Stockfish different to humans). These point toward important future research efforts.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# originality\nFraming collaboration (compatibility) as the ability to cope with partners under the specific cases here is novel. While AI teaming in chess is well-established, using stochastic turn taking or hand-brain teams as a way to assess agent coordination is novel. The proposed agents are all reasonable extensions of methods for opponent or partner modeling to these scenarios.\n\n# quality\nThe experiments thoroughly examine the performance improvements of the varying approaches under two different conditions. The tests quantify uncertainty and demonstrate a clear benefit both to long-term win rates and intermediary probabilities of selecting good moves (optimal with respect to a given agents' predictions). These demonstrate that the models for agent pairing are achieving the desired effect of producing teams stronger than their individual composite agents.\n\n# clarity\nThe text is clear on the technical approach and provides good framing of the core frameworks.\n\n# significance\nHuman-AI collaboration is of growing interest (and public concern) as AI systems are more widely deployed. Developing new methods to assess how teams work together and can be robust to differing agent capabilities is a timely and important topic. This will be widely of interest across subcommunities at ICLR (and beyond).\n\nWorking specifically in games has clear significance to research in game playing, reinforcement learning, and efforts at multi-agent coordination.", "weaknesses": "# originality\nNo major comments. If there is room, the related work could make further reference to the literature on zero-shot coordination, which shares the goal of enabling teams of agents to work well together, but subject to the constraint of lacking a model of the partner agents. Some examples:\n\n- Hu, Hengyuan, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. \"\"other-play\" for zero-shot coordination.\" In International Conference on Machine Learning, pp. 4399-4410. PMLR, 2020.\n- Lupu, Andrei, Brandon Cui, Hengyuan Hu, and Jakob Foerster. \"Trajectory diversity for zero-shot coordination.\" In International conference on machine learning, pp. 7204-7213. PMLR, 2021.\n- Treutlein, J., Dennis, M., Oesterheld, C. and Foerster, J., 2021, July. A new formalism, method and open issues for zero-shot coordination. In International Conference on Machine Learning (pp. 10413-10423). PMLR.\n\n# quality\nResults on the mechanisms behind the improvement gained by different focal agents are murky. For example, \"tricking\" may be implicated for EXP, but not TREE or ATT (though I may have misunderstood the text on this point). In general, it is hard to assess how strong some of these results are and what the implications may be. More specific questions are provided below.\n\nConversely, few results test the generalization capabilities of the agents, particularly when relaxing the assumption of access to a perfect model of partners and opponents.\n\n# clarity\nThe results presentation is somewhat murky, particularly in the detailed analysis of mechanisms (see the questions for specific details). In general the results would benefit from a clear statement of the outcomes of the analyses at the start of the section, followed by the existing detailed description of results. This will help readers glean the overall conclusions that get lost in some of the details.\n\n# significance\nThe primary limitation to the impact of the results is the requirement for agents to have models of all relevant agents in the scenario. One experiment slightly relaxes this constraint and the results become substantially weaker. It would be beneficial to add some remarks on this point and future directions mitigate these weaknesses with imperfect partners. As a first work on this method I do not view these as substantial limitations, but they deserve some additional comment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)