Safe Opponent-Exploitation Subgame Refinement
Authors: Mingyang Liu
Chengjie Wu
Qihan Liu
Yansen Jing
Jun Yang
Pingzhong Tang
Chongjie Zhang
Venue: NeurIPS 2022 Accept (2022)

Abstract: In zero-sum games, an NE strategy tends to be overly conservative confronted with opponents of limited rationality, because it does not actively exploit their weaknesses. From another perspective, best responding to an estimated opponent model is vulnerable to estimation errors and lacks safety guarantees. Inspired by the recent success of real-time search algorithms in developing superhuman AI, we investigate the dilemma of safety and opponent exploitation and present a novel real-time search framework, called Safe Exploitation Search (SES), which continuously interpolates between the two extremes of online strategy refinement. We provide SES with a theoretically upper-bounded exploitability and a lower-bounded evaluation performance. Additionally, SES enables computationally efficient online adaptation to a possibly updating opponent model, while previous safe exploitation methods have to recompute for the whole game. Empirical results show that SES significantly outperforms NE baselines and previous algorithms while keeping exploitability low at the same time.

Safe Opponent-Exploitttio Subgame Refinement
Mingyang Liu'!, Chengjie Wu"!, Qihan Liu?, Yansen Jing', Jun Yang?,
Pingzhong Tang', Chongjie Zhang'
Institute for Interdisciplinary Information Sciences. Tsinghua University Department of Automation, Tsinghua University
(liumy19, wucj19, 1qh20, j jingys19)@mailss tsinghua. edu. cn fyangjun603, kenshin, chongjie)@tsinghune edu.
Abstract
In sero-sum games, an NE strategy tends to be overly conservative confronted with opponents of limited rationality, because it does not actively exploit their weaknesses. From another perspective, best responding to an estimated opponent model vulnerable to estimation errors and lacks safety guarantees. Inspired by the recent success of real-time search algorithms in developing superhuman AI, we investigate the dilemma of safety and opponent exploitation and present a novel real-time search framework, called Safe Exploitation Search (SES), which continuously interpolates between the two extremes of online strategy refinement. We provide SES with theoretically upper-bounded exploitability and a lower- bounded evaluation performance. Additionally, SES enables computationally efficient online adaptation possibly updating opponent model, while previous safe exploitation methods have to recompute for the whole game. Empirical results show that SES significantly outperforms NE baselines and previous algorithms while keeping exploitability low at the same time.
Introduction
Behind the recent breakthroughs of superhuman AIs in Go Siiler et al. 2016 2017 Schrittwieser
et al. 2020), heads-up no-limit Texas hold' em (HUNL) (Brown al.] 2018 Moraveík al. 2017 Brown and Sandholm 2019 Brown et al. 2020). and Hanabi [eerer et al. 2020). search plays a vital role. In perfect information games, Monte Carlo tree search (MCTS) is widely applied improve policy's strength. In zero-sum imperfect information games such as poker, search algorithms are used find Nash equilibrium (NE) approximation in subgames encountered in real time [Brwnn and
Sandholm 2017 Burch et al. 2014.. They are both theoretically sounded and empirically powerful.
In zero-sum games, NE-based search algorithms [Burch et al. 2014 Moraveik et al. 2016 Brown and Sandholm 2017 Brown et al. 2018) find safe strategies with low exploitability and produce strong baselines against all opponents [Brown and Sandholm 2019 However, it may be overly conservative confronted with opponents with limited rationality, and fail to take advantage of their weaknesses to obtain higher rewards [McCracken and Bow ling] 2004 Johanson al. 2007 Li
and Mikkullainen 2018 8). From another perspective, there have been extensive studies on opponent
exploitation address the problem. Some typical works (Carmel and Markovitch 1996 Billings
et al. 2003 Gilpin and Sandholm, 2006 Li and Mikkulainen) [018]] model the opponent's strategy based on previous observations and then search for new strategy exploit this model. However, these methods often neglect the significance of the strategy safety, thus being highly exploitable.
Few exceptions including Johanson et al. [2007] and Ganztried and Sandholm 2015a aim to search
for safe and robust counter-strategies. Ganzfried and Sandholm [2015a] provides a characterization of
'Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

safe deviations from NE in repeated games. Restricted Nash response (RNR) Johanson et al. 2007 finds a Pareto optimal strategy with respect to safety and exploitation in the full game. However, is computationally inefficient because it needs to recompute a strategy for the whole game whenever the opponent model is updated. This can be even infeasible in an online setting where the opponent model is being updated continuously with streamed data.
In this paper, we study the dilemma of safety and opponent exploitation and present new scalable real-time search framework Safe Exploitation Search (SES) that continuously interpolates between the two extremes of strategy search, hence unifying safe search and opponent exploitation. It enables computationally efficient online adaptations to a continuously changing opponent model, which is hard address by previous safe exploitation algorithms. The safety criterion requires the refined strategy to stay close to NE, formally speaking, to expose limited exploitability against any opponents, while the opponent exploitation criterion requires the strategy to adapt to its specific opponent and to exploit its weaknesses. We propose novel maximization objective in the subgame search framework which combines the safety objective and exploitation, controlled by the exploitation level a. We construct a new gadget game optimize this objective, which enables our method's scalability to large games such as Texas Hold' em. Theoretically, we prove that SES is guaranteed to outperform NE at the cost of some constant increase in its own exploitability confronted with non-NE opponents.
Empirically, we evaluate the effectiveness of our search algorithm in didactic matrix game poker games: Leduc Hold' em (Southey et al. 2005) and Flop Hold' em Poker (FHP) (Brown et al. (019)] The experiment results demonstrate that our algorithm significantly outperforms NE baselines against non-NE opponents and keeps low exploitability at the same time. Additionally, we show that SES is not only much more computationally efficient than previous safe exploitation methods but also more robust to estimation errors in opponent models.
Related work
This paper investigates the problem of safe opponent exploitation two-player zero-sum imperfect information games. We propose novel search algorithm that balances between NE and exploiting opponents. Two major relevant research areas are search algorithms in imperfect information games and opponent exploitation.
Search in imperfect information games. In recent literature, search techniques are witnessed to be important in developing strong Al strategies in both perfect and imperfect information games [Burch et al. 2014 Moravcik al. 2016 Brown and Sandholm [017]. Texas hold 'em poker widely employed as a benchmark for imperfect information games. A primary part of the long-term research on Texas hold'em poker is the evolution of subgame solving algorithms, which aim at achieving a more accurate Nash equilibrium approximation in the subgame encountered given a pre-computed strategy for the full game which we refer to as the blueprint strategy. Unsafe search (Billings et al.
2003 Ganzfried and Sandholm 2015b Gilpin and Sandholm 2006 2007 estimates the subgame reach probability assuming the opponent follows blueprint, and searches for a refined subgame strategy. Subgame resolving (Burch et al., 2014] and maxmargin search [Moravcik al. 2016
are theoretically sounded safe search algorithms which ensure that the subgame strategy obtained is no worse than the blueprint. They search in gadget game and achieve safety by providing the opponent with the option not entering the current subgame. DeepStack [Moravcik et al., 2017]
and Libratus (Brown al. 2018 build strong poker ÁIs with the aid of search. Beyond poker, search algorithms for subgame refinement have also shown promise in improving joint strategies in cooperative imperfect information games such as Hanabi [Lerer et al. 2020 and the bidding phase of contract bridge (Tian et al. 2020]. The purpose of our search a algorithm is different from previous methods in poker literature. We seek to exploit opponents while keeping exploitability low, rather than simply approximating NE.
Opponent exploitation. Most previous opponent exploitation researches (Carmel and Markovitch
1996 Billings et al. 2003 Gilpin and Sandholm 2006 Li and Mikkullainen 2018 typically model
the opponent's strategy based on previous observations and then search for new strategy exploit this model, but put little emphasis on safety.
One similar work is Johanson et al. (2007) which proposes p-restricted Nash response (RNR) to find a safe exploitation strategy to the estimated opponent's 's strategy. calculates a Nash equilibrium for the whole game restricting that the opponent plays the estimated strategy o gfix with probability p.

---

and any strategy with probability 1 - p. In that paper, Johanson et al. [007]] prove that p-RNR to o fix is Pareto optimal with respect exploitation and safety, However, does not provide an explicit bound. Additionally, whenever the estimated opponent model changes or we want to use a different p to balance between safery and exploitation, the original p-RNR has to recompute the strategy for the whole game. It is computationally inefficient in an online setting, where the opponent model is updated after every round with new game data. Our algorithm instead takes modeling error into account and provides explicit bounds for both safety and exploitation. With the aid of real-time search, it only searches for strategies in subgames encountered instead of the whole game.
Ganzfried and Sandholm 2015a study safe exploitation strategies in repeated games, which is a different setting from this paper. Intuitively, it achieves safety by risking in exploitability at most what it has earned over NE in expectation in previous rounds. Therefore, its expected value in the whole repeated game never worse than the NE. contrast, this paper focuses on the safety of stage game strategies. Furthermore, our algorithm is complementary to Ganztried and Sandholm 2015a Ganzfried and Sandholm 2015a] calculate an e-safe best response for the whole game at each iteration with LP. This procedure is one of the main limitations on the algorithm's scalability. Our algorithm can be a possible efficient substitute for the calculation.
Bernasconi-de Luca et al. [021]] use a UCB-like algorithm to learn how to encourage opponent's engagement in repeated games setting by guaranteeing that the opponent's 's utility lies in the desired range. Moraveik et al. [2017] use similar mixing distribution technique as ours to speed-up up resolving procedure for NE by fixing the distribution of infosets on top of the subgame with a prediction of some unknown NE strategy. This work does not study opponent exploitation. Besides, it does not provide a safety or an exploitation bound for the algorithm.
To our knowledge, we are the first paper to investigate the safe opponent exploitation problem in subgame resolving schemes. Subgame resolving enables online adaptations to a continuously changing opponent model, eliminating the need to recompute a whole game strategy. It offers computational benefits in practical opponent exploitation circumstances. Our experiments in section S demonstrate its efficiency and robustness.
There is extensive research [Albrecht and Stone 2018 agent modeling. However, this paper only focuses on the safe exploitation algorithm, but not the agent modeling techniques. We can use off-the-shelf agent modeling algorithms to estimate the opponent's strategies.
3 Notations and background
An extensive-form imperfect information game G Z,, " I. describes sequen- tial interactions among agents, where agents have private information. A finite set P consists of players and a chance node which represents the stochastic nature of the environment. The set of non-terminal decision nodes is denoted as H. and Z is set of terminal nodes leaves. The set of possible actions is A, and H + 2| is a function which assigns to each decision node h E H a set of legal actions. A player function p H - P assigns to each decision node a player p E P who acts at that node. If action a leads from h to h', we write h = h1. If there exists a sequence of actions leading from h to h/, we write h 5 h². At each node h E H, the acting player p = p(h) picks an action from legal actions E (h). and leads node h into its child h- a. The chance node always samples an action from its own distribution a.. which is common knowledge to all players. Utility functions are u (M1, Ug...., ). where
Z + R defines the utility of player at terminal node 2 E Z. The nature of imperfect information is characterized by infosets I = I2..I....... II,, where I = 1 ... Is,R;) ) is a partition of H for player i. Two states in the same infoset must have the same acting player and the same legal action sets. We use [(h) to denote the infoset that belongs to. A player p cannot distinguish between states h and h if T(h1) T(h2), and thus should behave identically on all states in the same infoset.
The strategy of a player p op Ip x + R, where (1, a) is a distribution over valid actions on infoset 1. For simplicity, we also use op a) denote o,, (h), a). We use र" (h) to denote the prob- ability of reaching state h from the root when agents choose a strategy profile o = (01, 02. [L>-oChAp(hh) a Tnthe (h', a) to de- note the probability of reaching h when player p always chooses the action that leads to whenever possible. x² (h, h²) is the reaching probability of h'' from h. x² ah a, h²) )' is the the probability of
an)).
Formally, x" (h) II (p(w)) a). We use (h,
3

---

reaching h from h if action a is taken at h. These probabilities can be formally defined in a similar manner. A blueprint a pre-computed strategy for the full game.
The expected utility of player p given strategy profile a is us = => x" (z)up(2). The counterfac- tual value ug (I, a) is the expected utility that player will obtain after taking action a at infoset I, given the joint policy profile is Mathematically, it is the weighted sum of expected values at all states h E
(1, = EnELSEZ T²p (h)(T)" (h a, 2)up(z)
(I)
EnEr T- (h))
We further define vs (T) EaEa Te mh (T,))
In the rest of the paper, we focus on two-player zero-sum games with perfect recall. Zero-sum means Vz E Z, (1)= + (=) O. Perfect recall means that no player will forget the information which has been obtained previously in the game. This is a common assumption in related literature.
A best response strategy (o- arg maxo, Up ( 2) for player p is the strategy that maximize his own expected utility against fixed opponent strategy a-p- The exploitability of strategy op is exp(op) = up zero-sum games. It measures the performance of op against its best response comparing with the NE. A counterfactual best response CB eg () 2 max; up b). Counterfactual best response is a best response, but not vice versa. The
(5..BR_,(0.)) where o² is the optimal strategy, and is an NE in two-player
is a strategy where (,(4, a) > 0 if and only if
counterfactual best response value CBV 10-32--000 (() = Up (CBR,(0-p)0--) (7) the expected utility of the counterfactual best response policy. Since we focus on two-player zero-sum games, we will use CBV (I) as a shorthand notation for (I))
We follow the imperfect information subgame definition as in Burch et al. [2144). An augmented infoset contains states which cannot be distinguished by the remaining players.
Definition 3.1. An imperfect information subgame S forest of trees, closed under both the descendant relation and membership within augmented infosets for any player. Let Stop be the set of nodes which are roots of each tree S.
4 Method
In this section, we introduce our novel search algorithm called safe exploitation search (SES), which exploits the weaknesses of the opponent while ensuring a bounded exploitability efficiently. Let a be the pre-computed blueprint strategy. Without loss of generality, assume search for player 2's refined strategy of by applying SES to all subgames e S. Finally, the refined strategy for P2 after search is oz which is the same 02 in {LSVS E 1 & S} and is replaced with og in S S.
4.1 Safe Exploitation Search
Our algorithm offers unified approach balance these two demands with theoretical guarantees. The objective of our search algorithm is to find a new subgame strategy of for S E S which maximizes
SE(OE)= =a Σ p(H)) - CBV (r))
- (12).
(1 a-) min 1/ES
where E [0, 1] is hyper-parameter controlling the exploitation level, and ((T/) is the estimated probability of player 1 entering infoset I Stop. Given P2's strategy (which is the blueprint σ2)) and Pl's actual strategy (which does not have to be the blueprint 01), the real probability of player 1 entering infoset T/ E Stop (which we denote as p(T?)) is determined. p(T?) is an estimation of p(()). For instance, in poker, it is the estimated distribution of private cards player I holds. Both theoretically and empirically, such estimation does not have to be fully accurate. It can be done with off-the-shelf opponent modeling techniques, which lies beyond the focus of this paper.

---

Intuitively, the maximization objective achieves balance between opponent exploitation and safety, controlled by exploitation level a. The first part of the objective is maximized when Enfes... (U))CBV (I)) is minimized. It aims finding strategy oं which results in the lowest value for PI under the assumption that the reach probabilities is p. It can be interpreted as exploiting the estimated PI's strategy. The second part of the objective demands the resolved strategy to behave well against any reach probability distribution. We use the subgame margin min m
(UT(T)) CBV (T)) [Moravcik et al. 2016] which can be regarded as the worst- case utility increase for P2.
Our search objective is convex combination of exploitation and safety, which is closely related to previous safe exploitation research [McCracken and Bowling 2004 Johanson et al., 2007]. RNR Pohanson et al., 2007] calculates an exploitation strategy by computing an NE in the full game, restricting the opponent to play its fixed strategy with probability p and any other strategy with 1 - RNR is proved to be Pareto optimal with respect to safety and exploitation. However, neither provides an explicit bound on exploitability and performance nor takes modeling errors into account. Furthermore, without search, RNR has to recompute for the whole game whenever the opponent's strategy changes, which limits its efficiency. Experiment section 5.3 demonstrates that SES is much more computationally efficient, and section 5.4 shows that, even we augment RNR with search framework, SES is still much more robust to estimation errors in opponent strategy.
By maximizing the objective 2 we provide sound theoretical results for both safety and opponent exploitation. Additionally. we provide analyses of how (1) exploitation level a, (2) accuracy of opponent modeling, and (3) strength of the blueprint strategy impact the theoretical bound. By gradually increasing from 0 to 1, our algorithm tends to exploit rather than keep safe.
Theorem 4.1. (safety) Let S be a disjoint set of subgames S. Let o² = -(ii, 02) be the NE where P2's strategy is constrained to be the same with 02 outside S. Define A- =maKgeB.IfSSton ICBV,' (11)- of (T))- Let ((Ti) be the reach probability given by oi. Let p(Ti) be the estimation of reach probability p(I given by the real opponent strategy. Define T = maxses.1ESS.o
-
P(T))
Whenever (2r + 1)xx >> O, we have bounded exploitability given by:
2 exp(o')) S explo0) + 1 (2T + 1)a A.
(3)
Recall that og is the refined strategy after search. The proof is provided in Appendix This theorem implies that the exploitability of the new strategy is smaller than that of strategy 02 plus constant value, which is the closest strategy to NE if constrained to differ from 02 only in S. The corresponding theoretical result of maxmargin search (Moravcik et al. 2016). a safe search algorithm with no opponent exploitation abilities, is cxp(o²) S exp(aς) +2A. Comparing these two results, we can interpret the term 2/(1- (2r 1)a) the additional risk introduced by exploiting the opponent. If exploitation level a = O, then our bound is as tight as that of maxmargin search [Moravcik et al. 2016). The bound also gets tighter if the J gets smaller, or the blueprint 02 is closer to dz-
Theorem 4.2. (opponent exploitation) Let = -11Ø- pll1 be the LI distance of the distribution p(It) and p(It). Let = minses ma (CBV CBV 2 0. We use BR BRR (a) to denote the strategy for player which maximizes its utility in subgame S E S against a-p under the constraint that BR B. (a) and op differs only inside S. By maximizing objective for all S E S, the refined strategy o.2 satisfies
(BR) (o2)
{ (BR (o2) (
U2
(S) 2 U2
(S)
(4)
a (1- (92-24) -
The proof is provided in Appendix B Observe that the reach probability p is characterized by Pl's strategy outside S and & is its estimation. Because the search algorithm always find stronger response strategy for P1 in S (which is exactly BR. (o¹)) as well, opponent exploitation refers to adapting to Pl's 's's strategy a outside S. This theorem implies that the utility of the new strategy
5

---

0/2 is lower bounded by the utility of σ2 when both confronted with Pl's unknown strategy outside S. It provides theoretical guarantees for the opponent exploitation ability of our algorithm. can be interpreted as estimation error. The lower bound increases if the estimation error get smaller or the blueprint 02 is closer ozz We show empirically how exploitation level and estimation error impact both safety and exploitation abilities in section S
4.2 Gadget Game
S
Sz
Figure 1: The gadget game of SES. The shadow and dashed line indicate that player 2 cannot distinguish between the two branches. C represents chance node, P1 represents player I's action node. 81 and S2 two identical copies of the subgame S with utility shifted.
In order to find of which maximize objective a strright- method to reformulate the maximization problem as Linear Programming problem [Moravcik et al. 2016) However, LP solvers (Koller et al. [994] cannot handle large-scale problems. Alternatively, inspired by Moravcik et al. [016]. we create gadget game and then apply iteration-based NE algorithms such as CFR Zinkevich et al. 2007 Tammelin et al. 2015 Lanctot et al. 2009) in the gadget game. The gadget game is carefully designed such that the NE solution found is exactly the solution to the original optimization problem.
As shown in Figure [ the original subgame is copied into two identical parts S1, in the gadget game. Player 2's infosets stretch over both branches, while player can distinguish between the two parts. We use P1 to denote player 1, and P2 for player 2. The procedure of constructing such gadget game can be summarized into steps as described below:
1. Create a chance node at the top of the gadget game. It goes to the left part with probability 1 a, and the right part with probability a. The outcome visible P1 but not P2. Therefore. corresponding nodes in both branches are in the same infosets for P2, and his strategy o will be the same for both parts. For Pl, notice that SI and S2 only differs the distribution of the infosets of player 1. Given the strategies of player 2 in S1 and S2 are the same, the difference between the distribution of infosets of player on the top of the subgame will not change the counterfactual value of player 1. Therefore, we will have the same counterfactual regret and the same strategies generated by CFR for player S1 and S2.
2. We subtract un (z) by of (Ti) for all = C h, h E Ii where I denotes the infoset on the top of the subgame, and add u2(2) by of (Tf) in order keep the subgame zero-sum on both S1 and
S2. By doing so, the objective of p2 will change from maximizing -CBV (TA) maximizing
of (TE) C (7)).
3. As for the left part of the gadget game, the Pl node enables P1 to enter an arbitrary infoset
R. The following chance nodes sample specific state with probability proportional to =_, (h) for
all E 1,. Since this is a zero-sum game, in an NE strategy, he will enter the one with lowest
of (4f) ( az) which is exactly the minimization in the second term of SE(af).
The chance node in the right part will sample an infoset Ij according to reach probability p (T)) The following chance nodes again sample a specific state with probability proportional to x (h) for all h 4, So the NE objective of this part is exactly the summation in the first term of SE

---

The pseudocode of SES is shown in Appendix A We also provide didactic matrix game in Appendix Das as an example show the necessity of considering safety and expected payoff simultaneously, and to demonstrate the superiority of SES over a simple mixing strategy.
5 Experiment
Our experiment is done in Leduc Hold' em [Southey et al, 2005 and Flop Hold' em Poker (FHP) [Brown et al. 2019]. Leduc Hold' em is smaller-scale poker game and FHP is larger one. The rules of these two pokers are provided in Appendix C
We demonstrate the exploitability and evaluation performance of SES against opponents of various strengths. The exploitability measures search algorithm's 's safety, while head-to-head evaluation measures the ability of opponent exploitation. We also illustrate how estimation accuracy of opponent's 's strategy and the exploitation level a impact the results. Please refer to Appendix for implementation details.
5.1 Opponents
In our experiments, we test the performance of our algorithm against opponents of various strengths. We create 3 types of opponents with 3 random seeds each. The first one is an approximation of NE, and is regarded as a strong opponent. For the second and third type of opponents, we enumerate every infoset in the blueprint strategy and shift the action distribution randomly with probability Prehumte = 0.3 or 0.7. We multiply the probability of each action by a random variable from Uniform(0. 1), and then re-normalize the probability distribution. The procedure is motivated by Brown et al. (2018), in which such method is applied to create number of diverse but reasonably strong agents. Even when Prsufftc = 0.7, the strategy keeps close to NE with average LL distance of each infoset 132 comparing to 1.036 of a random strategy to NE. So they are regarded as opponents who are not fully rational but with competitive strength.
5.2 Safe Opponent Search
In Figure 2 we demonstrate the head-to-head evaluation performances and corresponding exploitabil- ity of the refined strategies found by SES against opponents of various strengths, under different exploitation level a and estimation errors of opponent's ' strategy. Different lines in each plot refers to corresponding estimation error c, which is the L1 distance of ý and p. We evaluate our refined strategy when c 0.0, 0.3, 0.6, 0.9, 1.2. Please refer to Appendix E for details of generating opponent estimations. The blue line is the result of blueprint strategy without conducting any search.
Generally speaking, SES balances between safety and opponent exploitation. The increase of exploitation level helps win more chips from opponents, while resulting in the increase of the strategy's own exploitability. As can be seen Figure - the exploitability increases when the exploitation level a grows from 0 to 1, which is consistent with Theorem FHP when e is small: the exploitability surprisingly keeps decreasing even SES puts more emphasis on opponent exploitation. Similar situations have also occurred in previous literature [Brown and Sandholm 2017]. The reason is that our opponent is quite close to NE outside the subgame which will make p close to when is small, which means the in Theorem 4.1 is small. As a result, we will have a wwwexpplotaability resolved strategy when using unsafe search and the exploitability increases increases.
One exception is in
When the estimation completely correct (e = 0.0, the yellow line), the expected payoff in FHP (3rd row in Figure 2 increases as the exploitation level grows higher. In Ledue poker, since the game is very small, the pre-computed blueprint is very close to NE. Therefore, when confronted with relatively strong opponents (Prahuffle 0.0, 0.3) which are also close to NE, actually few things can be done other than sticking with the blueprint. So the improvement introduced by SES is small. When facing relatively weak opponent (Pratuftte 0.7), the improvement margin is slightly larger.
SES relies on an estimation of opponent's strategy. In order to test the robustness of our algorithm when the prediction of p([f) is not accurate, we evaluate the performance of our algorithm with different values of estimation error e. As illustrated in Figure the exploitability increases and the expected payoff drops when grows larger. The result is expected since an accurate estimation always provides benefits. However, also demonstrates that SES can still achieve a trade-off between safety and opponent exploitation even when is considerably high. For instance, in FHP, is between 0 and
7

---

EvelustioniPto.w m 0.0)
EvalustiondP O 3131
Evaluation(P m 0.7)
Explotabiity(PPooee # 0.01
Exp ExpoatabbiiityPPPs 0.3)
Explsitability Proece i 0.71
Lurde 2.5
224
Co
350
250
180
130
399
18
82
3.4
47
0,2
0.4
80
14
Evalustoon/Pyuure 0.0)
Evaluation(PPaune 0.31
Evailutionn Soutte = 0.7)
(1263
12
ExplottablillPraucs 0.0)
Explattabiity/Pauutt
Explotablity(PPaaae 0.7)
Figure 2: Experiment results on Ledue (Row 1 & 2) and FHP (Row 3 & 4). Row 1 & 3: Head-to-head payoffs against corresponding opponents. Row 2 & 4: Exploitability. Each row represents a type of opponent with Pratumte 0.0, 0.3, 0.7. The X- X-axis is the exploitation level a.
2, and c = 1.2 means that the predicted distribution is almost random. When c S 0.6, the expected payoff still keeps increasing with respect to a. In case of bad estimation, we can always choose smaller a to ensure safety.
5.3 Comparison with Restricted Nash Response
We also compare SES with restricted Nash response (RNR) [Johanson et al.) 2007 a previous safe exploitation algorithm, in FHP. RNR calculates an NE for for the whole game restricting that the opponent plays the estimated strategy Ffix with probability p. and any strategy with probability 1 p. In each round, we limit the computation time of (No(mllll to 10 CPU second which is the same for SES. However, as stated in section F. RNR needs to recompute strategy for the whole game in each round. It cannot converge in 10s. So we also compare with RNR(big), which has a budget of 10M CFR iterations in each round (around 190 CPU second in time). In contrast, SES only uses 10M CFR iterations to calculate its blueprint once. As is shown in Figure S SES significantly outperforms (no(ma)) in both exploitability and evaluation. SES also achieves much lower exploitability than RNR(big) and comparable evaluation results with much less computation time.
5.4 Comparison with EXP-STRATEGY
It is possible to augment p- RNR with real-time search. For subgame S, we can create a similar gadget game to SES Figure D
The difference is that it keeps the opponent strategy fixed to its
We test it on Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz
8

---

Evalustian/PPuars 0.0)
Evailaation 0.31
Evallattoon Pours - 0.7) mome
Explolabblil/Pra..c
Explatabilty(PPsuus
Explotibbllt(PPoows 0.7)
216t
Figure Comparison between SES and RNR. Each row represents a type of opponent with Prshulle 0.0, 0.3, 0.7. The X-axis is the parameter for SES and for RNR.
Table 1: Exploitability and evaluation performance SES and STG.EY.A. Strategy estimation
error is 0.1, and Fic 0.3.
a(p)
EXP-STRATEGY
SES
0.0
29.43(#0.01)
29.43(=0.00)
0.3
34.75(110.13
28.06(110.4)
EXPLOITABILITY
0.7
134.37(42.59)
41.95(=0.39)
1.0
981.47(#11.86)
55.10(=11133)
0.0
5.15(+0011)
5.15(+0.19)
0.3
1.63(++0.40)
6.21(+
EVALUATION
0.7
-27.22(+0.91)
6.00(+0.81)
1.0
-88.94(16.07)
3.30(110.96)
estimation the whole exploitation part, while SES only uses the reach probability á calculated from the estimated opponent strategy and allows the search algorithm to find opponent strategies in the exploitation part as well. To our best knowledge, this algorithm does not exist previous literature, and can be regarded as an ablation study for SES. We call it EXP-STRATEGY.
Table T shows the exploitability and evaluation performance of SES and STRATTYATEYY under different exploitation level a (or in RNR). We add errors in the estimation of opponent strategy, and use the same estimation for both algorithms. SES performs much better than EXP-STRATEGY. maintains lower exploitability and achieves higher evaluation performanced The experiment shows that EXP-STRATEGY is sensitive to modeling errors, because relies on the estimated strategy in the whole subgame. SES, which exploits distribution of infoset instead of the full opponent strategy, more robust. Please refer to Appendix for full comparison results.
Similar to SES, we derive the theoretical bounds for both exploitability and exploitation for EXP- STRATEGY. And the theory also demonstrates that EXP-STRATEGY is more sensitive to the accuracy of the estimation. Please refer to Appendix G for details.
6 Conclusion
We propose a novel safe exploitation search (SES) algorithm which unifies both safe search and opponent exploitation. With the aid of real-time search, SES make online adaptations changing opponent model. We also prove safety and opponent exploitation guarantees of SES in Theorem and Theorem 4.2 The experimental results in our designed matrix game confirm the
"The parameter a in SES and p in RNR are not directly comparable. To be precise, we should compare the "frontier" of exploitability and exploitation for a, E [0, 1]. For instance, under the same evaluation performance, which algorithm achieves lower exploitability. In table I SES is strictly better.

---

existence of the refined strategy which is both safe and actively exploiting the opponent. In games of poker, our method outperforms NE baselines while keeping exploitability low. SES is also much more efficient than previous safe exploitation algorithms without search. Additionally, SES is more robust opponent modeling errors.
One limitation of SES is that it relies on an estimation of opponent. Although the assumption is common in opponent exploitation literature, efficient opponent modeling still an active research area, Additionally, the exploitation level a is now regarded as hyperparameter in SES. We find that should be tuned for each specific purpose and automatically learning a is not trivial (see Appendix H for a brief discussion). We leave this for future work.
Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers for their insightful discussions and helpful suggestions. This work is supported in part by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (62176135).
References
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):444-489 2016. doi: 10.1038/nature166 1961. URL httpt:ttp//....... org/ 10. 1038/ http://nnturee66666
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 2017.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):004-69) 2020.
Noam Brown, Tuomas Sandholm, and Brandon Amos. Depth-limited solving for imperfect- information games. In Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems, Montréal, Canada, pages 7674-7685, 2018.
Matej Moraveík, Martina Schmid, Neil Burch, Viliam Lisy, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael H. Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356:508-513, 2017.
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456): 885-890, 2019.
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining deep reinforcement learning and search for imperfect-infformaionn games. arXiv preprint arXiv:2007.13544, 2020.
Adam Lerer, Hengyuan Hu. Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games. Proceedings of the AAAI Conference on Artificial Apr. 2020. doi: 10. 1a.11609aaii.v34ee v34i05.6200000000 /ttp/tsss /ttpa///pppp/..a///arttl//e/ee/e http://arrrcceeeeeeeaeee aaai www.orrgi/iooo index.nddxx...m/ AAAI/
Intelligence, 34(05)471877777
Noam Brown and Tuomas Sandholm. Safe and nested subgame solving for imperfecttinfooration games. Proceedings of the s International Conference on Neural Information Processing Systems, pages 689-699, 2017.
Neil Burch, Michael Johanson, and Michael Bowling. Solving imperfect information games using decomposition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 28, 2014.
10

---

Matej Moravcik, Martin Schmid, Karel Ha, Milan Hladik, and Stephen Gaukrodger. Refining subgames in large imperfect information games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Peter McCracken and Michael Bowling. Safe strategies for agent modelling in games. In Artificial Multiagent Learning, Papers from the 2004 AAAI Fall Symposium. Arlington, VA, USA, October 22-24, 2004, volume OS-04-02, pages 103- 1 10. AAAI Press, 2004. URLhttps: https: //www//wwww/. aaai
otg/L/barrrSSypppsaa/////////// 020 0144. php
Michael Johanson, Michael Bowling, and Martin Zinkevich. Computing robust counter-strategies. 2007.
Xun Li and Risto Miikkulainen. Dynamic adaptation and opponent exploitation in computer poker. In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
David Carmel and Shaul Markovitch. Learning models of intelligent agents. In AAAI/AAI,, Vol. I, pages 62-67, 1996.
Darse Billings, Neil Burch, Aaron Davidson, Robert Holte, Jonathan Schaeffer, Terence Schauenberg. and Duane Szafron. Approximating game- theoretc optimal strategies for full-scale poker. In IJCAI, volume 3, page 661, 2003.
Andrew Gilpin and Tuomas Sandholm. A competitive texas hold' em poker player via automated abstraction and real-time equilibrium computation. In AAAI, pages 1007-1013, 2006.
Sam Ganzfried and Tuomas Sandholm. Safe opponent exploitation. ACM Transactions Economics and Computation (TEAC), 3(2):1-28, 2015a.
Finnegan Southey, Michael Bowling, Bryce Larson, Carmelo Piccione, Neil Burch, Darse Billings, and Chris Rayner. Bayes' bluff: opponent modelling in poker. In Proceedings of the Twenty-First Conference on Uncertainty Artificial Intelligence, pages 550-558, 2005.
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In International conference machine learning, pages 802. PMLR, 2019.
Sam Ganzfried and Tuomas Sandholm. Endgame solving in large imperfect-informaane games. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 37-45. Citeseer, 2015b.
Andrew Gilpin and Tuomas Sandholm. Better automated abstraction techniques for imperfect information games, with application to texas hold' mold'em poker. In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, pages 1-8, 2007.
Yuandong Tian, Qucheng Gong, and Tina Jiang. Joint policy search for multi-agent collaboration with imperfect information. arXiv preprint arXiv:2008. 06499, 2020.
Martino Bernasconi-de Luca, Federico Cacciamani, Simone Fioravanti, Nicola Gatti, Alberto March- esi, and Francesco Trovò. Exploiting opponents under utility constraints in sequential games. In Advances in Neural Information Processing Systems, volume 34, pages 13177-13188 2021.
Stefano V. Albrecht and Peter Stone. Autonomous agents modelling other agents: A compre- hensive survey and open problems. Artificial Intelligence, 258:66 95, 2018. ISSN 0004-3702. doi: https://bbl.og//0..006......iti
10.018..00022... URL https: wttp:://wwww.c www.sseiinnneeeet.cccmmm com/
Daphne Koller, Nimrod Megiddo, and Bernhard Von Stengel. Fast algorithms for finding randomized strategies in game trees. In Proceedings of the twenry-sixth annual ACM symposium Theory of computing, pages 750-759, 1994.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. Advances in neural information processing systems, 20: 1729-1736, 2007.
Oskari Tammelin, Neil Burch, Michael Johanson, and Michael Bowling. Solving heads-up limit texas hold'em In founth-fourth international joint conference artificial intelligence, 2015.
11

---

Mare Lanctot, Kevin Waugh, Martin Zinkevich, and Michael H Bowling. Monte carlo sampling for regret minimization in extensive games. NIPS, pages 1078-1086, 2009.
Michael Johanson, Neil Burch, Richard Valenzano, and Michael Bowling. Evaluating state-space abstractions in extensive-fore games. In Proceedings of the 2013 international conference on Autonomous agents and mulli-agent systems, pages 271-278, 2013.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No] or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] The code and licence of the code would be released upon the paper acceptance.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section g
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix[] [[]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs are included the appendix.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See section 5.1 and Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
12

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "XQ8SdZbkIwz", "reviewer": "Reviewer_ByZa", "summary": "This paper introduces a search algorithm for imperfect-information games that balances between opponent exploitation and being unexploitable by adding a parameter to interpolate between the two extremes. The authors present experimental results in a variety of games showing that the algorithm achieves most of the benefits of both, having both low exploitability and is able to exploit effectively.", "questions": "Are there differences between this algorithm and the one used in DeepStack?", "limitations": "The authors have adequately addressed limitations.", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and easy to follow. It places its results well in the context of related search algorithms. Also, the results are convincing.\n\nThe weakness is the novelty. The algorithm essentially just interpolates between two already-known search algorithms. Moreover, I think the algorithm is essentially identical to the one already used in DeepStack (see the \"Opponent Ranges in Resolving\" section in the appendix). The main difference being that the authors propose using a model of the opponent as the policy, whereas DeepStack used its own policy net.\n\nUpdate: Based on the author responses it's true that DeepStack is not the right comparison. However, the technique in the paper does appear to be a rather straightforward interpolation of Maxmargin and Unsafe search. For that reason I am still somewhat concerned about the novelty of the algorithm.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PsvpUXREW7", "reviewer": "Reviewer_Zzn9", "summary": "The authors introduce a novel algorithm, Safe Exploitation Search (SES), that allows trading off safety and opponent exploitability in imperfect information zero-sum games. SES comes with theoretical upper bounds for exploitability, and lower bounds for performance. Unlike previous approaches, SES can be efficiently computed in an online fashion. The authors complement their theoretical investigation by a series of experiments using opponents of varying strength on two different poker variants.\n ", "questions": "* I am mildly confused by your use of $v$, which is defined conditioned on both an infoset AND action, but the action is later frequently omitted (such as in Equation (2)). What happens to the action?\n\n* How does the uncertainty arising from opponent modeling propagate into SES? Can SES be modified to take uncertainty over the opponent model into account?\n\n", "limitations": "The authors do a good job of discussing both limitations of their work, as well potential negative social impact.\n\n# raised score to 7 after rebuttal", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe paper is well-written and organised, the authors' claims and results are clearly presented and, to the best of my assessment, sound. SES is addressing a well-motivated need, and is a useful contribution to the community.\n\n# Weaknesses\n\n* The paper should be spell-checked - there are quite a few minor issues.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6UTmKNQb6j6", "reviewer": "Reviewer_cYqF", "summary": "In zero-sum games, opponents with limited rationality can be exploited if they do not play Nash equilibrium strategy. \nAn exploiter can refine its precomputed blueprint (an approximate NE strategy) in the visited subgames as the game is played online, to take advantage of the opponent's irrational play, using Safe Exploitation Search (SES).\nThe refinement is done by optimizing a function which trades off safety and exploitation, set by a hyperparameter \"alpha\". \nAuthors show safety and opponent exploitation bounds based on this hyperparameter, and uncertainty in modelling the opponent and approximation error of the blueprint or newly found strategy.\nIn practical terms, the player's strategy in a subgame is found using a special gadget game, which is an alpha-combination of max-margin gadget (for safety) and value solving subgame (for opponent exploitation).", "questions": "What is the reason for using precomputed blueprint for P2? Would not the algorithm work with no blueprint strategy? If I understand it correctly, blueprint is used only for theoretical derivations, to ensure strategy is well defined for the whole game (outside of the searched path).\n", "limitations": "Author claim their work does not pose negative societal impacts. Although opponent exploitation in games like Poker can have limited social impact, in general algorithmic exploitation of human irrationality in decision-making under uncertainty can have grave negative social impact and should be properly discussed.\n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n- Explicit bounds on safety and opponent exploitation in terms of alpha.\n- Empirical results on two poker games that show safety and exploitation as function of tradeoff alpha, and with different opponent modelling errors.\n- Comparison with prior algorithm RNR, which is more computationally demanding. SES significantly outperforms RNR in both safety and exploitation.\n- Ablation study of SES, where opponent strategy is fixed in the value-solving subgame -- performs worse than original. (A possible takeaway is that ranges+CBVs are enough for exploitation search).\n\nWeaknesses:\n\n- Authors do not explain how they compute opponent CBVs needed for max-margin gadget. (I suspect they computed these recursively over the whole game tree).\n  Computing opponent CBVs can be non-trivial in large-scale problems, especially when we try to incoporate opponent modelling. This relates to following point: The work does not use depth-limited subgames (with value function evaluations), and cannot be therefore used efficiently to solve large-scale problems. In fact, authors claim L50 \"enables our method’s scalability to large games such as Texas Hold’em\", which they i) do not demonstrate and ii) is probably difficult due to computation of opponent CBVs.\n  I advise this to be mentioned explicitly in the paper and weaken the claim of scalability. (i.e. claim it is future work, and not done by this paper).\n- L7-8: \"SES smoothly interpolates between the two extremes of online strategy refinement\" -- There is no \"smoothness\" proposition in the paper, only  upper and lower bounds on exploitability and safety, respectively. (For a claim of smoothness I would expect something like a proof of Lipschitz continuity with respect to hyperparameter alpha.)\n\n\nOther comments:\n\n- L246-L247: Since σ^S_1 is the best-response to σ^S_2 and the two parts only differ at how to go to each infoset of player 1, player 1 will also keep his strategy the same in both parts. -- This is likely not true. If it was, there is no point in creating copies of the S into S_1 and S_2 in the first place? The top chance node/P1 node can change the expected utilities u(z)*reach_prob_{1,2,c}(z) and hence PL1 NE strategy in the two subgames.\n- L248: I^i_1 unclear which infosets these refer to, are these the top infosets of the subgame S?\n- L249: \"the objective of a Nash Equilibrium of p2\" -- NE are solutions to the max-min optimization. Just maximizing for one player is not enough to find a NE in a game.\n- L250: Maximization is not well defined.\n- Appendix A: Resolve Current Node is not defined.\n- Appendix D: Figure 4: Top/Bottom should be Left/Right.\n- The $\\mathcal{S}$ is used as a disjoint set of subgames S (Thm 4.1), but should it not be the set set of subgames visited in an online play? (I.e from root to terminal).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept