SlowGuess/NeurIPS_2023_OCR huggingface dataset:
no- $\Phi$ -regret algorithm for  $\mathcal{X}' = \Delta(\mathcal{X})$  by sampling  $\mathcal{X} \ni \mathbf{x}$  from any  $\mathbf{x}' \in \mathcal{X}'$  so as to guarantee that  $\mathbb{E}[\mathbf{x}] = \mathbf{x}'$ . Since this is exactly the setting we study in this paper, this folklore observation (see also Farina et al. [2022]) enables us to focus on the following problem: does a deterministic no- $\Phi$ -regret algorithm for the set of sequence-form strategies  $\mathcal{X} = \mathcal{Q}_i$  of any player in an extensive-form game, with guaranteed sublinear  $\Phi$ -regret in the worst case, exist? In this paper we answer the question for the positive.
From regret to equilibrium The setting of Learning in Games refers to the situation in which all players employ a learning algorithm, receiving as loss the negative of the gradient of their own utility evaluated in the strategies output by all the other players. A fascinating aspect of no-  $\Phi$  -regret learning dynamics is that if each player of a game employs a no-  $\Phi$  -regret algorithm, then the empirical frequency of play converges almost surely to the set of  $\Phi$  -equilibria, which are notions of correlated equilibria, in which the rationality of players is bounded by the size of the set  $\Phi$ . Formally, for a set  $\Phi$  of strategy deviations, a  $\Phi$  -equilibrium is defined as follows.
Definition 2.3. For a  $n$ -player extensive-form game  $G$  and a set  $\Phi_i$  of deviations for each player, a  $\{\Phi_i\}$ -equilibrium is a joint distribution  $\mu \in \Delta(\Pi_1 \times \dots \times \Pi_n)$  such that for each player  $i$ , and every deviation  $\phi \in \Phi_i$  it holds that
$$
\mathbb {E} _ {\boldsymbol {x} \sim \mu} \left[ u _ {i} (\boldsymbol {x}) \right] \geq \mathbb {E} _ {\boldsymbol {x} \sim \mu} \left[ u _ {i} \left(\phi \left(\boldsymbol {x} _ {i}\right), \boldsymbol {x} _ {- i}\right) \right]
$$
That is, no player  $i$  has an incentive to unilaterally deviate from the recommended joint strategy  $\mathbf{x}$  using any transformation  $\phi \in \Phi_i$ .
This general framework captures several important notions of equilibrium across a variety of game theoretic models. For example, in both NFGs and EFGs, no-external regret dynamics converge to the set of Coarse Correlated Equilibria. In NFGs, no-swap regret dynamics converge to the set of Correlated Equilibria [Blum and Mansour, 2007]. In EFGs, Farina et al. [2022] recently proved that a specific subset  $\Phi$  of linear transformations called trigger deviations lead to the set of EFCE.
Reducing  $\Phi$ -regret to external regret An elegant construction by Gordon et al. [2008] enables constructing no- $\Phi$ -regret algorithms for a generic set  $\mathcal{X}$  starting from a no-external-regret algorithm for  $\Phi$ . We briefly recall the result.
Theorem 2.4 (Gordon et al. [2008]). Let  $\mathcal{R}$  be an external regret minimizer having the set of transformations  $\Phi$  as its action space, and achieving sublinear external regret  $\mathrm{Reg}^{(T)}$ . Additionally, assume that for all  $\phi \in \Phi$  there exists a fixed point  $\phi(\pmb{x}) = \pmb{x} \in \mathcal{X}$ . Then, a  $\Phi$ -regret minimizer  $\mathcal{R}_{\Phi}$  can be constructed as follows:
- To output a strategy  $\pmb{x}^{(t)}$  at iteration  $t$  of  $\mathcal{R}_{\Phi}$ , obtain an output  $\phi^{(t)} \in \Phi$  of the external regret minimizer  $\mathcal{R}$ , and return one of its fixed points  $\pmb{x}^{(t)} = \phi^{(t)}(\pmb{x}^{(t)})$ .
- For every linear loss function  $\ell^{(t)}$  received by  $\mathcal{R}_{\Phi}$ , construct the linear function  $L^{(t)}: \phi \mapsto \ell^{(t)}(\phi(\boldsymbol{x}^{(t)}))$  and pass it as loss to  $\mathcal{R}$ .
Let  $\Phi\text{-Reg}^{(T)}$  be the  $\Phi$ -regret of  $\mathcal{R}_{\Phi}$ . Under the previous construction, it holds that
$$
\Phi \text {- R e g} ^ {(T)} = \operatorname {R e g} ^ {(T)} \quad \forall T = 1, 2, \dots
$$
Thus, if  $\mathcal{R}$  is an external regret minimizer then  $\mathcal{R}_{\Phi}$  is a  $\Phi$ -regret minimizer.
3 A No-Linear-Swap Regret Algorithm with Polynomial-Time Iterations
In this section, we describe our no-linear-swap-regret algorithm for the set of sequence-form strategies  $\mathcal{Q}$  of a generic player in any perfect-recall imperfect-information extensive-form game. The algorithm follows the general template for constructing  $\Phi$ -regret minimizers given by Gordon et al. [2008] and recalled in Theorem 2.4. For this we need two components:
i) an efficient external regret minimizer for the set  $\mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}$  of all matrices inducing linear transformations from  $\mathcal{Q}$  to  $\bar{\mathcal{Q}}$ ,
ii) an efficiently computable fixed point oracle for matrices  $\mathbf{A} \in \mathcal{M}_{\mathcal{Q} \to \mathcal{Q}}$ , returning  $\pmb{x} = \mathbf{A}\pmb{x} \in \mathcal{Q}$ .
The existence of a fixed point, required in ii), is easy to establish by Brouwer's fixed point theorem, since the polytope of sequence-form strategies is compact and convex, and the continuous function  $\pmb{x} \mapsto \mathbf{A}\pmb{x}$  maps  $\mathcal{Q}$  to itself by definition. Furthermore, as it will become apparent later in the section, all elements  $\mathbf{A} \in \mathcal{M}_{\mathcal{Q} \to \mathcal{Q}}$  have entries in  $[0,1]^{\Sigma \times \Sigma}$ . Hence, requirement ii) can be satisfied directly by solving the linear feasibility program  $\{\text{find } \pmb{x} : \mathbf{A}\pmb{x} = \pmb{x}, \pmb{x} \in \mathcal{Q}\}$  using any of the known polynomial-time algorithms for linear programming. Establishing requirement i) is where the heart of the matter is, and it is the focus of much of the paper. Here, we give intuition for the main insights that contribute to the algorithm. All proofs are deferred to the appendix.
3.1 The Structure of Linear Transformations of Sequence-Form Strategy Polytopes
The crucial step in our construction is to establish a series of results shedding light on the fundamental geometry of the set  $\mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}$  of all linear transformations from a sequence-form polytope  $\mathcal{Q}$  to itself. In fact, our results extend beyond functions from  $\mathcal{Q}$  to  $\mathcal{Q}$  to more general functions from  $\mathcal{Q}$  to a generic compact polytope  $\mathcal{P} \coloneqq \{\pmb{x} \in \mathbb{R}^d : \mathbf{P}\pmb{x} = \pmb{p}, \pmb{x} \geq \pmb{0}\}$  for arbitrary  $\mathbf{P}$  and  $\pmb{p}$ . We establish the following characterization theorem, which shows that when the functions are expressed in matrix form, the set  $\mathcal{M}_{\mathcal{Q}\to \mathcal{P}}$  can be captured by a polynomial number of constraints. The proof is deferred to Appendix B.
Theorem 3.1. Let  $\mathcal{Q}$  be a sequence-form strategy space and let  $\mathcal{P}$  be any bounded polytope of the form  $\mathcal{P} := \{\pmb{x} \in \mathbb{R}^d : \mathbf{P}\pmb{x} = \pmb{p}, \pmb{x} \geq \pmb{0}\} \subseteq [0, \gamma]^d$ , where  $\mathbf{P} \in \mathbb{R}^{k \times d}$ . Then, for any linear function  $f : \mathcal{Q} \rightarrow \mathcal{P}$ , there exists a matrix  $\mathbf{A}$  in the polytope
$$
\mathcal {M} _ {\mathcal {Q} \to \mathcal {P}} := \left\{\mathbf {A} = \left[ \begin{array}{c c c} \dots & \mathbf {A} _ {(\sigma)} & \dots \end{array} \right] \in \mathbb {R} ^ {d \times \Sigma}: \begin{array}{l l l} (3) & \mathbf {P A} _ {(j a)} = \boldsymbol {b} _ {j} & \forall j a \in \Sigma^ {\perp} \\ (4) & \mathbf {A} _ {(\sigma)} = \mathbf {0} & \forall \sigma \in \Sigma \setminus \Sigma^ {\perp} \\ (5) & \sum_ {j ^ {\prime} \in \mathcal {C} _ {\varnothing}} \boldsymbol {b} _ {j ^ {\prime}} = \boldsymbol {p} \\ (6) & \sum_ {j ^ {\prime} \in \mathcal {C} _ {j a}} \boldsymbol {b} _ {j ^ {\prime}} = \boldsymbol {b} _ {j} & \forall j a \in \Sigma \setminus \Sigma^ {\perp} \\ (7) & \mathbf {A} _ {(\sigma)} \in [ 0, \gamma ] ^ {d} & \forall \sigma \in \Sigma \\ (8) & \boldsymbol {b} _ {j} \in \mathbb {R} ^ {k} & \forall j \in \mathcal {J} \end{array} \right\}
$$
such that  $f(\pmb{x}) = \mathbf{A}\pmb{x}$  for all  $\pmb{x} \in \mathcal{Q}$ . Conversely, any  $\mathbf{A} \in \mathcal{M}_{\mathcal{Q} \to \mathcal{P}}$  defines a linear function  $\pmb{x} \mapsto \mathbf{A}\pmb{x}$  from  $\mathcal{Q}$  to  $\mathcal{P}$ , that is, such that  $\mathbf{A}\pmb{x} \in \mathcal{P}$  for all  $\pmb{x} \in \mathcal{Q}$ .
The proof operates by induction in several steps. At its core, it exploits the combinatorial structure of sequence-form strategy polytopes, which can be decomposed into sub-problems using a series of Cartesian products and convex hulls. A high-level description of the induction is as follows:
- The Base Case corresponds to the case of being at a leaf decision point. In this case, the set of deviations corresponds to all linear transformations from a probability  $n$ -simplex into a given polytope  $\mathcal{P}$ . This set is equivalent to all  $d \times n$  matrices whose columns are points in  $\mathcal{P}$ , which can easily be verified formally. This corresponds to constraint (3) in the characterization of  $\mathcal{M}_{\mathcal{Q} \rightarrow \mathcal{P}}$ .
- For the Inductive Step, we are at an intermediate decision point  $j$ , that is, one for which at least one action leads to further decision points.
- In Lemma B.8 we show that any terminal action  $a$  at  $j$  leads to a column in the transformation matrix that is necessarily a valid point in the polytope  $\mathcal{P}$ . This is similar to the base case, and again leads to constraint (3) in the characterization.
- In Lemma B.7, we look at the other case of a non-terminal action  $a$  at  $j$ . We prove that there always exists an equivalent transformation matrix whose column corresponding to sequence  $ja$  is identically 0 (constraint (4)). This allows for the "crux" of the transformation to happen in the subtrees below  $ja$  or equivalently, the subtrees rooted at the children decision points  $\mathcal{C}_{ja}$  of  $ja$ .
- A key difficulty to conclude the proof is in using the assumption of the inductive step to characterize all such valid transformations. The set of strategies in the subtrees rooted at the children decision points  $\mathcal{C}_{ja}$  is in general the Cartesian product of the strategies in these subtrees. This explains the need for the fairly technical Proposition B.4, whose goal is to precisely characterize valid transformations of Cartesian products. This leads to constraints (5) and (6) in our final characterization.
We also remark that while the theorem calls for the polytope  $\mathcal{P}$  to be in the form  $\mathcal{P} = \{\pmb{x} \in \mathbb{R}^d : \mathbf{P}\pmb{x} = \pmb{p}, \pmb{x} \geq \pmb{0}\}$ , with little work the result can also be extended to handle other representations such as  $\{\pmb{x} \in \mathbb{R}^d : \mathbf{P}\pmb{x} \leq \pmb{p}\}$ . We opted for the form specified in the theorem since it most directly leads to the proof, and since the constraints that define the sequence-form strategy polytope (Definition 2.2) are already in the form of the statement.
In particular, by setting  $\mathcal{P} = \mathcal{Q}$  in Theorem 3.1 (in this case, the dimensions of  $\mathbf{P}$  will be  $k = |\mathcal{I}| + 1$ , and  $d = |\Sigma|$ ), we conclude that the set of linear functions from  $\mathcal{Q}$  to itself is a compact and convex polytope  $\mathcal{M}_{\mathcal{Q} \rightarrow \mathcal{Q}} \subseteq [0,1]^{\Sigma \times \Sigma}$ , defined by  $O(|\Sigma|^2)$  linear constraints. As discussed, this polynomial characterization of  $\mathcal{M}_{\mathcal{Q} \rightarrow \mathcal{Q}}$  is the fundamental insight that enables polynomial-time minimization of linear-swap regret in general extensive-form games.
3.2 Our No-Linear-Swap Regret Algorithm
From here, constructing a no-external-regret algorithm for  $\mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}$  is relatively straightforward, using standard tools from the rich literature of online learning. For example, in Algorithm 1, we propose a solution employing online projected gradient descent [Gordon, 1999, Zinkevich, 2003].
Algorithm 1:  $\Phi$ -Regret minimizer for the set  $\Phi = \mathcal{M}_{\mathcal{Q} \to \mathcal{Q}}$
Data:  $\mathbf{A}^{(1)}\in \mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}$  and fixed point  $\pmb{x}^{(1)}$  of  $\mathbf{A}^{(1)}$ , learning rates  $\eta^{(t)} > 0$   
for  $t = 1,2,\ldots$  do  
Output  $\pmb{x}^{(t)}$   
Receive  $\ell^{(t)}$  and pay  $\langle \ell^{(t)},\pmb{x}^{(t)}\rangle$   
Set  $\mathbf{L}^{(t)} = \ell^{(t)}(\pmb{x}^{(t)})^{\top}$ $\mathbf{A}^{(t + 1)} = \Pi_{\mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}}(\mathbf{A}^{(t)} - \eta^{(t)}\mathbf{L}^{(t)}) = \arg \min_{\mathbf{Y}\in \mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}}\| \mathbf{A}^{(t)} - \eta^{(t)}\mathbf{L}^{(t)} - \mathbf{Y}\| _F^2$   
Compute a fixed point  $\pmb{x}^{(t + 1)} = \mathbf{A}^{(t + 1)}\pmb{x}^{(t + 1)}\in \mathcal{Q}$  of matrix  $\mathbf{A}^{(t + 1)}$
Combining that no-external-regret algorithm for  $\Phi$  with the construction by Gordon et al. [2008], we can then establish the following linear-swap regret and iteration complexity bounds for Algorithm 1.
Theorem 3.2 (Informal). Let  $\Sigma$  denote the set of sequences of the learning player in the extensive-form game, and let  $\eta^{(t)} = 1 / \sqrt{t}$  for all  $t$ . Then, for any sequence of loss vectors  $\ell^{(t)} \in [0,1]^{\Sigma}$ , Algorithm 1 guarantees linear-swap regret  $O(|\Sigma|^2\sqrt{T})$  after any number  $T$  of iterations, and runs in  $O(\mathrm{poly}(|\Sigma|)\log^2 t)$  time for each iteration  $t$ .
The formal version of the theorem is given in Theorem D.1. It is worth noting that the polynomial-sized description of  $\mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}$  is crucial in establishing the polynomial running time of the algorithm, both in the projection step (5) and in the fixed point computation step (6). We also remark that the choice of online projected gradient descent combined with the ellipsoid method for projections were arbitrary and the useful properties of  $\mathcal{M}_{\mathcal{Q}\rightarrow \mathcal{Q}}$  are retained when using it with any efficient regret minimizer.
4 Linear-Deviation Correlated Equilibrium
As we discussed in the preliminaries, when all players in a game employ no- $\Phi$ -regret learning algorithms, then the empirical frequency of play converges to the set of  $\Phi$ -equilibria almost surely. Similarly, when  $\Phi = \mathcal{M}_{\mathcal{Q} \to \mathcal{Q}}$  the players act based on "no-linear-swap regret" dynamics and converge to a notion of  $\Phi$ -equilibrium we call linear-deviation correlated equilibrium (LCE). In this section we present some notable properties of the LCE. In particular, we discuss its relation to other already established equilibria, as well as the computational tractability of optimal equilibrium selection.
4.1 Relation to CE and EFCE
The  $\Phi$ -regret minimization framework, offers a natural way to build a hierarchy of the corresponding  $\Phi$ -equilibria based on the relationship of the  $\Phi$  sets of deviations. In particular, if for the sets  $\Phi_1, \Phi_2$  it holds that  $\Phi_1 \subseteq \Phi_2$ , then the set of  $\Phi_2$ -equilibria is a subset of the set of  $\Phi_1$ -equilibria. Since
the Correlated Equilibrium is defined using the set of all swap deviations, we conclude that any  $\Phi$ -equilibrium, including the LCE, is a superset of CE. What is the relationship then of LCE with the extensive-form correlated equilibrium (EFCE)? Farina et al. [2022] showed that the set  $\Phi^{\mathrm{EFCE}}$  inducing EFCE is the set of all "trigger deviations", which can be expressed as linear transformations of extensive-form strategies. Consequently, the set  $\Phi^{\mathrm{EFCE}}$  is a subset of all linear transformations and thus, it holds that  $\mathrm{CE} \subseteq \mathrm{LCE} \subseteq \mathrm{EFCE}$ . In examples E.1 and E.3 of the appendix we show that there exist specific games in which either  $\mathrm{CE} \neq \mathrm{LCE}$ , or  $\mathrm{LCE} \neq \mathrm{EFCE}$ . Hence, we conclude that the previous inclusions are strict and it holds  $\mathrm{CE} \subsetneq \mathrm{LCE} \subsetneq \mathrm{EFCE}$ .
For Example E.1 we use a signaling game from von Stengel and Forges [2008] with a known EFCE and we identify a linear transformation that is not captured by the trigger deviations of EFCE. Specifically, it is possible to perform linear transformations on sequences of a subtree based on the strategies on other subtrees of the TFSDP. For Example E.3 we have found a specific game through computational search that has a LCE which is not a normal-form correlated equilibrium. To do that we identify a particular normal-form swap that is non-linear.
Empirical evaluation To further illustrate the separation between no-linear-swap-regret dynamics and no-trigger-regret dynamics, used for EFCE, we provide experimental evidence that minimizing linear-swap-regret also minimizes trigger-regret (Figure 2, left), while minimizing trigger-regret does not minimize linear-swap regret. Specifically, in Figure 2 we compare our no-linear-swap-regret learning dynamics (given in Algorithm 1) to the no-trigger-regret algorithm introduced by Farina et al. [2022]. More details about the implementation of the algorithms is available in Appendix F. In the left plot, we measure on the y-axis the average trigger regret incurred when all players use one or the other dynamics. Since trigger deviations are special cases of linear deviations, as expected, we observe that both dynamics are able to minimize trigger regret. Conversely, in the right plot of Figure 2, the y-axis measures linear-swap-regret. We observe that while our dynamics validate the sublinear regret performance proven in Theorem 3.2, the no-trigger-regret dynamics of Farina et al. [2022] exhibit an erratic behavior that is hardly compatible with a vanishing average regret. This suggests that no-linear-swap-regret is indeed a strictly stronger notion of hindsight rationality.
Figure 2: (Left) Average trigger regret per iteration for both a linear-swap-regret minimizer and a trigger-regret minimizer. (Right) Average linear-swap regret per iteration for the same two minimizers.
4.2 Hardness of Maximizing Social Welfare
In many cases we are interested in knowing whether it is possible to select an Equilibrium with maximum Social Welfare. Let MAXPAY-LCE be the problem of finding an LCE in EFGs that maximizes the sum (or any linear combination) of all player's utilities. Below, we prove that we cannot efficiently solve MAXPAY-LCE, unless  $\mathrm{P = NP}$ , even for 2 players if chance moves are allowed, and even for 3 players otherwise. We follow the structure of the same hardness proof for the problem MAXPAY-CE of finding an optimal CE in EFGs. Specifically, von Stengel and Forges [2008] use a reduction from SAT to prove that deciding whether MAXPAY-CE can attain the maximum value is NP-hard even for 2 players. To do that, they devise a way to map any SAT instance into a polynomially large game tree in which the root is the chance player, the second level corresponds to one player, and
the third level corresponds to the other player. The utilities for both players are exactly the same, thus the players will have to coordinate to maximize their payoff irrespective of the linear combination of utilities we aim to maximize.
Theorem 4.1. For two-player, perfect-recall extensive-form games with chance moves, the problem MAXPAY-LCE is not solvable in polynomial time, unless  $P = NP$ .
Remark 4.2. The problem retains its hardness if we remove the chance node and add a third player instead. As showed in von Stengel and Forges [2008], in that case we can always build a polynomially-sized game tree that forces the third player to act as a chance node.
In this paper we have shown the existence of uncoupled no-linear-swap regret dynamics with polynomial-time iteration complexity in the game tree size in any extensive-form game. This significantly extends prior results related to extensive-form correlated equilibria, and begets learning agents that learn not to regret a significantly larger set of strategy transformations than what was known to be possible before. A crucial technical contribution we made to establish our result, and which might be of independent interest, is providing a polynomial characterization of the set of all linear transformations from a sequence-form strategy polytope to itself. Specifically, we showed that such a set of transformations can be expressed as a convex polytope with a polynomial number of linear constraints, by leveraging the rich combinatorial structure of the sequence-form strategies. Moreover, these no-linear-swap regret dynamics converge to linear-deviation correlated equilibria in extensive-form games, which are a novel type of equilibria that lies strictly between normal-form and extensive-form correlated equilibria.
These new results leave open a few interesting future research directions. Even though we know that there exist polynomial-time uncoupled dynamics converging to linear-deviation correlated equilibrium, we conjecture that it is also possible to obtain an efficient centralized algorithm similar to the Ellipsoid Against Hope for computing EFCE in extensive-form games by Huang and von Stengel [2008]. Additionally, it is an intriguing question to understand whether a no-linear-swap regret algorithm exists that achieves  $O(\log T)$  regret per-player, as is the case for no-trigger regret [Anagnostides et al., 2023]. Furthermore, it would be interesting to further explore problems of equilibrium selection related to LCE, possibly by devising suitable Fixed-Parameter Algorithms in the spirit of Zhang et al. [2022]. Finally, the problem of understanding what is the most hindsight rational type of deviations based on which we can construct efficient regret minimizers in extensive-form games remains a major open question.
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):pp. 508-513, 2017.
Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418-424, 2018.
Noam Brown and Tuomas Sandholm. Superhuman AI for multiplayer poker. Science, 365(6456): 885-890, 2019. doi: 10.1126/science.aay2400.
Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T. Connor, Neil Burch, Thomas Anthony, Stephen McAleer, Romuald Elie, Sarah H. Cen, Zhe Wang, Audrunas Gruslys, Aleksandra Malysheva, Mina Khan, Sherjil Ozair, Finbarr Timbers, Toby Pohlen, Tom Eccles, Mark Rowland, Marc Lanctot, Jean-Baptiste Lespiau, Bilal Piot, Shayegan Omidshafiei, Edward Lockhart, Laurent Sifre, Nathalie Beauguerlange, Remi Munos, David Silver, Satinder Singh, Demis Hassabis, and Karl Tuyls. Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, 378(6623):990-996, 2022.
Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexander H Miller, and Noam Brown. Mastering the game of no-press Diplomacy via human-regularized reinforcement learning and planning. In International Conference on Learning Representations (ICLR), 2023.