SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, in our previous analysis we assumed that it is possible to compute an exact solution  $\mathbf{A}^{(t + 1)} = \Pi_{\mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}}\bigl (\mathbf{A}^{(t)} - \eta^{(t)}\mathbf{L}^{(t)}\bigr)$  of the projection step, which is an instance of convex quadratic programming, meaning that we can only get an approximate solution [Vishnoi, 2021]. In the following Lemma we prove that an  $\epsilon$ -approximate projection does not affect the regret for a single iteration of
the algorithm, if  $\epsilon$  is sufficiently small. The inequality we prove is similar to the one in Lemma 2.12 from Orabona [2022].
Lemma D.2. Let  $\mathbf{Y}^{*} = \Pi_{\mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}}\bigl (\mathbf{A}^{(t)} - \eta^{(t)}\mathbf{L}^{(t)}\bigr)$  and suppose that  $\mathbf{A}^{(t + 1)}\in \mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}$  is such that  $\| \mathbf{A}^{(t + 1)} - \mathbf{Y}^*\| _F^2\leq \epsilon^{(t)}$ , then for any  $\mathbf{X}\in \mathcal{M}_{\mathcal{Q}\to \mathcal{Q}}$  it holds
$$
\langle \mathbf {L} ^ {(t)}, \mathbf {X} - \mathbf {A} ^ {(t)} \rangle_ {F} \leq \frac {1}{2 \eta^ {(t)}} \| \mathbf {A} ^ {(t)} - \mathbf {X} \| _ {F} ^ {2} - \frac {1}{2 \eta^ {(t)}} \| \mathbf {A} ^ {(t + 1)} - \mathbf {X} \| _ {F} ^ {2} + \frac {\eta^ {(t)}}{2} \| \mathbf {L} ^ {(t)} \| _ {F} ^ {2} + \frac {D}{\eta^ {(t)}} \frac {1}{\epsilon^ {(t)}} \| \mathbf {X} \| _ {F} ^ {2}.
$$
Proof. From Lemma 2.12 of Orabona [2022] we know that
$$
\langle \mathbf {L} ^ {(t)}, \mathbf {X} - \mathbf {A} ^ {(t)} \rangle_ {F} \leq \frac {1}{2 \eta^ {(t)}} \| \mathbf {A} ^ {(t)} - \mathbf {X} \| _ {F} ^ {2} - \frac {1}{2 \eta^ {(t)}} \| \mathbf {Y} ^ {*} - \mathbf {X} \| _ {F} ^ {2} + \frac {\eta^ {(t)}}{2} \| \mathbf {L} ^ {(t)} \| _ {F} ^ {2}.
$$
Additionally, it holds that
$$
\begin{array}{l} \| \mathbf {A} ^ {(t + 1)} - \mathbf {X} \| _ {F} ^ {2} = \| \mathbf {A} ^ {(t + 1)} - \mathbf {Y} ^ {*} + \mathbf {Y} ^ {*} - \mathbf {X} \| _ {F} ^ {2} \\ = \| \mathbf {A} ^ {(t + 1)} - \mathbf {Y} ^ {*} \| _ {F} ^ {2} + \| \mathbf {Y} ^ {*} - \mathbf {X} \| _ {F} ^ {2} + 2 \langle \mathbf {A} ^ {(t + 1)} - \mathbf {Y} ^ {*}, \mathbf {Y} ^ {*} - \mathbf {X} \rangle_ {F} \\ \leq \| \mathbf {Y} ^ {*} - \mathbf {X} \| _ {F} ^ {2} + 2 \langle \mathbf {A} ^ {(t + 1)} - \mathbf {Y} ^ {*}, \mathbf {A} ^ {(t + 1)} - \mathbf {Y} ^ {*} + \mathbf {Y} ^ {*} - \mathbf {X} \rangle_ {F} \\ = \| \mathbf {Y} ^ {*} - \mathbf {X} \| _ {F} ^ {2} + 2 \langle \mathbf {A} ^ {(t + 1)} - \mathbf {Y} ^ {*}, \mathbf {A} ^ {(t + 1)} - \mathbf {X} \rangle_ {F} \\ \leq \| \mathbf {Y} ^ {*} - \mathbf {X} \| _ {F} ^ {2} + 2 \| \mathbf {A} ^ {(t + 1)} - \mathbf {Y} ^ {*} \| _ {F} \| \mathbf {A} ^ {(t + 1)} - \mathbf {X} \| _ {F} \\ \leq \left\| \mathbf {Y} ^ {*} - \mathbf {X} \right\| _ {F} ^ {2} + 2 D \epsilon^ {(t)}. \\ \end{array}
$$
The Lemma follows by combining the previous two inequalities.
If we set  $\epsilon^{(t)} = 1 / t^{5 / 2}$  then for our choice of  $\eta^{(t)} = \frac{D}{L\sqrt{t}}$ , the error term becomes  $L / t^2$ . Summing over  $T$  timesteps we thus get an additive error of  $O(L) = O(|\Sigma|)$  in the regret, and our total linear-swap regret bound remains  $O(|\Sigma|^2\sqrt{T})$ .
We now move to analyzing the per-iteration time complexity of Algorithm 1. The most computationally heavy steps are (5) and (6). To compute the fixed point at step (6) we can use any polynomial-time LP algorithm. We can also perform the projection step (5) in polynomial time using the ellipsoid method [Vishnoi, 2021]. For this we reduce it to a suitable Semidefinite Program with  $O(|\Sigma|^2)$  variables and use the Cholesky factorization [Gartner and Matousek, 2014] as the separation oracle, thus responding to separation queries in  $O(|\Sigma|^6)$  time. Note that it is possible to guarantee  $\| \mathbf{A}^{(t + 1)} - \mathbf{Y}^* \|_F^2 \leq \epsilon^{(t)}$  as the ellipsoid method can output a point  $\mathbf{A}^{(t + 1)} \in \mathcal{M}_{\mathcal{Q} \to \mathcal{Q}}$  such that  $\| \mathbf{A}^{(t)} - \eta^{(t)}\mathbf{L}^{(t)} - \mathbf{A}^{(t + 1)} \|_F \leq \| \mathbf{A}^{(t)} - \eta^{(t)}\mathbf{L}^{(t)} - \mathbf{Y}^* \|_F + \epsilon$  and furthermore, the Frobenius norm is a 2-strongly convex function.
To apply the ellipsoid method we further need bounds on the Frobenius norm of  $\mathbf{Y} \in \mathcal{M}_{\mathcal{Q} \to \mathcal{Q}}$  and the maximum projection distance. For the norm of  $\mathbf{Y}$ , we pick an upper bound of  $R = D = |\Sigma|$ . For the lower bound  $r$  we note that  $\mathbf{Y}x \in \mathcal{Q}$  for all  $x \in \mathcal{Q}$ , which implies  $\mathbf{Y}[\varnothing]\mathbf{x} = 1 \Rightarrow \| \mathbf{Y}[\varnothing]\|_1 \geq 1$ . Thus we get  $r \geq \frac{1}{|\Sigma|}$ . Similarly, we bound the projection distance between 0 and  $D$ . Based on these bounds and on Theorem 13.1 from Vishnoi [2021] we conclude that the total per-iteration time complexity of Algorithm 1 is  $O\left(|\Sigma|^{10}\log (|\Sigma|)\log^2 (t)\right)$ .
Proof. We use the exact same argument employed in the paper by von Stengel and Forges [2008] by reducing SAT to the MAXPAY-LCE problem. Specifically, for each instance of SAT having  $n$  clauses and  $m$  variables we construct a two-player extensive-form game of size polynomial in  $n$  and  $m$ . In the beginning, there is a chance move that picks one of  $n$  possible actions uniformly at random - one for each SAT clause. Then is the turn of Player 2 who has  $n$  distinct singleton information sets corresponding to the chance node actions, and respectively to the  $n$  clauses of the SAT instance. Let  $L_{i}$  be the set of literals (negated or non-negated variables) included in the  $i$ -th clause of the SAT
instance. In information set  $i$  of Player 2 there exist  $|L_i|$  actions, one for each literal in  $L_i$ . Finally, each literal leads to a different decision node for Player 1 who has as many decision nodes as the number of literals in the SAT instance, and in each node there exist exactly 2 possible actions: TRUE and FALSE. However, Player 1 only has  $m$  information sets corresponding to the  $m$  SAT variables with each information set  $x$  grouping together all nodes corresponding to literals of the variable  $x$ . This way, Player 1 only chooses the truth value of a variable without knowing from which literal Player 2 has picked this variable. The utilities for both players are equal to 1 if the truth value picked by Player 1 satisfies the literal picked by Player 2, and both utilities are 0 otherwise.
In this game, there exists a pure strategy attaining payoff 1 for each player if and only if the SAT instance is satisfiable – namely Player 1 always acts based on the satisfying assignment and Player 2 picks for every clause a literal that is known to be TRUE. Otherwise, the maximum payoff for each pure strategy is at most  $1 - 1/n$ . Given that a LCE describes a convex combination of pure strategies, it follows that the maximum total expected payoff of the sum of the two players' utilities will either be 2 when the SAT instance is satisfiable, or it will be at most  $2(1 - 1/n)$  when the instance is not satisfiable. Additionally, this holds not just for the sum but for any linear combination of player utilities. Thus, the problem of deciding whether MAXPAY-LCE can attain a value of at least  $k$  is NP-hard for any linear combination of utilities.
E Examples
Example E.1 (EFCE  $\neq$  LCE). Consider the following 2-player signaling game presented by von Stengel and Forges [2008]
Below are the normal-form payoff matrix of the signaling game (left) and the extensive-form correlated equilibrium (right) given by von Stengel and Forges [2008]:
1 2	lXlY	lXrY	rXlY	rXrY
XGXB	5,5	5,5	0,6	0,6
XGYB	5,5	2,8	3,3	0,6
YGXB	5,5	3,3	2,8	0,6
YGYB	5,5	0,6	5,5	0,6
lXLY	lXrY	rXLY	rXrY
XGXb	0	1/4	0	0
XGYB	0	1/4	0	0
YGXB	0	0	1/4	0
YGYB	0	0	1/4	0
However, we can observe that this EFCE is not a linear-deviation correlated equilibrium because, for example, Player 1 can increase their payoff by a value of  $3/2$  using the following transformation:
$$
X _ {G} X _ {B} \mapsto X _ {G} X _ {B}
$$
i.e., map the reduced-normal-form plan  $(1,0,1,0) \mapsto (1,0,1,0)$
$$
X _ {G} Y _ {B} \mapsto X _ {G} X _ {B}
$$
i.e., map the reduced-normal-form plan  $(1,0,0,1)\mapsto (1,0,1,0)$
$$
Y _ {G} X _ {B} \mapsto Y _ {G} Y _ {B}
$$
i.e., map the reduced-normal-form plan  $(0,1,1,0) \mapsto (0,1,0,1)$
$$
Y _ {G} Y _ {B} \mapsto Y _ {G} Y _ {B}
$$
i.e., map the reduced-normal-form plan  $(0,1,0,1) \mapsto (0,1,0,1)$
(Above, we have implicitly assumed that the strategy vectors encode probability of actions in the arbitrary order  $X_{G}, Y_{G}, X_{B}, Y_{B}$ ). The above transformation is linear, since it can be represented via the matrix
$$
\left( \begin{array}{c c c c} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \end{array} \right).
$$
This would swap the pure strategy  $X_{G}Y_{B}$  with  $X_{G}X_{B}$  and strategy  $Y_{G}X_{B}$  with  $Y_{G}Y_{B}$ . Crucially, the strategy at the subtree of information set  $B$  is determined by the strategy at the subtree of information set  $G$ . Thus, the transformed value for each sequence does not purely depend on the ancestors of that sequence, but can also depend on strategies belonging to "sibling" subtrees. Hence, this example proves that  $LCE \neq EFCE$ .
In fact, we can even verify that in this toy example the set of LCE coincides with the set of all normal-form CE. We refer the interested reader to von Stengel and Forges [2008] for a more detailed discussion of the different behaviors that players can exhibit at an EFCE vs a normal-form CE, which in this specific example also corresponds to a comparison between the EFCE and the LCE.
Remark E.2. The linear transformation given in Example E.1 also serves to show that the Behavioral Deviations defined in Morrill et al. [2021] are not a superset of all linear transformations. Additionally, behavioral deviations are not a subset of linear transformations, as the latter act only on reduced strategies. Thus the two sets of deviations are incomparable.
Example E.3 (LCE  $\neq$  CE). This example was found through computational search. Consider the 2-player game with the following game tree:
Based on the previous game tree, the normal-form payoff matrix of the game is shown below.
1 2	QlWl	QlWr	QrWl	QrWr
A1B1	50.5, 0.0	50.5, 0.25	-27.75, -78.75	-27.75, -78.5
A1B2	0.0, 0.0	0.5, 1.0	-0.75, 0.0	-0.25, 1.0
A2B1	50.0, 0.5	49.5, -0.75	-27.0, -78.25	-27.5, -79.5
A2B2	-0.5, 0.5	-0.5, 0.0	0.0, 0.5	0.0, 0.0
We can now verify that the following is a linear-deviation correlated equilibrium for this game. The verification can be done computationally by expressing all constraints of Theorem 3.1 as a Linear Program.
QlWl	QlWr	QrWr	QrWr
A1B1	1/5	0	0	0
A1B2	0	1/5	0	0
A2B1	1/5	0	0	0
A2B2	0	0	1/5	1/5
However, the swap  $\{A_1B_2\mapsto A_1B_1,A_2B_1\mapsto A_1B_1\}$  can increase player 1's payoff by 50.5.
Furthermore, we can verify that this swap is not linear as follows. First assume that it was linear and could be written as a matrix  $\mathbf{A} \in [0,1]^{\Sigma \times \Sigma}$ . Then the matrix has to be consistent with the following transformations:
$$
A _ {1} B _ {1} \mapsto A _ {1} B _ {1}
$$
$$
A _ {1} B _ {2} \mapsto A _ {1} B _ {1}
$$
$$
A _ {2} B _ {1} \mapsto A _ {1} B _ {1}
$$
$$
A _ {2} B _ {2} \mapsto A _ {2} B _ {2}
$$
For convenience of referring to matrix rows and columns we number the four sequences as:
$$
A _ {1}: 0, \quad A _ {2}: 1, \quad B _ {1}: 2, \quad B _ {2}: 3
$$
If we focus on the second transformation,  $A_{1}B_{2} \mapsto A_{1}B_{1}$ , we conclude that  $\mathbf{A}[:,0] + \mathbf{A}[:,3] = (1,0,1,0)^{\top}$  which implies that  $\mathbf{A}[1,0] = \mathbf{A}[3,0] = \mathbf{A}[1,3] = \mathbf{A}[3,3] = 0$ . Now, if we subtract the respective equations of the last two swaps we get  $\mathbf{A}[:,2] - \mathbf{A}[:,3] = (1,-1,1,-1)^{\top}$ . Since  $\mathbf{A} \in [0,1]^{\Sigma \times \Sigma}$ , the last equation implies  $\mathbf{A}[1,3] = 1$  which contradicts the previous constraint of  $\mathbf{A}[1,3] = 0$ . Thus, we have found a valid normal-form swap that cannot be expressed as a linear transformation of sequence-form strategies. Hence, this example proves that  $LCE \neq CE$ .
F Details on Empirical Evaluation
In this section we provide details about the implementation of our algorithm, as well as the compute resources and game instances used.
Implementation of our no-linear-swap-regret dynamics We implemented our no-linear-swap-regret algorithm (Algorithm 1) in the  $\mathrm{C + + }$  programming language using the Gurobi commercial optimization solver [Gurobi Optimization, LLC, 2023], version 10. We use Gurobi for the following purposes.
- To compute the projection needed on Line 5 of Algorithm 1. We remark that while Gurobi is typically recognized as a linear and integer linear programming solver, modern versions include tuned code for convex quadratic programming. In particular, we used the barrier algorithm to compute the Euclidean projections onto the polytope  $\mathcal{M}_{\mathcal{Q}\rightarrow \mathcal{Q}}$  required at every iteration of our algorithm.
- To compute the fixed points of the matrices  $\mathbf{A} \in \mathcal{M}_{\mathcal{Q} \to \mathcal{Q}}$ , that is, finding  $\mathcal{Q} \ni \boldsymbol{x} = \mathbf{A}\boldsymbol{x}$ . As discussed in Section 3 this is a polynomially-sized linear program.
- To measure the linear-swap regret incurred after any  $T$  iterations, which is plotted on the y-axes of Figure 2. This corresponds to solving the linear optimization problem
$$
\max  _ {\mathbf {A} \in \mathcal {M} _ {\mathcal {Q} \rightarrow \mathcal {Q}}} \left\{\frac {1}{T} \sum_ {t = 1} ^ {T} \langle \boldsymbol {\ell} ^ {(t)}, \boldsymbol {x} ^ {(t)} - \mathbf {A} \boldsymbol {x} ^ {(t)} \rangle \right\}.
$$
We did very minimal tuning of the constant learning rate  $\eta$  used for online projected gradient descent, trying values  $\eta \in \{0.05, 0.1, 0.5\}$  (we remark that a constant value of  $\eta \approx 1 / \sqrt{T}$  is theoretically sound). We found that  $\eta = 0.1$ , which is used in the plots of Figure 2, performed best.
Implementation of no-trigger-regret dynamics We implemented the no-trigger-regret algorithm of Farina et al. [2022] in the  $\mathrm{C + + }$  programming language. In this case, there is no need to use Gurobi, since, as the original authors show, the polytope of trigger deviation functions admits a convenient combinatorial characterization that enables us to sidestep linear programming. Rather, we implemented the algorithm and the computation of the trigger regret directly leveraging the combinatorial structure.
Computational resources used Minimal computational resources were used. All code ran on a personal laptop for roughly 12 hours.
Game instance used We ran our code on the standard benchmark game of Kuhn poker Kuhn [1950]. We used a three-player variant of the game. Compared to the original game, which only considers a simplified deck make of cards out of only three possible ranks (Jack, Queen, or Kind), we use a full deck of 13 possible card ranks. The game has 156 information sets, 315 sequences, and 22308 terminal states.
G Further Remarks on the Reduction from No-  $\Phi$  -Regret to External Regret
A no- $\Phi$ -regret algorithm is typically defined as outputting deterministic behavior from a finite set  $\mathcal{X}$  (for example, deterministic reduced-normal-form plans in extensive-form games, or actions in normal-form games), which can be potentially sampled at random. This is the setting used by, for example, Hart and Mas-Colell [2000] and Farina et al. [2022]. However, we remark that when the transformations  $\phi \in \Phi$  are linear, any such device can be constructed starting from an algorithm that outputs points  $x' \in \mathcal{X}' := \mathrm{conv}(\mathcal{X})$ , and then sampling  $x$  unbiasedly in accordance with  $x'$ , that is, so that  $\mathbb{E}[x] = x'$ . The reason why this is useful is that constructing the latter object is usually simpler, as  $\mathcal{X}'$  is a closed and convex set, and is therefore amenable to the wide array of online optimization techniques that have been developed over the years.
More formally, let  $\Phi$  be a set of linear transformations that map  $\mathcal{X}$  to itself. A no- $\Phi$ -regret algorithm for  $\mathcal{X}' = \mathrm{conv}(\mathcal{X})$  guarantees that, no matter the sequence of loss vectors  $\ell^{(t)}$ ,
$$
R ^ {\prime (T)} = \max  _ {\phi \in \Phi} \sum_ {t = 1} ^ {T} \langle \boldsymbol {\ell} ^ {(t)}, \boldsymbol {x} ^ {\prime (t)} - \phi (\boldsymbol {x} ^ {\prime (t)}) \rangle
$$
grows sublinearly. Consider now an algorithm that, after receiving  $\pmb{x}^{\prime(t)} \in \mathrm{conv}(\mathcal{X})$ , samples  $\pmb{x}^{(t)} \in \mathcal{X}$  unbiasedly, that is, so that  $\mathbb{E}[\pmb{x}^{(t)}] = \pmb{x}^{\prime(t)}$ . Then, by linearity of the transformations and
using the Azuma-Hoeffind concentration inequality, we obtain that for any  $\epsilon > 0$ ,
$$
\mathbb {P} \left[ \left| R ^ {\prime (T)} - \max  _ {\phi \in \Phi} \sum_ {t = 1} ^ {T} \langle \boldsymbol {\ell} ^ {(t)}, \boldsymbol {x} ^ {(t)} - \phi (\boldsymbol {x} ^ {(t)}) \rangle \right| \leq \Theta \left(\sqrt {T \log \frac {1}{\epsilon}}\right) \right] \geq 1 - \epsilon ,
$$
where the big-theta notation hides constants that depend on the payoff range of the game and the diameter of  $\mathcal{X}$  (a polynomial quantity in the game tree size). This shows that as long as the regret of the no- $\Phi$ -algorithm that operates over  $\mathcal{X}'$  is sublinear, then so is that of an algorithm that outputs points on  $\mathcal{X}$  by sampling unbiasedly from  $\mathcal{X}$ . We refer the interested reader to Section 4.2 ("From Deterministic to Mixed Strategies") of Farina et al. [2022].