SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {d i s t} _ {1} \left(X _ {t}, \mathcal {S}\right) \leq \sum_ {i \in \mathcal {N}} \sum_ {\alpha_ {i} \in \mathcal {A} _ {i} \backslash \mathcal {C} _ {i}} \varphi \left(c _ {2} - c _ {1} \tau_ {t}\right), \tag {D.30}
$$
As for the rate estimates of Corollary 2, the proof boils down to a simple derivation of the corresponding rate functions:
Proof of Corollary 2. By a straightforward calculation, we have:
1. If  $\theta(z) = z \log z$  then  $\varphi(z) = \exp(1 + z)$ .
2. If  $\theta(z) = -4\sqrt{z}$  then  $\varphi(z) = 4/z^2$ .
3. If  $\theta(z) = z^2/2$  then  $\varphi(z) = [z]_0^1$ .
Our claims then follow immediately from the rate estimate (17) of Theorem 2.
In all our experiments, we ran the EXP3 variant of bandit FTRL (B-FTRL) (cf. Algorithm 3) with step-size and sampling radius parameters  $\gamma_{t} = 0.2\times t^{-1 / 2}$  and  $\delta_t = 0.1\times t^{-0.15}$  respectively. The algorithm was run for  $T = 10^{4}$  iterations and, to reduce graphical clutter, we plotted only every third point of each trajectory. Trajectories have been colored throughout with darker hues indicating later times (e.g., light blue indicates that the trajectory is closer in time to its starting point, darker shades of blue indicate proximity to the termination time). The algorithm's initial conditions were taken from a uniform initialization grid of the form  $y_{1}\in \{-1,0,1\}^{3}$  and perturbed by a uniform random number in  $[-0.1, - 0.1]$  to avoid non-generic initializations.
In general, the two defining elements of (RL) are  $a$ ) the regularizer of the method; and  $b$ ) the feedback available to the players. From our experiments, we conclude that methods with Euclidean regularization tend to have faster identification rates (i.e., converge to the support of an equilibrium / club set faster), but they are more "extreme" than methods with an "entropy-like" regularizer (in the sense that players tend to play pure strategies more often). As for the feedback available to the players, payoff-based methods tend to have higher variance (and hence a slower rate of convergence) relative to methods with full information; otherwise, from a qualitative viewpoint, there are no perceptible differences in their limiting behavior.
Finally, optimistic / extra-oracle methods with full information exhibit better convergence properties in two-player zero-sum games (relative to standard FTRL policies); however, this is a fragile advantage
that evaporates in the presence of noise and/or uncertainty (in which case "vanilla" and "optimistic" methods are essentially indistinguishable). We illustrate these findings in Fig. 2.
Regarding Fig. 1, the payoffs of the chosen games were normalized to  $[-1, 1]$  and players are assumed to choose between two actions labeled "0" and "1". The specific tableaus are shown in the table below, next to the respective portrait (all taken from Fig. 1.