SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} F _ {t + 1} \leq F _ {t} - c \gamma_ {t} + \gamma_ {t} \left\langle U _ {i, t}, z _ {i} - X _ {i, t} \right\rangle + \gamma_ {t} \left\langle b _ {i, t}, z _ {i} - X _ {i, t} \right\rangle + \frac {\gamma_ {t} ^ {2}}{2 \kappa_ {h}} \| \hat {v} _ {i, t} \| _ {\infty} ^ {2} (C.9) \\ \leq F _ {t _ {0}} - c \sum_ {s = t _ {0}} ^ {t} \gamma_ {s} + \underbrace {\sum_ {s = t _ {0}} ^ {t} \gamma_ {s} \left\langle U _ {s , i} , z _ {i} - X _ {s , i} \right\rangle} _ {E _ {U, t}} + \underbrace {\sum_ {s = t _ {0}} ^ {t} \gamma_ {s} \left\langle b _ {s , i} , z _ {i} - X _ {s , i} \right\rangle} _ {E _ {b, t}} + \sum_ {s = t _ {0}} ^ {t} \frac {\left\| \hat {v} _ {s , i} \right\| _ {\infty} ^ {2}}{2 \kappa_ {h}} \gamma_ {s} ^ {2}. (C.10) \\ \end{array}
$$
We are thus left to show is that  $c\sum_{s = t_0}^t\gamma_s$  is the dominant term above. To do so, we proceed to examine each term individually:
- Second-order term: We first deal with the second-order term  $\sum_{s=t_0}^{t} \frac{\|\hat{v}_{s,i}\|_{\infty}^2}{2\kappa_h} \gamma_s^2$ . By expanding the  $\|\hat{v}_{s,i}\|_{\infty}^2$ , we readily get
$$
\frac {\sum_ {s = t _ {0}} ^ {t} \left\| \hat {v} _ {s , i} \right\| _ {\infty} ^ {2} \gamma_ {s} ^ {2}}{\tau_ {t}} = \mathcal {O} \left(\frac {\sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \left(1 + B _ {s} ^ {2} + \sigma_ {s} ^ {2}\right)}{\sum_ {s = 1} ^ {t} \gamma_ {s}}\right). \tag {C.11}
$$
However, by our assumptions on the parameters of (RL), we readily get
$$
\lim  _ {t \rightarrow \infty} \frac {\gamma_ {t} ^ {2} \left(1 + B _ {t} ^ {2} + \sigma_ {t} ^ {2}\right)}{\gamma_ {t}} = 0 \tag {C.12}
$$
so we conclude that
$$
\lim  _ {t \rightarrow \infty} \frac {\sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \left(1 + B _ {s} ^ {2} + \sigma_ {s} ^ {2}\right)}{\sum_ {s = 1} ^ {t} \gamma_ {s}} = 0 \tag {C.13}
$$
by the Stolz-Cesaro theorem.
- Bias term: By far the most immediate, the bias term  $E_{b,t}$  is bounded as
$$
E _ {b, t} \leq 2 \sum_ {s = t _ {0}} ^ {t} \| b _ {i, t} \| _ {\infty} \gamma_ {s} \leq 2 \sum_ {s = t _ {0}} ^ {t} B _ {s} \gamma_ {s} = \mathrm {o} \left(\sum_ {s = t _ {0}} ^ {t} \gamma_ {s}\right) \quad \text {a s} t \rightarrow \infty . \tag {C.14}
$$
- Noise term: Finally, the noise term  $E_{U,t}$  is bounded by means of the Azuma-Hoeffding inequality, cf. Lemma A.3 in Appendix A. Specifically, with probability at least  $1 - \varepsilon' / 2$ , we have
$$
\begin{array}{l} E _ {U, t} := \sum_ {s = t _ {0}} ^ {t} \gamma_ {s} \left\langle U _ {s, i}, z _ {i} - X _ {s, i} \right\rangle \\ \leq 2 \left(\sum_ {s = t _ {0}} ^ {t} \| U _ {s, i} \| _ {\infty} ^ {2} \gamma_ {s} ^ {2}\right) ^ {1 / 2} \sqrt {2 \log \left(\frac {4 t ^ {2}}{\varepsilon^ {\prime}}\right)} \\ \leq 2 \left(\sum_ {s = t _ {0}} ^ {t} \sigma_ {s} ^ {2} \gamma_ {s} ^ {2}\right) ^ {1 / 2} \sqrt {2 \log \left(\frac {4 t ^ {2}}{\varepsilon^ {\prime}}\right)}. \tag {C.15} \\ \end{array}
$$
for all  $t \geq t_0$ . To proceed, note that a second application of the Stolz-Cesàro theorem yields  $\sum_{s=t_0}^{t} \sigma_s^2 \gamma_s^2 = o(\sum_{s=t_0}^{t} \gamma_s)$  and, moreover, note that  $\log(4t^2 / \varepsilon') = O(\sum_{s=t_0}^{t} \gamma_s)$ . Taking square roots and multiplying then yields that
$$
E _ {U, t} = \mathrm {o} \left(\sum_ {s = t _ {0}} ^ {t} \gamma_ {s}\right) \tag {C.16}
$$
with probability at least  $1 - \varepsilon'/2$ .
We are now in a position to establish the bound Eq. (C.6). Indeed, putting Eqs. (C.13), (C.14) and (C.16) together, we readily infer that there exists  $t_1 \geq t_0$  such that, with probability at least  $1 - \varepsilon'/2$ , we have
$$
\sum_ {s = t _ {0}} ^ {t} \gamma_ {s} \left\langle U _ {s, i}, z _ {i} - X _ {s, i} \right\rangle + \sum_ {s = t _ {0}} ^ {t} \gamma_ {s} \left\langle b _ {s, i}, z _ {i} - X _ {s, i} \right\rangle + \sum_ {s = t _ {0}} ^ {t} \frac {\left\| \hat {v} _ {s , i} \right\| _ {\infty} ^ {2}}{2 \kappa_ {h}} \gamma_ {s} ^ {2} \leq \frac {c}{2} \sum_ {s = t _ {0}} ^ {t} \gamma_ {s} \tag {C.17}
$$
for all  $t \geq t_1$ . This proves Eq. (C.6) and concludes our proof.
Summarizing the above, we have shown that, with probability at least  $1 - \varepsilon'/2$ , we have
$$
F _ {t + 1} \leq F _ {t _ {0}} - \frac {c}{2} \sum_ {s = t _ {0}} ^ {t} \gamma_ {s} \rightarrow - \infty \quad \text {a s} t \rightarrow \infty . \tag {C.18}
$$
Since  $F$  is nonnegative (by Lemma A.2), we have established that the event where  $\mathcal{L}(X)$  is not resilient to  $(i, z_i)$  is an event of probability zero. However, since there are uncountably many strategic deviations, the proof is not yet complete; the last step involves an approximation by deviations with rational entries.
Step 3.  $\mathcal{L}(X)$  is almost-surely resilient.
Proof of Step 3. The key point of the proof is the observation that a closed set is resilient if and only if it is rationally resilient, i.e., it nullifies all rational deviations  $z_{i} \in \mathcal{X}_{i} \cap \mathbb{Q}^{\mathcal{A}_{i}}$  (which are countably many). Indeed, if  $\mathcal{L}(X)$  is not resilient with positive probability, then, likewise,  $\mathcal{L}(X)$  will not be rationally resilient with positive probability either. Because there are countably many rational deviations, there must be a rational strategic deviation  $(i, z_{i})$  (with  $z_{i} \in \mathcal{X}_{i} \cap \mathbb{Q}^{\mathcal{A}_{i}}$ ) to which  $\mathcal{L}(X)$  is not resilient. This comes in contradiction with the conclusions of Step 2.
This concludes the last required step, so the proof of Theorem 1 is now complete.
D Proof of Theorems 2 and 3
In this last appendix, our goal is to prove our characterization of club sets, namely:
Our proof strategy will be to construct a sheaf of "linearized" energy functions which, when bundled together, yield a suitable Lyapunov-like function for  $S$ . To do so, let  $\mathcal{C} = \prod_{i}\mathcal{C}_{i}$  denote the support of  $S$  (cf. the definition of club sets), and let
$$
\mathcal {Z} _ {i} = \left\{e _ {i \alpha_ {i} ^ {\prime}} - e _ {i \alpha_ {i}}: \alpha_ {i} \in \mathcal {C} _ {i}, \alpha_ {i} ^ {\prime} \in \mathcal {A} _ {i} \backslash \mathcal {C} _ {i} \right\} \tag {D.1}
$$
$$
\mathcal {Z} = \bigcup_ {i \in \mathcal {N}} \mathcal {Z} _ {i} \tag {D.2}
$$
denote the set of all pure strategic deviations from  $S$ . Then, our ensemble of candidate energy functions will be given by
$$
E _ {z} (y) = \langle y, z \rangle \quad \text {f o r} z \in \mathcal {Z}, y \in \mathcal {V} ^ {*}. \tag {D.3}
$$
The motivation for this definition is given by the following lemma.
Lemma D.1. Suppose that the sequence  $y_{t}\in \mathcal{V}^{*}$ $t = 1,2,\ldots$  has  $E_z(y_t)\to -\infty$  for all  $z\in \mathcal{Z}$  as  $t\rightarrow \infty$  . Then the sequence  $x_{t} = Q(y_{t})$  converges to  $\mathcal{S}$  as  $t\to \infty$
Proof. Let  $z = e_{i\alpha_i'} - e_{i\alpha_i}$  for some  $i \in \mathcal{N}$ ,  $\alpha_i \in \mathcal{C}_i$ , and  $\alpha_i' \in \mathcal{A}_i \setminus \mathcal{C}_i$ . Since  $E_z(y_t) \to -\infty$  by assumption, we get  $y_{i\alpha_i', t} - y_{i\alpha_i, t} \to -\infty$  and hence, by Lemma A.1, we conclude that  $Q_{i\alpha_i'}(x_t) \to 0$  as  $t \to \infty$ . In turn, given that this holds for all  $i \in \mathcal{N}$  and all  $\alpha_i' \in \mathcal{A}_i \setminus \mathcal{C}_i$ , we conclude that  $x_t = Q(y_t)$  converges to  $\mathcal{S}$ .
In view of the above, we will focus on showing that  $E_z(Y_t) \to -\infty$  for all  $z \in \mathcal{Z}$ . As a first step, we establish a basic template inequality for the evolution of  $E_z$  under (RL).
Lemma D.2. Fix some  $z \in \mathcal{Z}$  and let  $E_{t} \coloneqq E_{z}(Y_{t})$ . Then, for all  $t = 1, 2, \ldots$ , we have
$$
E _ {t + 1} \leq E _ {t} + \gamma_ {t} \left\langle v \left(X _ {t}\right), z \right\rangle + \gamma_ {t} \xi_ {t} + \gamma_ {t} \psi_ {t} \tag {D.4}
$$
where the error terms  $\xi_{t}$  and  $\psi_t$  are given by
$$
\xi_ {t} = \left\langle U _ {t}, z \right\rangle \quad a n d \quad \psi_ {t} = 2 B _ {t}. \tag {D.5}
$$
Proof. Simply set  $y \gets Y_{t+1}$  in  $E_z(y)$ , invoke the definition of the update  $Y_t \gets Y_{t+1}$  in (RL), and note that  $|\langle b_t, z \rangle| \leq \|z\| \|b_t\|_{\infty} \leq 2B_t$  by the definition of  $\mathcal{Z}$ .
The key take-away from (D.4) is that, if  $X_{t}$  is close to  $\mathcal{S}$  and  $\alpha_{i}\in \mathcal{C}_{i}$ ,  $\alpha_{i}^{\prime}\in \mathcal{A}_{i}\setminus \mathcal{C}_{i}$ , we will have
$$
\langle v \left(X _ {t}\right), z \rangle = v _ {i \alpha_ {i} ^ {\prime}} \left(X _ {t}\right) - v _ {i \alpha_ {i}} \left(X _ {t}\right) = u _ {i} \left(\alpha_ {i} ^ {\prime}; X _ {- i, t}\right) - u _ {i} \left(\alpha_ {i}; X _ {- i, t}\right) <   0 \tag {D.6}
$$
by the continuity of  $u_{i}$  and the assumption that  $S$  is a club set. More concretely, by the definition of the better-reply correspondence, we have
$$
\langle v \left(x ^ {*}\right), z \rangle <   0 \quad \text {f o r a l l} x ^ {*} \in S \text {a n d a l l} z \in \mathcal {Z} \tag {D.7}
$$
and hence, by continuity, there exists a neighborhood  $\mathcal{B}$  of  $S$  such that
$$
\langle v (x), z \rangle <   0 \quad \text {f o r a l l} x \in \mathcal {B} \text {a n d a l l} z \in \mathcal {Z}. \tag {D.8}
$$
In other words, as long as  $X_{t}$  is sufficiently close to  $S$ , (D.4) exhibits a consistent negative drift pushing  $E_{t}$  towards  $-\infty$ .
To exploit this "dynamic consistency" property of  $S$ , it will be convenient to introduce the family of sets
$$
\mathcal {D} (\epsilon) = \{y \in \mathcal {V} ^ {*}: \langle y, z \rangle <   - \epsilon \text {f o r a l l} z \in \mathcal {Z} \} \tag {D.9}
$$
As we show below, these sets are mapped under  $Q$  to neighborhoods of  $S$ , so they are particularly well-suited to serve as initialization domains for (RL). This is encoded in the following properties:
Lemma D.3. Let  $x = Q(y)$  for some  $y \in \mathcal{V}^*$ . Then, for all  $\alpha_i, \alpha_i'$ ,  $i \in \mathcal{N}$ , we have
$$
x _ {i \alpha_ {i}} \leq \varphi \left(\theta \left(1 ^ {-}\right) + y _ {i \alpha_ {i} ^ {\prime}} - y _ {i \alpha_ {i}}\right) \tag {D.10}
$$
with  $\varphi$  defined as per Theorem 3, i.e.,
$$
\varphi (z) = \left\{ \begin{array}{l l} 0 & \text {i f} z \leq \theta^ {\prime} \left(0 ^ {+}\right), \\ \left(\theta^ {\prime}\right) ^ {- 1} (z) & \text {i f} \theta^ {\prime} \left(0 ^ {+}\right) <   z <   \theta^ {\prime} \left(1 ^ {-}\right), \\ 1 & \text {i f} z \geq \theta^ {\prime} \left(1 ^ {-}\right). \end{array} \right. \tag {D.11}
$$
Corollary D.1. For all  $\delta > 0$  there exists some  $\epsilon_{\delta} \in \mathbb{R}$  such that, for all  $\epsilon > \epsilon_{\delta}$  and all  $y \in \mathcal{D}_{\epsilon}$ , we have
$$
Q _ {i} \alpha_ {i} ^ {\prime} (y _ {i}) <   \delta \quad f o r a l l \alpha_ {i} ^ {\prime} \in \mathcal {A} _ {i} \backslash \mathcal {C} _ {i} a n d a l l i \in \mathcal {N}. \tag {D.12}
$$
Proof of Lemma D.3. Suppressing the player index for simplicity, the first-order stationarity conditions for the convex problem (9) readily give
$$
y _ {\alpha} - \theta^ {\prime} \left(x _ {\alpha}\right) = \mu - v _ {\alpha}, \tag {D.13}
$$
where  $\mu$  is the Lagrange multiplier for the equality constraint  $\sum_{\alpha}x_{\alpha} = 1$ , and  $\nu_{\alpha}$  is the complementary slackness multiplier of the inequality constraint  $x_{\alpha}\geq 0$  (so  $\nu_{\alpha} = 0$  whenever  $x_{\alpha} > 0$ ). Thus, rewriting (D.13) for some  $\alpha \in \mathcal{A}$ , we get
$$
y _ {\alpha^ {\prime}} - y _ {\alpha} = \theta^ {\prime} \left(x _ {\alpha^ {\prime}}\right) - \theta^ {\prime} \left(x _ {\alpha}\right) + v _ {\alpha} - v _ {\alpha^ {\prime}} \tag {D.14}
$$
$$
\theta^ {\prime} \left(x _ {\alpha^ {\prime}}\right) = \theta^ {\prime} \left(x _ {\alpha}\right) + v _ {\alpha^ {\prime}} - v _ {\alpha} + y _ {\alpha^ {\prime}} - y _ {\alpha} \leq \theta^ {\prime} \left(1 ^ {-}\right) + v _ {\alpha^ {\prime}} + y _ {\alpha^ {\prime}} - y _ {\alpha}, \tag {D.15}
$$
where we used the fact that  $\nu_{\alpha} \geq 0$ . Now, if  $\theta'(1^-) + y_{\alpha'} - y_{\alpha} < \theta'(0^+)$  and  $x_{\alpha'} > 0$  (so  $\nu_{\alpha'} = 0$ ), we will have  $\theta'(x_{\alpha'}) < \theta'(0^+)$ , a contradiction. This shows that  $x_{\alpha'} = 0$  if  $\theta'(1^-) + y_{\alpha'} - y_{\alpha} < \theta'(0^+)$ , so (D.10) is satisfied in this case. Otherwise, if  $x_{\alpha'} > 0$ , we must have  $\nu_{\alpha'} = 0$  by complementary slackness, so (D.10) follows by applying the second branch of (D.11) to (D.15).
The above provides us with a fairly good handle on the local geometric and dynamic properties of  $S$ . On the flip side however, the various error terms in (D.5) may be positive, so  $E_{t}$  may fail to be decreasing and  $X_{t}$  may drift away from  $S$ . On that account, it will be convenient to introduce the aggregate error processes
$$
I _ {t} = \sum_ {s = 1} ^ {t} \gamma_ {s} \xi_ {s} \quad \text {a n d} \quad I I _ {t} = \sum_ {s = 1} ^ {t} \gamma_ {s} \psi_ {s}. \tag {D.16}
$$
Intuitively, the aggregates (D.16) measure the total effect of each error term in (D.4), so we will establish a first series of results under the following general requirements:
1. Subleading error growth:
$$
\lim  _ {t \rightarrow \infty} I _ {t} / \tau_ {t} = 0 \quad (\text {S u b . I})
$$
$$
\lim  _ {t \rightarrow \infty} \Pi_ {t} / \tau_ {t} = 0 \quad (\text {S u b . I I})
$$
where  $\tau_{t} = \sum_{s=1}^{t} \gamma_{s}$  and both limits are to be interpreted in the almost sure sense.
2. Drift dominance:
$$
\mathbb {P} \left(\mathrm {I} _ {t} \leq C \tau_ {t} ^ {\alpha} / 2 \text {f o r a l l} t\right) \geq 1 - \eta \tag {Dom.I}
$$
$$
\mathbb {P} \left(\Pi_ {t} \leq C \tau_ {t} ^ {\alpha} / 2 \text {f o r a l l} t\right) \geq 1 - \eta \tag {Dom.II}
$$
for some  $C > 0$  and  $\alpha \in [0,1)$ .
In a nutshell, (Sub) posits that the aggregate error processes  $\mathrm{I}_t$  and  $\mathrm{II}_t$  of (D.16) are subleading relative to the long-run drift of (D.4), while (Dom) goes a step further and asks that said errors are asymptotically dominated by the drift in (D.4). Accordingly, under these implicit error control conditions, we obtain the interim convergence result below:
Proposition D.1. Let  $S$  be a club set, fix some confidence threshold  $\eta > 0$ , and let  $X_{t} = Q(Y_{t})$  be the sequence of play generated by (RL). If (Sub) and (Dom) hold, there exists an unbounded initialization domain  $\mathcal{D} \subseteq \mathcal{V}^{*}$  such that
$$
\mathbb {P} \left(X _ {t} \text {c o n v e r g e s t o} \mathcal {S} \mid Y _ {1} \in \mathcal {D}\right) \geq 1 - 2 \eta . \tag {D.19}
$$
Proof of Proposition D.1. Fix some  $z \in \mathcal{Z}$ , let  $E_t = E_z(Y_t)$ , and pick  $\alpha \in [0, 1)$  so that (Dom) holds for some  $C > 0$ . In addition, set  $c = -\sup_{x \in \mathcal{B}} \langle v(x), z \rangle > 0$ , let  $t_0 = \inf \{t : c\tau_t > C\tau_t^\alpha\}$ , and write  $\Delta E = \max_t \{C\tau_t^\alpha - c\tau_t\}$ . Then, if  $Y_1$  is initialized in  $\mathcal{D} \gets \mathcal{D}(\epsilon + \Delta E)$  where  $\epsilon$  is such that  $\mathcal{D}(\epsilon) \subseteq \mathcal{B}$ , we will have  $Y_t \in \mathcal{D}(\epsilon)$  for all  $t$ . Indeed, this being trivially the case for  $t = 1$ , assume it to be the case for all  $s = 1, 2, \ldots, t$ . Then, by (D.4) and our inductive hypothesis, we get
$$
E _ {t + 1} \leq E _ {1} - \sum_ {s = 1} ^ {t} \gamma_ {s} \left\langle v \left(X _ {s}\right), z \right\rangle + I _ {t} + \Pi_ {t} \leq - \epsilon - \Delta E - c \tau_ {t} + C \tau_ {t} ^ {\alpha} / 2 + C \tau_ {t} ^ {\alpha} / 2 \leq - \epsilon - \Delta E + \Delta E = - \epsilon \tag {D.20}
$$
i.e.,  $E_{t + 1}\in \mathcal{D}(\epsilon)$  , as claimed.
Now, since  $E_{t}\in \mathcal{D}(\epsilon)$  for all  $t$ , we conclude that
$$
E _ {t + 1} \leq E _ {1} - c \tau_ {t} + I _ {t} + \Pi_ {t} \quad \text {f o r a l l} t = 1, 2, \dots \tag {D.21}
$$
Thus, if (Sub) holds, we readily get  $E_{t} \to -\infty$  with probability 1 on the event that (Dom.I) and (Dom.II) both hold. This implies that  $E_{t} \to -\infty$ , and since  $z \in \mathcal{Z}$  above is arbitrary, we conclude that  $X_{t} \to \mathcal{S}$  with probability at least  $1 - 2\eta$ , as claimed.
We are now in a position to prove Theorem 2.
Proof of Theorem 2. Our proof will hinge on showing that (Sub) and (Dom) hold under the stated step-size and sampling parameter schedules. Our claim will then follow by a direct application of Proposition D.1 and a reduction to a suitable surface of  $\mathcal{X}$ .
First, regarding (Sub), the law of large numbers for martingale difference sequences [25, Theorem 2.18] shows that  $\mathrm{I}_t / \tau_t\to 0$  with probability 1 on the event  $\left\{\sum_{t}\gamma_{t}^{2}\mathbb{E}[\xi_{t}^{2}\mid \mathcal{F}_{t}]\right\} /\tau_{t}^{2} < \infty \}$ . However
$$
\mathbb {E} \left[ \xi_ {t} ^ {2} \mid \mathcal {F} _ {t} \right] \leq 2 ^ {2} \mathbb {E} \left[ \| U _ {t} \| _ {\infty} ^ {2} \mid \mathcal {F} _ {t} \right] \leq 2 ^ {2} \sigma_ {t} ^ {2} = \mathcal {O} \left(t ^ {2 \ell_ {\sigma}}\right) \tag {D.22}
$$
$$
\sum_ {t} \frac {\gamma_ {t} ^ {2} \mathbb {E} \left[ \xi_ {t} ^ {2} \mid \mathcal {F} _ {t} \right]}{\tau_ {t} ^ {2}} = \mathcal {O} \left(\sum_ {t} \frac {\gamma_ {t} ^ {2} \sigma_ {t} ^ {2}}{\tau_ {t} ^ {2}}\right) = \mathcal {O} \left(\sum_ {t} \frac {t ^ {- 2 \ell_ {\gamma}} t ^ {2 \ell_ {\sigma}}}{t ^ {2 (1 - \ell_ {\gamma})}}\right) = \mathcal {O} \left(\sum_ {t} \frac {1}{t ^ {2 - 2 \ell_ {\sigma}}}\right) <   \infty \tag {D.23}
$$
given that  $\ell_{\sigma} < 1/2$ . This establishes (Sub.I); the remaining requirement (Sub.II) follows trivially by noting that  $\sum_{s=1}^{t} \gamma_{s} B_{s} / \sum_{s=1}^{t} \gamma_{s} \to 0$  if and only if  $B_{t} \to 0$ , which is immediate from the theorem's assumptions.
Second, regarding (Dom), since  $B_{t}$  is deterministic and  $B_{t} = \mathcal{O}(1 / t^{\ell_{b}})$  for some  $\ell_b > 0$ , it is always possible to find  $C > 0$  and  $\alpha \in (0,1)$  so that (Dom.II) holds. We are thus left to establish (Dom.I). To that end, let  $\mathrm{I}_t^* = \sup_{1\leq s\leq t}|\mathrm{I}_t|$  and set  $P_{t}\coloneqq \mathbb{P}\big(\mathrm{I}_{t}^{*} > C\tau_{t}^{\alpha} / 2\big)$  so
$$
P _ {t} \leq \frac {\mathbb {E} \left[ \left| I _ {t} \right| ^ {q} \right]}{(C / 2) ^ {q} \tau_ {t} ^ {\alpha q}} \leq c _ {q} \frac {\mathbb {E} \left[ \left(\sum_ {s = 1} ^ {t} \gamma_ {s} ^ {2} \| U _ {s} \| _ {\infty} ^ {2}\right) ^ {q / 2} \right]}{\tau_ {t} ^ {\alpha q}} \tag {D.24}
$$
where  $c_{q}$  is a positive constant depending only on  $C$  and  $q$ , and we used Kolmogorov's inequality (Lemma A.4) in the first step and the Burkholder-Davis-Gundy inequality (Lemma A.6) in the second.
To proceed, we will require the following variant of Hölder's inequality [9, p. 15]:
$$
\left(\sum_ {s = 1} ^ {t} a _ {s} b _ {s}\right) ^ {\rho} \leq \left(\sum_ {s = 1} ^ {t} a _ {s} ^ {\frac {\lambda \rho}{\rho - 1}}\right) ^ {\rho - 1} \sum_ {s = 1} ^ {t} a _ {s} ^ {(1 - \lambda) \rho} b _ {s} ^ {\rho} \tag {D.25}
$$
valid for all  $a_{s}, b_{s} \geq 0$  and all  $\rho > 1$ ,  $\lambda \in [0,1)$ . Then, substituting  $a_{s} \gets \gamma_{s}^{2}$ ,  $b_{s} \gets \|U_{s}\|_{\infty}^{2}$ ,  $\rho \gets q/2$  and  $\lambda \gets 1/2 - 1/q$ , (D.24) gives
$$
P _ {t} \leq c _ {q} \frac {\left(\sum_ {s = 1} ^ {t} \gamma_ {s}\right) ^ {q / 2 - 1} \sum_ {s = 1} ^ {t} \gamma_ {s} ^ {1 + q / 2} \mathbb {E} \left[ \| U _ {s} \| _ {\infty} ^ {q} \right]}{\tau_ {t} ^ {\alpha q}} \leq c _ {q} \frac {\sum_ {s = 1} ^ {t} \gamma_ {s} ^ {1 + q / 2} \sigma_ {s} ^ {q}}{\tau_ {t} ^ {1 + (\alpha - 1 / 2) q}} \tag {D.26}
$$
We now consider two cases, depending on whether the numerator of (D.26) is summable or not.
Case 1:  $\ell_{\gamma}(1 + q / 2) \geq 1 + q\ell_{\sigma}$ . In this case, the numerator of (D.26) is summable under the theorem's assumptions, so the fraction in (D.26) behaves as  $\mathcal{O}(1 / t^{(1 - \ell_{\gamma})(1 + (\alpha - 1 / 2)q)})$ .
Case 2:  $\ell_{\gamma}(1 + q / 2) < 1 + q\ell_{\sigma}$ . In this case, the numerator of (D.26) is not summable under the theorem's assumptions, so the fraction in (D.26) behaves as  $\mathcal{O}\big(t^{1 - \ell_{\gamma}(1 + q / 2) + q\ell_{\sigma}} / t^{(1 - \ell_{\gamma})(1 + (\alpha - 1 / 2)q)}\big)$ .
Thus, working out the various exponents, a tedious - but otherwise straightforward - calculation shows that there exists some  $\alpha \in (0,1)$  such that  $P_{t}$  is summable as long as  $\ell_{\sigma} < 1 / 2 - 1 / q$  and  $0 \leq \ell_{\gamma} < q / (2 + q)$ . Hence, if  $\gamma$  is sufficiently small relative to  $\eta$ , we conclude that
$$
\mathbb {P} \left(\mathrm {I} _ {t} \leq C \tau_ {t} ^ {\alpha} / 2 \text {f o r a l l} t\right) \geq 1 - \sum_ {t} P _ {t} \geq 1 - \eta / 2. \tag {D.27}
$$
Finally, if  $\ell_{\gamma} > 1 / 2 + \ell_{\sigma}$ , (Dom.I) is a straightforward consequence of (D.24) for  $q = 2$ .
With all this in hand, the final steps of our proof proceed as follows:
Closedness  $\Rightarrow$  Stability. Our assertion follows by invoking Proposition D.1.
Stability  $\Longrightarrow$  Closedness. Suppose that  $S$  is not club. Then there exists some pure strategy  $\alpha \in \mathcal{C}$  and some deviation  $\alpha' \notin \mathcal{C}$  such that the deviation from  $\alpha$  to  $\alpha'$  is not costly to the deviating player. Thus, if we consider the restriction of the game to the face spanned by  $\alpha$  and  $\alpha'$  (a single-player game with two strategies), the corresponding score difference will be
$$
y _ {\alpha^ {\prime}, t} - y _ {\alpha , t} \geq \sum_ {s = 1} \gamma_ {s} b _ {s} + \sum_ {s = 1} \gamma_ {s} U _ {s} \tag {D.28}
$$
By our standing assumptions for  $b_{t}$  and  $U_{t}$  (and Doob's martingale convergence theorem for the latter), both  $\sum_{s=1} \gamma_{s} b_{s}$  and  $\sum_{s=1} \gamma_{s} U_{s}$  will be bounded from below by some (a.s.) finite random variable  $A_{0}$ . Since  $\theta$  is steep, it follows that, with probability 1,  $\lim \inf_{t \to \infty} (y_{\alpha, t}) > 0$ , so  $\mathcal{C}$  cannot be stable.
Minimality  $\Rightarrow$  Irreducible Stability. Suppose that  $S$  is m-club. Then, by our previous claim,  $S$  is stochastically asymptotically stable. If  $S$  contains a proper subsurface  $S' \subseteq S$  that is also stochastically asymptotically stable,  $S'$  must be club by the converse implication of the first part of the theorem. However, in that case,  $S$  would not be m-club, a contradiction which proves our claim.
Irreducible Stability  $\Rightarrow$  Minimality. For our last claim, assume that  $S$  is irreducibly stable. By the first part of our theorem, this implies that  $S$  is club. Then, if it so happens that  $S$  is not m-club, it would contain a proper club subsurface  $S' \subsetneq S$ ; by the first part of our theorem, this set would be itself stochastically asymptotically stable, in contradiction to the irreducibility assumption. This shows that  $S$  is m-club and concludes our proof.
We are only left to establish the convergence rate estimate of Theorem 3.
Figure 2: The long-run behavior of Algorithms 1-3 in a  $2 \times 2 \times 2$  game. Algorithms 1 and 2 were run with a logit choice map as per (HEDGE); Algorithm 3 was run with both variants, EXP3 and TSALLIS-INF. All algorithms were run for  $5 \times 10^{5}$  iterations with  $\gamma_{t} = 1 / t^{0.4}$  and  $\delta_{t} = 0.1 / t^{0.15}$ ; color indicates time, with darker hues indicating later iterations. The face to the left is closed under better replies, so  $X_{t}$  converges quickly to said face (as per Theorems 2 and 3).
Proof of Theorem 3. Going back to (D.21) and invoking Lemma D.3 shows that there exist constants  $c_{1} > 0$  and  $c_{2}\in \mathbb{R}$  such that, for all  $\alpha_{i}\in \mathcal{A}_{i}\setminus \mathcal{C}_{i},i\in \mathcal{N}$ , we have
$$
X _ {i \alpha_ {i}, t} \leq \varphi \left(\theta \left(1 ^ {-}\right) + E _ {t}\right) \leq \varphi \left(c _ {2} - c _ {1} \tau_ {t}\right) \tag {D.29}
$$
with probability 1 on the events of (Dom). We thus get