SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this light, Theorem 2 essentially states that the only robust prediction that can be made for the outcome of a regularized learning process is (minimal) closedness under better replies. This interpretation has significant cutting power for the emergence of rational behavior. To begin, in terms of equilibrium play, it effortlessly implies that a pure strategy profile is stochastically asymptotically stable under (RL) if and only if it is a strict Nash equilibrium. A version of this equivalence was only recently proved in [20] and [22] (in continuous and discrete time respectively), so Theorem 2 can be seen as a far-reaching generalization of these recent results. More to the point, since every m-club set  $S$  contains an essential equilibrium component that is fully supported in  $S$ , Theorem 2 also provides an important link between dynamic and structural stability: if an equilibrium – or a component of equilibria – is not robust to perturbations of the underlying game, it cannot be robustly identified by a regularized learning process (and vice versa). This remark is of particular importance for extensive-form games as such games often have non-generic equilibrium components that cannot be treated otherwise by the existing theory.
Finally, we should stress that Theorems 2 and 3 guarantee convergence even with a constant step-size. Together with the finite-time convergence guarantees of Corollary 2 for projection-based methods, this feature is a testament to the robustness of club sets as, in the presence of uncertainty, convergence almost always requires a vanishing step-size which can slow convergence down to a crawl. We find this robust convergence landscape particularly intriguing for future research on the topic.
PM is also with the Archimedes Research Unit – Athena RC – Department of Mathematics, National & Kapodistrian University of Athens. This work has been partially supported by the French National Research Agency (ANR) in the framework of the “Investissements d’avenir” program (ANR-15-IDEX-02), the LabEx PERSYVAL (ANR-11-LABX-0025-01), MIAI@Grenoble Alpes (ANR-19-P3IA-0003), and project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program
[48] van Damme, E. Stability and perfection of Nash equilibria. Springer-Verlag, Berlin, 1987.
[49] Viossat, Y. and Zapechelyuk, A. No-regret dynamics and fictitious play. Journal of Economic Theory, 148(2):825-842, March 2013.
[50] Vovk, V. G. Aggregating strategies. In COLT '90: Proceedings of the 3rd Workshop on Computational Learning Theory, pp. 371-383, 1990.
[51] Weibull, J. W. Evolutionary Game Theory. MIT Press, Cambridge, MA, 1995.
[52] Zimmert, J. and Seldin, Y. An optimal algorithm for stochastic and adversarial bandits. In AISTATS '19: Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, 2019.
[53] Zimmert, J. and Seldin, Y. Tsallis-INF: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research, 22(28):1-49, 2021.
A Auxiliary results
In this appendix we collect some basic properties of the regularized choice maps and some results from probability theory that will be useful in the sequel.
A.1. Regularized choice maps and their properties. Throughout this appendix, we will suppress the player index  $i \in \mathcal{N}$ , and we will follow standard conventions in convex analysis [44] that treat  $h$  as an extended-real-valued function  $h \colon \mathcal{V} \to \mathbb{R} \cup \{\infty\}$  with  $h(x) = \infty$  for all  $x \in \mathcal{V} \setminus \mathcal{X}$ . With this in mind, the subdifferential of a  $h$  at  $x \in \mathcal{X}$  is defined as
$$
\partial h (x) := \{y \in \mathcal {Y}: h \left(x ^ {\prime}\right) \geq h (x) + \langle y, x ^ {\prime} - x \rangle \text {f o r a l l} x ^ {\prime} \in \mathcal {X} \}, \tag {A.1}
$$
where  $\mathcal{V}$  denotes here the algebraic dual  $\mathcal{V}^*$  of  $\mathcal{V}$ . Accordingly, the domain of subdifferentiability of  $h$  is  $\operatorname{dom} \partial h \coloneqq \{x \in \operatorname{dom} h : \partial h \neq \emptyset\}$ , and the convex conjugate of  $h$  is defined as
$$
h ^ {*} (y) = \max  _ {x \in \mathcal {X}} \left\{\langle y, x \rangle - h (x) \right\} \tag {A.2}
$$
for all  $y \in \mathcal{V}$ . We then have the following basic results.
Lemma A.1. Let  $h$  be a regularizer on  $\mathcal{X}$ , and let  $Q: \mathcal{Y} \to \mathcal{X}$  be the induced choice map. Then:
1.  $Q$  is single-valued, and, for all  $x \in \mathcal{X}$ ,  $y \in \mathcal{Y}$ , we have  $x = Q(y) \iff y \in \partial h(x)$ .
2. For all  $x\in \mathrm{ri}\mathcal{X}$  , we have  $\partial h(x) = \{((\theta '(x_{\alpha}) + \mu)_{\alpha \in \mathcal{A}}:\mu \in \mathbb{R}\} .$
3. For all  $y \in \mathcal{V}$ , we have  $Q(y) = \nabla h^{*}(y)$ .
4.  $Q$  is  $(1 / K)$ -Lipschitz continuous with  $K \coloneqq \inf_{(0,1]} \theta''(z)$ . In particular, as a special case, the logit choice map  $\Lambda$  is 1-Lipschitz continuous in the  $(L^1, L^\infty)$  pair of norms on  $\mathcal{Y}$  and  $\mathcal{X}$  respectively.
5. If  $y_{\alpha} - y_{\alpha'} \to -\infty$  for some  $\alpha' \neq \alpha$ , then  $Q_{\alpha}(y) \to 0$ .
Remark. Some of the properties presented in Lemma A.1 are well known in the literature on regularized learning methods (see e.g., [33] and references therein), but we provide a proof of the entire lemma for completeness.
Proof of Lemma A.1. For the first property of  $Q$ , note that the maximum in (9) is attained for all  $y \in \mathcal{V}$  because  $h$  is lower-semicontinuous (l.s.c.) and strongly convex. Furthermore,  $x$  solves (9) if and only if  $y - \partial h(x) \ni 0$ , i.e., if and only if  $y \in \partial h(x)$ .
For our second claim, if  $x \in \mathrm{ri}(\mathcal{X})$ , the first-order stationarity conditions for the convex problem (9) that defines  $Q$  become
$$
y _ {\alpha} - \theta^ {\prime} \left(x _ {\alpha}\right) = \mu \quad \text {f o r a l l} \alpha \in \mathcal {A}, \tag {A.3}
$$
because the inequality constraints  $x_{\alpha} \geq 0$  are all inactive (recall that  $x \in \mathrm{ri}(\mathcal{X})$  by assumption). Now, by the first part of the theorem we have  $x = Q(y)$  if and only if  $y \in \partial h(x)$ , so we conclude that  $\partial h(x) = \{ (\theta'(x_{\alpha}) + \mu)_{\alpha \in \mathcal{A}} : \mu \in \mathbb{R} \}$ , as claimed.
For the fourth item, the expression  $Q = \nabla h^{*}$  is an immediate consequence of Danskin's theorem, while the Lipschitz continuity of  $Q$  follows from standard results, see e.g., [44, Theorem 12.60(b)].
For our last claim, let  $y_{t}$  be a sequence in  $\mathcal{V}$  such that  $y_{\alpha ,t} - y_{\alpha^{\prime},t}\to -\infty$  and let  $x_{t} = Q(y_{t})$ . Then, by descending to a subsequence if necessary, assume there exists some  $\varepsilon >0$  such that  $x_{\alpha ,t}\geq \varepsilon >0$  for all  $t$ . Then, by the defining relation  $Q(y) = \arg \max \{\langle y,x\rangle -h(x)\}$  of  $Q$ , we have:
$$
\left\langle y _ {t}, x _ {t} \right\rangle - h \left(x _ {t}\right) \geq \left\langle y _ {t}, x ^ {\prime} \right\rangle - h \left(x ^ {\prime}\right) \tag {A.4}
$$
for all  $x' \in \mathcal{X}$ . Therefore, taking  $x_t' = x_t + \varepsilon (e_{\alpha'} - e_\alpha)$ , we readily obtain
$$
\varepsilon \left(y _ {\alpha , t} - y _ {\alpha^ {\prime}, t}\right) \geq h \left(x _ {t}\right) - h \left(x _ {t} ^ {\prime}\right) \geq \min  h - \max  h \tag {A.5}
$$
which contradicts our original assumption that  $y_{\alpha ,t} - y_{\alpha^{\prime},t}\to -\infty$  . With  $\mathcal{X}$  compact, the above shows that  $x_{\alpha}^{*} = 0$  for any limit point  $x^{*}$  of  $x_{t}$  , i.e.  $Q_{\alpha}(y_t)\rightarrow 0$
The second collection of results concerns the Fenchel coupling, an energy function that was first introduced in [33, 34] and is defined as follows:
$$
F (p, y) = h (p) + h ^ {*} (y) - \langle y, p \rangle \quad \text {f o r a l l} p \in \mathcal {X} \text {a n d} y \in \mathcal {Y}. \tag {A.6}
$$
This coupling will play a major role in the proofs of Theorem 1, so we prove two of its most basic properties below.
Lemma A.2. For all  $p \in \mathcal{X}$  and all  $y, y' \in \mathcal{Y}$ , we have:
a)  $F(p,y)\geq \frac{1}{2} K\| Q(y) - p\| ^2.$  (A.7a)
b)  $F(p,y^{\prime})\leq F(p,y) + \langle y^{\prime} - y,Q(y) - p\rangle +\frac{1}{2K}\| y^{\prime} - y\|_{\infty}^{2}.$  (A.7b)
In particular, if  $h(0) = 0$ , we have
$$
\left. (K / 2) \| Q (y) \| ^ {2} \leq h ^ {*} (y) \leq - \min  h + \langle y, Q (y) \rangle + (2 / K) \| y \| _ {\infty} ^ {2} \quad f o r a l l y \in \mathcal {Y}. \right. \tag {A.8}
$$
Proof of Lemma A.2. By the strong convexity of  $h$  relative to  $\| \cdot \|$  (cf. Lemma A.1), we have
$$
\begin{array}{l} h (x) + t \langle y, p - x \rangle \leq h (x + t (p - x)) \\ \leq t h (p) + (1 - t) h (x) - \frac {1}{2} K t (1 - t) \| x - p \| ^ {2}, \tag {A.9} \\ \end{array}
$$
leading to the bound
$$
\frac {1}{2} K (1 - t) \| x - p \| ^ {2} \leq h (p) - h (x) - \langle y, p - x \rangle = F (p, y) \tag {A.10}
$$
for all  $t \in (0,1]$ . The bound (A.7a) then follows by letting  $t \to 0^+$  in (A.10).
For our second claim, we have
$$
\begin{array}{l} F (p, y ^ {\prime}) = h (p) + h ^ {*} (y ^ {\prime}) - \langle y ^ {\prime}, p \rangle \\ \leq h (p) + h ^ {*} (y) + \langle y ^ {\prime} - y, \nabla h ^ {*} (y) \rangle + \frac {1}{2 K} \| y ^ {\prime} - y \| _ {\infty} ^ {2} - \langle y ^ {\prime}, p \rangle \\ = F (p, y) + \left\langle y ^ {\prime} - y, Q (y) - p \right\rangle + \frac {1}{2 K} \| y ^ {\prime} - y \| _ {\infty} ^ {2}, \tag {A.11} \\ \end{array}
$$
where the inequality in the second line follows from the fact that  $h^*$  is  $(1 / K)$ -strongly smooth [44, Theorem 12.60(e)].
A.2. Basic results from probability theory. We conclude this appendix with some useful results from probability theory that we will use freely throughout the sequel. For a complete treatment, we refer the reader to Hall & Heyde [25].
Lemma A.3 (Azuma-Hoeffding inequality). Let  $M_t \in \mathbb{R}$ ,  $t = 1, 2, \ldots$ , be a martingale with  $\| M_t - M_{t-1} \|_\infty \leq \sigma_t$  (a.s.). Then, for all  $\eta > 0$ , we have
$$
\mathbb {P} \left(| M _ {t} | \leq \left(2 \log \left(2 t ^ {2} / \eta\right) \sum_ {s = 1} ^ {t} \sigma_ {s} ^ {2}\right) ^ {1 / 2} f o r a l l t\right) \geq 1 - \eta . \tag {A.12}
$$
Lemma A.4 (Kolmogorov's inequality). Let  $Z_{t} \in \mathbb{R}$ ,  $t = 1,2,\ldots$ , be a martingale difference sequence that is bounded in  $L^2$ . Then:
$$
\mathbb {P} \left(\max  _ {s \leq t} \sum_ {\ell = 1} ^ {s} Z _ {\ell} \geq \varepsilon\right) \leq \frac {1}{\varepsilon^ {2}} \mathbb {E} \left[ \left(\sum_ {s = 1} ^ {t} Z _ {s}\right) ^ {2} \right] \quad f o r a l l \varepsilon > 0. \tag {A.13}
$$
Lemma A.5 (Doob's maximal inequality). Let  $Z_{t} \in \mathbb{R}, t = 1,2,\ldots$ , be a martingale difference sequence that is bounded in  $L^p$  for some  $p \geq 1$ . Then
$$
\mathbb {P} \left(\max  _ {s \leq t} \left| Z _ {s} \right| > \varepsilon\right) \leq \frac {1}{\varepsilon^ {p}} \mathbb {E} \left[ \left| Z _ {t} \right| ^ {p} \right] \quad f o r a l l \varepsilon > 0. \tag {A.14}
$$
Lemma A.6 (Burkholder-Davis-Gundy inequality). Let  $Z_{t}$ ,  $t = 1,2,\ldots$ , be a martingale difference sequence in  $\mathbb{R}^n$ . Then, for all  $p > 1$ , there exist constants  $c_{p}, C_{p}$  that depend only on  $p$  and are such that
$$
c _ {p} \mathbb {E} \left[ \sum_ {s = 1} ^ {t} \| Z _ {s} \| _ {2} ^ {2} \right] ^ {p / 2} \leq \mathbb {E} \left[ \max  _ {s \leq t} \left\| \sum_ {\ell = 1} ^ {s} Z _ {\ell} \right\| _ {2} ^ {p} \right] \leq C _ {p} \mathbb {E} \left[ \sum_ {s = 1} ^ {t} \| Z _ {s} \| _ {2} ^ {2} \right] ^ {p / 2}. \tag {A.15}
$$
Lemma A.7 (Robbins-Siegmund). Let  $\mathcal{F}_t, t = 1,2,\ldots$ , be a filtration on a complete probability space  $(\Omega, \mathcal{F}, \mathbb{P})$ , and suppose that the sequences  $X_t, L_t$  and  $K_t$ $\mathcal{F}_t$ -measurable, nonnegative, and such that
$$
\mathbb {E} \left[ X _ {t + 1} \mid \mathcal {F} _ {t} \right] \leq X _ {t} \left(1 + L _ {t}\right) + K _ {t} \quad \text {w i t h p r o b a b i l i t y} 1. \tag {A.16}
$$
Then,  $X_{t}$  converges to some random variable  $X_{\infty}$  with probability 1 on the event  $\left\{\sum_{t=1}^{\infty} L_{t} < \infty \text{ and } \sum_{t=1}^{\infty} K_{t} < \infty \right\}$ .
B Specific algorithms and their properties
B.1. Known algorithms as special cases of (RL). To complement our analysis in the main part of our paper, we detail below how Algorithms 1-3 can be recast in the general framework of (RL). To lighten notation, we will assume that  $b_{t}$ ,  $U_{t}$  and  $\hat{v}_{t}$  are respectively bounded as
$$
\left\| b _ {t} \right\| _ {\infty} \leq B _ {t} \quad \left\| U _ {t} \right\| _ {\infty} \leq \sigma_ {t} \quad \text {a n d} \quad \left\| \hat {v} _ {t} \right\| _ {\infty} \leq M _ {t} \tag {B.1}
$$
and we will set
$$
G := \max  _ {i \in \mathcal {N}} \max  _ {\alpha \in \mathcal {A}} | v _ {i} (\alpha) | \tag {B.2}
$$
so we can take  $M_{t} = G + B_{t} + \sigma_{t}$  in (B.1). We will also make free use of the fact that  $v$  is Lipschitz continuous on  $\mathcal{X}$ , and we will write  $L$  for its Lipschitz modulus in the  $(L^{1}, L^{\infty})$  pair of norms on  $\mathcal{X}$  and  $\mathcal{Y}$  respectively, viz.
$$
\left\| v \left(x ^ {\prime}\right) - v (x) \right\| _ {\infty} \leq L \left\| x ^ {\prime} - x \right\| _ {1} \quad \text {f o r a l l} x, x ^ {\prime} \in \mathcal {X}. \tag {B.3}
$$
We now proceed to establish the required bounds for Algorithms 1-3:
Algorithm 1. Since  $\hat{v}_t = v(X_t)$ , we readily get  $b_{t} = U_{t} = 0$  by definition, so Algorithm 1 fits the scheme (RL) for free with  $\ell_b = \infty$ ,  $\ell_{\sigma} = 0$ .
Algorithm 2. For the case of (Opt-FTRL), we have  $\hat{v}_t = 2v(X_t) - v(X_{t-1})$  so  $b_t = v(X_t) - v(X_{t-1})$ , which is  $\mathcal{F}_t$ -measurable. We thus get
$$
\begin{array}{l} \left\| b _ {t} \right\| _ {\infty} = \left\| \mathbb {E} \left[ \hat {v} _ {t} \mid \mathcal {F} _ {t} \right] - v (X _ {t}) \right\| _ {\infty} \leq \mathbb {E} \left[ \| v (X _ {t}) - v (X _ {t - 1}) \| _ {\infty} \mid \mathcal {F} _ {t} \right] \\ \leq L \mathbb {E} \left[ \| X _ {t} - X _ {t - 1} \| \mid \mathcal {F} _ {t} \right] \quad \# \text {b y} (B.3) \\ = L \mathbb {E} \left[ \| Q \left(Y _ {t}\right) - Q \left(Y _ {t - 1}\right) \| _ {\infty} \mid \mathcal {F} _ {t} \right] \quad \# \text {b y (O p t - F T R L)} \\ \leq (L / K) \mathbb {E} [ \| Y _ {t} - Y _ {t - 1} \| _ {\infty} \mid \mathcal {F} _ {t} ] \quad \# \text {b y} \\ \leq \gamma_ {t} (L / K) \mathbb {E} \left[ 2 v \left(X _ {t}\right) - v \left(X _ {t - 1}\right) \mid \mathcal {F} _ {t} \right] \quad \# \text {b y} (\text {O p t - F T R L}) \\ \leq 3 L G / K \cdot \gamma_ {t} \quad \# \text {b y} (B. 2) \\ = \mathcal {O} \left(\gamma_ {t}\right) = \mathcal {O} \left(1 / t ^ {\ell_ {\gamma}}\right) (B.4) \\ \end{array}
$$
Moreover, given that  $\hat{v}$  is  $\mathcal{F}_t$ -measurable, we readily get  $U_{t} = 0$
Algorithm 3. Since  $\hat{\alpha}_t$  is sampled according to  $\hat{X}_t = (1 - \delta_t)X_{i,t} + \delta_t\mathrm{unif}_{\mathcal{A}_i}$  (cf. Eq. (11) in Section 3), we readily obtain  $\mathbb{E}[\hat{v}_{i,t}|\mathcal{F}_t] = v_i(\hat{X}_t)$ , and hence, by (B.3), we get
$$
B _ {t} = \mathcal {O} \left(\left\| \hat {X} _ {t} - X _ {t} \right\|\right) = \mathcal {O} \left(\delta_ {t}\right) = \mathcal {O} \left(1 / t ^ {\ell_ {\delta}}\right). \tag {B.5}
$$
Moreover, since  $\hat{X}_{i\alpha_i,t}\geq \delta_t / A_i$  , it follows that  $\| \hat{v}_t\|_{\infty} = \mathcal{O}(1 / \delta_t) = \mathcal{O}(t^{\ell_\delta})$
For comparison purposes, we illustrate the algorithms' behavior in a simple  $2 \times 2 \times 2$  game in Fig. 2 in Appendix E.
B.2. Further algorithms and illustrations. To demonstrate the breadth of (RL) as an algorithmic template, we provide below some more examples of algorithms from the game-theoretic literature that can be recast as special cases thereof (see also Table 1 for a recap).
Algorithm 4 (Mirror-prox). A progenitor of (Opt-FTRL) is the so-called mirror-prox (MP) algorithm [29, 37], which updates as:
$$
\begin{array}{l l} \tilde {Y} _ {t} = Y _ {t} + \gamma_ {t} v (X _ {t}) & Y _ {t + 1} = Y _ {t} + \gamma_ {t} v (\tilde {X} _ {t}) \\ \tilde {X} _ {t} = Q (\tilde {Y} _ {t}) & X _ {t + 1} = Q (Y _ {t + 1}). \end{array} \tag {MP}
$$
The main difference between (MP) and (Opt-FTRL) is that the former utilizes two surrogate gain vectors per iteration – meaning in particular that the interim, leading state  $\tilde{X}_t$  is generated with payoff information from  $X_{t}$ , not  $\tilde{X}_{t - 1}$ . This method has been used extensively in the literature for solving variational inequalities and two-player, zero-sum games, cf. Juditsky et al. [29] and references therein.
A calculation similar to that for (Opt-FTRL) shows that Algorithm 4 has  $B_{t} = \mathcal{O}(1 / t^{\ell_{\gamma}})$  and  $\sigma_t = 0$  because the algorithm has no further randomization.
Representative	Regularizer (θ)	Feedback	Bias (Bt)	Variance (σt)
Algorithm 1	HEDGE	z log z	full info	0	0
Algorithm 2	OMW	z log z	full info	O(1/tlγ)	0
Algorithm 3	EXP3	z log z	payoff	O(1/tlδ)	O(tlδ)
Algorithm 3	TSALLIS-INF	-4√z	payoff	O(1/tlδ)	O(tlδ)
Algorithm 4	MP	general	full info	O(1/tlγ)	0
Algorithm 5	CMW	z log z	full info	O(1/tlδ)	0
Table 1: A range of algorithms adhering to the general template (RL) and their bias and variance characteristics when run with a step-size sequence of the form  $\gamma_t = \gamma / t^{\ell_\gamma}$ ,  $\ell_\gamma \in (0,1]$ , and, where applicable, a sampling parameter  $\delta_t = \delta / t^{\ell_\delta}$ .
Algorithm 5 (Clairvoyant multiplicative weights). A recent variant of the exponential/multiplicative weights (HEDGE) algorithm is the so-called clairvoyant multiplicative weights (CMW) algorithm [39]
$$
Y _ {i, t + 1} = Y _ {i, t} + \gamma_ {t} v _ {i} \left(X _ {t + 1}\right) \quad X _ {i, t + 1} = \Lambda_ {i} \left(Y _ {i, t + 1}\right). \tag {CMW}
$$
The main difference between (CMW) and (HEDGE) is that the proxy payoff vector  $\hat{v}_t$  in (CMW) is based on the future state  $X_{t + 1}$  and not the current state  $X_{t}$ . To perform this "clairvoyant" update, the players of the game must coordinate to solve an implicit fixed point problem, so (CMW) is only meaningful when one has access to the payoff function  $v(\cdot)$ . In this regard, (CMW) can be seen as a Bregman proximal point method in the general spirit of Bauschke et al. [8].
To cast (CMW) as an instance of the generalized template (RL), simply note that the sequence of input signals is given by  $\hat{v}_t = v(X_{t + 1})$ , so  $U_{t} = 0$  and  $b_{t} = v(X_{t + 1}) - v(X_{t}) = \mathcal{O}(\gamma_{t}) = \mathcal{O}(1 / t^{\ell_{\gamma}})$ .
Our main goal in this appendix will be to prove Theorem 1 on the resilience properties of (RL). For convenience, we restate below the relevant result for ease of reference:
Proof. Our proof that  $\mathcal{L}(X)$  is resilient hinges on an energy-based technique that we will employ repeatedly in other parts of our analysis. To begin, introduce a player-strategy deviation pair  $(i,z_{i})$ , and say that a set is resilient to  $(i,z_{i})$  if there exists an element of the set, say  $x^{*}$ , which counters said deviation, i.e., such that  $u_{i}(x^{*})\geq u_{i}(z_{i};x_{-i}^{*})$ . In this specific case, our proof proceeds by contradiction, namely by assuming that, with positive probability,  $\mathcal{L}(X)$  is not resilient to  $(i,z_{i})$ . The main steps of our proof unfold as follows:
Step 1. Assume that  $\mathcal{L}(X)$  is not resilient to  $(i, z_i)$  with positive probability. Then there exists  $c, \epsilon, t_0 > 0$  such that
$$
\mathbb {P} \left(u _ {i} \left(z _ {i}; X _ {t, - i}\right) \geq u _ {i} \left(X _ {t}\right) + c f o r a l l t \geq t _ {0}\right) \geq \epsilon . \tag {C.1}
$$
Proof of Step 1. The function  $f: x \in \mathcal{X} \mapsto u_i(z_i; x_{-i}) - u_i(x)$  is continuous and  $\mathcal{X}$  is compact, so there is a definite function  $\eta \equiv \eta(\delta)$  such that if  $\| x - x' \| \leq \eta(\delta)$ , then  $|f(x) - f(x')| \leq \delta$ . Now, by assumption,  $\{\forall x^* \in \mathcal{L}(X), u_i(z_i; x_{-i}^*) > u_i(x^*)\}$  is of positive probability. We thus get
$$
\begin{array}{l} 0 <   \mathbb {P} \left\{\forall x ^ {*} \in \mathcal {L} (X), u _ {i} \left(z _ {i}; x _ {- i} ^ {*}\right) > u _ {i} \left(x ^ {*}\right) \right\} \\ = \mathbb {P} \left\{\inf  _ {x ^ {*} \in \mathcal {L} (X)} \left(u _ {i} \left(z _ {i}; x _ {- i} ^ {*}\right) - u _ {i} \left(x ^ {*}\right)\right) > 0 \right\} (C.2a) \\ = \mathbb {P} \left(\bigcup_ {m > 0} \left\{\inf  _ {x ^ {*} \in \mathcal {L} (X)} \left(u _ {i} \left(z _ {i}; x _ {- i} ^ {*}\right) - u _ {i} \left(x ^ {*}\right)\right) > 2 ^ {- m} \right\}\right) (C.2b) \\ \leq \frac {1}{2} \mathbb {P} \left\{\forall x ^ {*} \in \mathcal {L} (X), u _ {i} \left(z _ {i}; x _ {- i} ^ {*}\right) - u _ {i} \left(x ^ {*}\right) > 2 c \right\} (C.2c) \\ \end{array}
$$
for some  $c > 0$  in (C.2c), and where (C.2a) is because  $\mathcal{L}(X)$  is closed - hence compact - almost surely. Therefore, by definition of  $\eta(\cdot)$ ,
$$
0 <   \mathbb {P} \left\{\forall x ^ {*} \in \mathcal {X}, \operatorname {d i s t} \left(x ^ {*}, \mathcal {L} (X)\right) \leq \eta (c) \Rightarrow u _ {i} \left(z _ {i}; x _ {- i} ^ {*}\right) - u _ {i} \left(x ^ {*}\right) > c \right\} = 2 \epsilon \tag {C.2d}
$$
Now, let  $t_0$  such that  $\mathbb{P}\{\forall t\geq t_0,\mathrm{dist}(X_t,\mathcal{L}(X))\leq \eta (c)\} >1 - \frac{\epsilon}{2}$ . Then by construction, we get
$$
\mathbb {P} \left\{\forall t \geq t _ {0}, u _ {i} \left(z _ {i}; X _ {t, - i}\right) > u _ {i} \left(X _ {t}\right) + c \right\} > \epsilon . \tag {C.3}
$$
Intuitively, the existence of an action that consistently outperforms  $X_{t}$  runs contrary to the behavior that one would expect from any regularized learning algorithm. We will proceed to make this intuition precise below by means of an energy argument. To that end, consider the Fenchel coupling
$$
F _ {t} = h _ {i} \left(z _ {i}\right) + h _ {i} ^ {*} \left(Y _ {i, t}\right) - \left\langle Y _ {i, t}, z _ {i} \right\rangle \tag {C.4}
$$
Then, by Lemma A.2 in Appendix A, we readily get that
$$
F _ {t + 1} \leq F _ {t} - \gamma_ {t} \left\langle \hat {v} _ {i, t}, z _ {i} - X _ {i, t} \right\rangle + \frac {\gamma_ {t} ^ {2}}{2 \kappa_ {h}} \| \hat {v} _ {i, t} \| _ {\infty} ^ {2}. \tag {C.5}
$$
where, in obvious notation, we are identifying  $z_{i} \in \mathcal{A}_{i}$  with the corresponding vertex  $e_{z_i}$  of  $\mathcal{X}_i = \Delta (\mathcal{A}_i)$ . To proceed, the main idea will be to relate  $\gamma_t\langle \hat{v}_{i,t},z_i - X_{i,t}\rangle$  to its "perfect" counterpart  $\gamma_t\langle v_i(X_t),z_i - X_{i,t}\rangle$ . We formalize this below.
Step 2. If  $\mathcal{L}(X)$  is not resilient to  $(i, z_i)$ , there exists  $t_1 \geq t_0$  such that, with probability  $\varepsilon'/2 > 0$ , and for all  $t \geq t_1$ , we have
$$
F _ {t} \leq F _ {t _ {0}} - \frac {c}{2} \sum_ {s = t _ {0}} ^ {t} \gamma_ {s}. \tag {C.6}
$$
Proof of Step 2. With probability  $\varepsilon^{\prime}$  and for all  $t\geq t_0$  , we have
$$
\begin{array}{l} \gamma_ {t} \left\langle \hat {v} _ {i, t}, z _ {i} - X _ {i, t} \right\rangle = \gamma_ {t} \left\langle v _ {i} \left(X _ {t}\right), z _ {i} - X _ {i, t} \right\rangle + \gamma_ {t} \left\langle U _ {i, t}, z _ {i} - X _ {i, t} \right\rangle + \gamma_ {t} \left\langle b _ {i, t}, z _ {i} - X _ {i, t} \right\rangle (C.7) \\ \geq \left[ c + \left\langle U _ {i, t}, z _ {i} - X _ {i, t} \right\rangle + \left\langle b _ {i, t}, z _ {i} - X _ {i, t} \right\rangle \right] \gamma_ {t}. (C.8) \\ \end{array}
$$
The combination of Eqs. (C.5) and (C.8) then provides the following upper bound of  $F_{t + 1}$ :