SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hence, taking  $t \to \infty$ , and using that  $\sum_{t \geq 1} \gamma_t = \infty$ , we would get that  $D(\mathbf{X}^*, \mathbf{X}_t) \to -\infty$  which is a contradiction, since  $D(\mathbf{X}^*, \mathbf{X}_t) \geq 0$ .
Hence, there exists a subsequence  $\{\mathbf{X}_{t_k}:k\geq 1\}$  such that  $\lim_{k\to \infty}\mathbf{X}_{t_k} = \mathbf{X}^*$ , i.e.,
$$
\lim  _ {k \rightarrow \infty} D \left(\mathbf {X} ^ {*}, \mathbf {X} _ {t _ {k}}\right) = 0. \tag {D.38}
$$
Step 5. Existence of  $\lim_{t\to \infty}D(\mathbf{X}^*,\mathbf{X}_t)$
We define the sequence of events  $\{E_t : t \geq 1\}$  as
$$
E _ {t} := \left\{\sup  _ {s \leq t - 1} Z _ {s} + \Psi_ {s} + M _ {s} <   \frac {3}{4} \varepsilon \right\} \quad \text {f o r} t \geq 2 \tag {D.39}
$$
$$
E _ {1} := \left\{\mathbf {X} _ {1} \in \mathcal {U} _ {\varepsilon / 4} \right\} \tag {D.40}
$$
Then we have that  $E_{t} \in \mathcal{F}_{t}$  and  $E_{t} \subseteq \{\mathbf{X}_{s} \in \mathcal{U}_{\mathrm{vs}} : s = 1, \dots, t\}$ .
Defining the random process  $\{\tilde{D}_t:t\geq 1\}$  as
$$
\tilde {D} _ {t} = D \left(\mathbf {X} ^ {*}, \mathbf {X} _ {t}\right) \mathbb {1} _ {E _ {t}} \tag {D.41}
$$
Then, by (D.16) we have
$$
\begin{array}{l} D \left(\mathbf {X} ^ {*}, \mathbf {X} _ {t + 1}\right) \leq D \left(\mathbf {X} ^ {*}, \mathbf {X} _ {1}\right) + \gamma_ {t} \operatorname {t r} \left[ \mathbf {V} \left(\mathbf {X} _ {t}\right) \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] + \operatorname {d i a m} (\mathcal {X}) \gamma_ {t} \| b _ {t} \| _ {F} \\ + \gamma_ {t} \operatorname {t r} \left[ U _ {t} \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] + \frac {1}{2} \gamma_ {t} ^ {2} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \tag {D.42} \\ \end{array}
$$
Multiplying the above relation with  $\mathbb{1}_{E_t}$ , and noting that  $\mathbb{1}_{E_{t+1}} \leq \mathbb{1}_{E_t}$ , since  $E_{t+1} \subseteq E_t$ , we have
$$
\begin{array}{l} \tilde {D} _ {t + 1} \leq \tilde {D} _ {t} + \gamma_ {t} \operatorname {t r} \left[ \mathbf {V} \left(\mathbf {X} _ {t}\right) \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] \mathbb {1} _ {E _ {t}} + \operatorname {d i a m} (\boldsymbol {\chi}) \gamma_ {t} \| b _ {t} \| _ {F} \mathbb {1} _ {E _ {t}} \\ + \gamma_ {t} \operatorname {t r} \left[ U _ {t} \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] \mathbb {1} _ {E _ {t}} + \frac {1}{2} \gamma_ {t} ^ {2} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \mathbb {1} _ {E _ {t}} (D.43) \\ \leq \tilde {D} _ {t} + \operatorname {d i a m} (\boldsymbol {\chi}) \gamma_ {t} \| b _ {t} \| _ {F} \mathbb {1} _ {E _ {t}} + \gamma_ {t} \operatorname {t r} \left[ U _ {t} \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] \mathbb {1} _ {E _ {t}} + \frac {1}{2} \gamma_ {t} ^ {2} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \mathbb {1} _ {E _ {t}} (D.44) \\ \end{array}
$$
where in the last step we used that  $\mathrm{tr}[\mathbf{V}(\mathbf{X}_t)(\mathbf{X}_t - \mathbf{X}^*)]\mathbb{1}_{E_t}\leq 0$ . Therefore, we obtain that:
$$
\mathbb {E} \left[ \tilde {D} _ {t + 1} \mid \mathcal {F} _ {t} \right] \leq \tilde {D} _ {t} + \operatorname {d i a m} (\boldsymbol {\chi}) \gamma_ {t} \mathbb {1} _ {E _ {t}} \mathbb {E} \left[ \| b _ {t} \| _ {F} \mid \mathcal {F} _ {t} \right] + \frac {1}{2} \gamma_ {t} ^ {2} \mathbb {1} _ {E _ {t}} \mathbb {E} \left[ \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \mid \mathcal {F} _ {t} \right] \tag {D.45}
$$
$$
\mathbb {E} \left[ \operatorname {t r} \left[ \mathbf {V} \left(\mathbf {X} _ {t}\right) \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] \mathbb {1} _ {E _ {t}} \mid \mathcal {F} _ {t} \right] = \mathbb {1} _ {E _ {t}} \mathbb {E} \left[ \operatorname {t r} \left[ \mathbf {V} \left(\mathbf {X} _ {t}\right) \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] \mid \mathcal {F} _ {t} \right] = 0 \tag {D.46}
$$
Therefore,  $\{\tilde{D}_t:t\geq 1\}$  is an almost super-martingale [52] and, thus, there exists  $\tilde{D}_{\infty}$  with  $\tilde{D}_{\infty}$  finite (a.s.) and  $\tilde{D}_t\to \tilde{D}_{\infty}$  (a.s.).
Since  $E = \cap_{t\geq 1}E_t$ , we have:
$$
\begin{array}{l} \mathbb {P} \left(\lim  _ {t \rightarrow \infty} D \left(\mathbf {X} ^ {*}, \mathbf {X} _ {t}\right) \text {e x i s t s} \mid E\right) = \frac {\mathbb {P} \left(\left\{\lim  _ {t \rightarrow \infty} D \left(\mathbf {X} ^ {*} , \mathbf {X} _ {t}\right) \text {e x i s t s} \right\} \cap E\right)}{\mathbb {P} (E)} (D.47) \\ = \frac {\mathbb {P} \left(\left\{\lim  _ {t \rightarrow \infty} \tilde {D} _ {t} \text {e x i s t s} \right\} \cap E\right)}{\mathbb {P} (E)} = 1 (D.48) \\ \end{array}
$$
Hence,  $\lim_{t\to \infty}\tilde{D}_t$  exists on  $E$  and by Step 3 we readily get that  $\lim_{t\to \infty}\tilde{D}_t = 0$  on  $E$ . Thus, by Lemma A.2, we get
$$
\lim  _ {t \rightarrow \infty} \mathbf {X} _ {t} = \mathbf {X} ^ {*} \text {o n t h e e v e n t} E \tag {D.49}
$$
and setting  $\mathcal{U} = \mathcal{U}_{\varepsilon /4}$  , we obtain
$$
\mathbb {P} \left(\lim  _ {t \rightarrow \infty} \mathbf {X} _ {t} = \mathbf {X} ^ {*}\right) \geq 1 - \eta \quad \text {w h e n e v e r} \mathbf {X} _ {1} \in \mathcal {U}. \tag {D.50}
$$
This concludes our discussion and our proof.
E Numerical experiments
In this last appendix, we provide a series of additional numerical simulations to validate and explore the performance of (MMW) with payoff-based feedback.
(a) Full information (MMW).
(b) (3MW) with the (2PE) estimator
(c) (3MW) with the (1PE) estimator
Figure 2: Trajectories of the three methods for different initial conditions. The red points correspond to player 1, and the blue points to player 2. The initial points of the red trajectories are marked with  $\bullet$ , while the initial points of the blue ones are marked with  $\blacksquare$ .
Trajectory analysis. First, we proceed to a trajectory analysis of the game setup presented in Section 7. Specifically, in Fig. 2, we provide a visualization of the actual trajectories of play generated by the three methods with the same parameters as before, for different initial conditions. The trajectories are presented in Bloch spheres [48], where the points  $|0\rangle$  and  $|1\rangle$  in the figure correspond to the density matrices
$$
| 0 \rangle = \left( \begin{array}{l l} 1 & 0 \\ 0 & 0 \end{array} \right) \quad \text {a n d} \quad | 1 \rangle = \left( \begin{array}{l l} 0 & 0 \\ 0 & 1 \end{array} \right) \tag {E.1}
$$
respectively. In all figures, the points in red indicate the trajectory of Player 1, while the points in blue are for Player 2. The initial points of the red trajectories are marked with  $\bullet$ , while the initial points of the blue ones are marked with  $\blacksquare$ . [Each column of Bloch spheres in Fig. 2 has the same initial conditions.]
An important remark here is that, as suggested by Theorem 4, the trajectories of all methods converge – and quite rapidly at that – to the game's (strict) Nash equilibrium. In fact, given that the trajectories converge to a pure state, this goes to explain the faster convergence rates observed in Fig. 1: instead
(a) Performance evaluation and comparison on  $\mathcal{Q}_2$
(b) Performance evaluation and comparison on  $\mathcal{Q}_3$
(c) Performance evaluation and comparison on  $\mathcal{Q}_4$
Figure 3: Performance evaluation of (3MW) with estimators provided by (2PE) and (1PE), and comparison with the full information algorithm (MMW).
of oscillating around a solution, the MMW orbits actually converge to equilibrium in this case, so the trailing average converges at a much faster rate. This holds in all zero-sum games with a pure equilibrium, thus indicating a very important class of zero-sum games where the worst-case guarantees of MMW algorithms can be significantly improved.
Convergence speed analysis. In addition to the game setup described in Section 7, we consider the following quantum games:
-  $\mathcal{Q}_2$ : quantum analogue of the  $2 \times 2$  min-max game with payoff matrix
$$
P _ {2} = \left( \begin{array}{l l} (1 0, - 1 0) & (1 0, - 1 0) \\ (- 1 0, 1 0) & (- 1 0, 1 0) \end{array} \right) \tag {Q2}
$$
-  $\mathcal{Q}_3$ : quantum analogue of the  $3 \times 3$  min-max game with payoff
$$
P _ {3} = \left( \begin{array}{l l l} (4, - 4) & (2, - 2) & (4, - 4) \\ (- 4, 4) & (- 2, 2) & (- 4, - 4) \\ (- 4, 4) & (- 2, 2) & (- 4, - 4) \end{array} \right) \tag {Q_3}
$$
-  $\mathcal{Q}_4$ : quantum analogue of the  $3 \times 3$  min-max game with payoff
$$
P _ {4} = \left( \begin{array}{l l l} (1 0, - 1 0) & (1 0, - 1 0) & (1 0, - 1 0) \\ (- 1 0, 1 0) & (- 1 0, 1 0) & (- 1 0, 1 0) \\ (- 1 0, 1 0) & (- 1 0, 1 0) & (- 1 0, 1 0) \end{array} \right) \tag {Q4}
$$
In Fig. 3, we evaluate the convergence properties of (3MW) using the estimators (2PE) and (1PE), and compare it with the full information variant (MMW), following the same setup as described in Section 7. Specifically, for each method, we perform 10 different runs, with  $T = 10^{5}$  steps each, and compute the mean value of the duality gap as a function of the iteration  $t = 1, 2, \ldots, T$ . The solid lines correspond to the mean values of the duality gap of each method, and the shaded regions enclose the area of  $\pm 1$  (sample) standard deviation among the 10 different runs. Note that the red line, which corresponds to the full information (MMW), does not have a shaded region, since there is no randomness in the algorithm. All the runs for the three different methods were initialized for  $\mathbf{Y} = 0$  and we used  $\gamma = 10^{-2}$  for all methods. In particular, for (3MW) with gradient estimates given by (2PE) estimator, we used a sampling radius  $\delta = 10^{-2}$ , and for (3MW) with (1PE) estimator, we used  $\delta = 10^{-1}$  (in tune with our theoretical results which suggest the use of a tighter sampling radius when mixed payoff information is available to the players). As highlighted in the main text, we observe that the decrease in performance is mild, and the different algorithms achieved better rates than their theoretical guarantees.