SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our approach to tackle these issues will be to exploit the reformulation of a quantum game as a continuous game with multilinear payoffs over the spectraplex (or, rather, a product thereof), and use ideas from bandit convex optimization – in the spirit of [21, 31] – to estimate the players' payoff gradients with minimal, scalar information requirements.
<sup>1</sup>The assumption that  $x_{i,t}$  has full support is only for technical reasons. In practice, it can be relaxed by using IWE with explicit exploration – see [32] for more details.
4.2. Gradient estimation via finite-difference quotients on the spectraplex. To provide some intuition for the analysis to come, consider first a single-variable smooth function  $f \colon \mathbb{R} \to \mathbb{R}$  and a point  $x \in \mathbb{R}$ . Then, for error tolerance  $\delta > 0$ , a two-point estimate of the derivative of  $f$  at  $x$  is given by the expression
$$
\hat {f} _ {x} = \frac {f (x + \delta) - f (x - \delta)}{2 \delta} \tag {11}
$$
Going to higher dimensions, letting  $f: \mathbb{R}^d \to \mathbb{R}$  be a smooth function,  $\{e_1, \ldots, e_d\}$  be the standard basis of  $\mathbb{R}^d$  and  $s$  drawn from  $\{e_1, \ldots, e_d\}$  uniformly at random, the estimator
$$
\hat {f} _ {x} = \frac {d}{2 \delta} [ f (x + \delta s) - f (x - \delta s) ] s \tag {12}
$$
is a  $\mathcal{O}(\delta)$ -approximation of the gradient, i.e.,  $\| \mathbb{E}_s[\hat{f}_x] - \nabla f(x)\| _F = \mathcal{O}(\delta)$ . This idea is the basis of the Kiefer-Wolffowitz stochastic approximation scheme [30] and will be the backbone of our work.
Now, to employ this type of estimator for a function over the set of density matrices  $\mathcal{X}$  in  $\mathbb{H}^d$ , we need to ensure two things: (i) the feasibility of the sampling direction, and (ii) the feasibility of the evaluation point. The first caveat is due to the fact that the set of the density matrices forms a lower dimensional manifold in the set of Hermitian operators, and therefore, not all directions from a base of  $\mathbb{H}^d$  are feasible. The second one is due to the fact that  $\mathcal{X}$  is bounded, thus, even if the sampling direction is feasible, the evaluation point can lie outside the set  $\mathcal{X}$ . We proceed to ensure all this in a series of concrete steps below.
Sampling Directions. We begin with the issue of defining a proper sampling set for the estimator's finite-difference directions. To that end, we will first construct an orthonormal basis of the tangent hull  $\mathcal{Z} = \{\mathbf{Z}\in \mathbb{H}^d:\operatorname {tr}\mathbf{Z} = 0\}$  of  $\mathcal{X}$ , i.e., the subspace of traceless matrices of  $\mathbb{H}^d$ . Note that if  $\mathbf{Z}\in \mathcal{Z}$  then for any  $\mathbf{X}\in \mathbb{H}^d$  it holds  $(a)\mathbf{X} + \mathbf{Z}\in \mathbb{H}^{d}$ , and  $(b)\operatorname {tr}[\mathbf{X} + \mathbf{Z}] = \operatorname {tr}[\mathbf{X}]$ .
Denoting by  $\Delta_{k\ell} \in \mathbb{H}^d$  the matrix with 1 in the  $(k,\ell)$ -position and 0's everywhere else, it is easy to see that the set  $\{\{\Delta_{j,j}\}_{j=1}^{d} \{\mathbf{e}_{k\ell}\}_{k < \ell}, \{\tilde{\mathbf{e}}_{k\ell}\}_{k < \ell}\}$  is an orthonormal basis of  $\mathbb{H}^d$ , where
$$
\mathbf {e} _ {k \ell} = \frac {1}{\sqrt {2}} \Delta_ {k \ell} + \frac {1}{\sqrt {2}} \Delta_ {\ell k} \quad \text {a n d} \quad \tilde {\mathbf {e}} _ {k \ell} = \frac {i}{\sqrt {2}} \Delta_ {k \ell} - \frac {i}{\sqrt {2}} \Delta_ {\ell k} \tag {13}
$$
for  $1 \leq k < \ell \leq d$ , where  $i$  is the imaginary unit with  $i^2 = -1$ . The next proposition provides a basis for the subspace  $\mathcal{Z}$ , whose proof lies in the appendix.
Proposition 1. Let  $\mathbf{E}_j$  be defined as  $\mathbf{E}_j = \frac{1}{\sqrt{j(j + 1)}}\left(\Delta_{11} + \dots + \Delta_{jj} - j\Delta_{j+1,j+1}\right)$  for  $j = 1, \ldots, d-1$ . Then, the set  $\mathcal{E} = \left\{\{\mathbf{E}_j\}_{j=1}^{d-1}, \{\mathbf{e}_{k\ell}\}_{k < \ell}, \{\tilde{\mathbf{e}}_{k\ell}\}_{k < \ell}\right\}$  is an orthonormal basis of  $\mathcal{Z}$ .
In the sequel, we will use this basis as an orthonormal sampler from which to pick the finite-difference directions for the estimation of  $\mathbf{V}$ .
Feasibility Adjustment. After establishing an orthonormal basis for  $\mathcal{Z}$  as per Proposition 1, we readily get that for any  $\mathbf{X} \in \mathcal{X}$ , any  $\mathbf{Z} \in \mathcal{E}^{\pm} := \left\{\{\pm \mathbf{E}_j\}_{j=1}^{d-1}, \{\pm \mathbf{e}_{k\ell}\}_{k<\ell}, \{\pm \tilde{\mathbf{e}}_{k\ell}\}_{k<\ell}\right\}$  and  $\delta > 0$ , the point  $\mathbf{X} + \delta \mathbf{Z}$  belongs to  $\mathcal{Z}$ . However, depending on the value of the exploration parameter  $\delta$  and the distance of  $\mathbf{X}$  from the boundary of  $\mathcal{X}$ , the point  $\mathbf{X} + \delta \mathbf{Z} \in \mathbb{H}^d$  may fail to lie in  $\mathcal{X}$  due to violation of the positive-semidefinite condition. On that account, we now treat the latter restriction, i.e., the feasibility of the evaluation point.
To tackle this, the idea is to transfer the point  $\mathbf{X}$  toward the interior of  $\mathcal{X}$  and move along the sampled direction from there. For this, we need to find a reference point  $\mathbf{R} \in \mathrm{ri}(\mathcal{X})$  and a "safety net"  $r > 0$  such that  $\mathbf{R} + r\mathbf{Z} \in \mathcal{X}$  for any  $\mathbf{Z} \in \mathcal{E}^{\pm}$ . Then, for  $\delta \in (0,r)$ , the point
$$
\mathbf {X} ^ {(\delta)} := \mathbf {X} + \frac {\delta}{r} (\mathbf {R} - \mathbf {X}) \tag {14}
$$
lies in  $\mathrm{ri}(\mathcal{X})$ , and moving along  $\mathbf{Z} \in \mathcal{E}^{\pm}$ , the point  $\mathbf{X}^{(\delta)} + \delta \mathbf{Z} = (1 - \frac{\delta}{r})\mathbf{X} + \frac{\delta}{r} (\mathbf{R} + r\mathbf{Z})$  remains in  $\mathcal{X}$  as a convex combination of two elements in  $\mathcal{X}$ . The following proposition provides an exact expression for  $\mathbf{R}$  and  $r$ , which we will use next to guarantee the feasibility of the sampled iterates.
Proposition 2. Let  $\mathbf{R} = \frac{1}{d}\sum_{j=1}^{d}\Delta_{jj}$ . Then, for  $r = \min\left\{\frac{1}{\sqrt{d(d-1)}}, \frac{\sqrt{2}}{d}\right\}$ , it holds that  $\mathbf{R} + r\mathbf{Z} \in \mathcal{X}$  for any direction  $\mathbf{Z} \in \mathcal{E}^{\pm}$ .
5 Bandit learning in zero-sum quantum games
With all these in hand, we are now ready to proceed to the presentation of the MMW with limited feedback information. To streamline our presentation, before delving into the more difficult "bandit feedback" case – where each player  $i \in \mathcal{N}$  only observes the realized payoff observable  $U_{i}(\omega)$  – we begin with the simpler case where players observe their mixed payoffs  $u_{i}$  at a given profile  $\mathbf{X} \in \mathcal{X}$ .
5.1. Learning with mixed payoff observations. Our main idea to exploit the observation of mixed payoffs and the finite-difference sampling to the fullest will be to introduce a "coordination phase" where players take a sampling step before updating their state variables and continue playing. In more detail, we will take an approach similar to Bervoets et al. [8] and assume that players alternate between an "exploration" and an "exploitation" update that allows them to sample the landscape of  $\mathcal{L}$  efficiently at each iteration. Concretely, writing  $\bar{\mathbf{X}}_t$  and  $\delta_t$  for the players' state profile and sampling radius  $\delta_t$  at stage  $t = 1,2,\ldots$ , the sequence of events that we envision proceeds as follows:
Step 1. Draw a sampling direction  $\mathbf{Z}_{i,t} \in \mathcal{E}_i$  and  $s_{i,t} \in \{\pm 1\}$  uniformly at random.
Step 2. (a) Play  $\mathbf{X}_{i,t}^{(\delta)} + s_{i,t}\delta_t\mathbf{Z}_{i,t}$  and observe  $u_{i}(\mathbf{X}_{t}^{(\delta)} + s_{t}\delta_{t}\mathbf{Z}_{t})$
(b) Play  $\mathbf{X}_{i,t}^{(\delta)} - s_{i,t}\delta_t\mathbf{Z}_{i,t}$  and observe  $u_{i}(\mathbf{X}_{t}^{(\delta)} - s_{t}\delta_{t}\mathbf{Z}_{t})$
Step 3. Approximate  $\mathbf{V}_i(\mathbf{X}_t)$  via the two-point estimator (2PE):
$$
\hat {\mathbf {V}} _ {i, t} := \frac {D _ {i}}{2 \delta_ {t}} \left[ u _ {i} \left(\mathbf {X} _ {t} ^ {(\delta)} + s _ {t} \delta_ {t} \mathbf {Z} _ {t}\right) - u _ {i} \left(\mathbf {X} _ {t} ^ {(\delta)} - s _ {t} \delta_ {t} \mathbf {Z} _ {t}\right) \right] s _ {i, t} \mathbf {Z} _ {i, t} \tag {2PE}
$$
where  $D_{i} = d_{i}^{2} - 1$  is the dimension of  $\mathbb{H}^{d_i}$ , and  $D\coloneqq \max_{i\in \mathcal{N}}D_{i}$
The main guarantee of the resulting  $(3\mathrm{MW}) + (2\mathrm{PE})$  algorithm may then be stated as follows:
Theorem 2. Suppose that each player of a 2-player zero-sum game  $\mathcal{Q}$  follows (3MW) for  $T$  epochs with learning rate  $\gamma$ , sampling radius  $\delta$ , and gradient estimates provided by (2PE). Then the players' empirical frequency of play enjoys the duality gap guarantee
$$
\mathbb {E} \left[ \operatorname {G a p} _ {\mathcal {L}} \left(\bar {\mathbf {X}} _ {T}\right) \right] \leq \frac {H}{\gamma T} + 8 D ^ {2} G ^ {2} \gamma + 1 6 D L \delta \tag {15}
$$
where  $H = \log (d_1d_2)$ . In particular, for  $\gamma = (DG)^{-1}\sqrt{H / (8T)}$  and  $\delta = (G / L)\sqrt{H / (8T)}$ , the players enjoy the equilibrium convergence guarantee
$$
\mathbb {E} \left[ \operatorname {G a p} _ {\mathcal {L}} \left(\bar {\mathbf {X}} _ {T}\right) \right] \leq 8 D G \sqrt {2 H / T}. \tag {16}
$$
Compared to Theorem 1, the convergence rate (16) of Theorem 2 is quite significant because it only differs by a factor which is linear in the dimension of the ambient space and otherwise maintains the same  $\mathcal{O}(\sqrt{T})$  dependence on the algorithm's runtime. In this regard, Theorem 2 shows that the "explore-exploit" sampler underlying (2PE) is essentially as powerful as the full information framework of Jain & Watrous [27] – and this, despite the fact that players no longer require access to the gradient matrix  $\mathbf{V}$  of  $\mathcal{L}$ . This echoes a range of previous findings in stochastic convex optimization for the efficiency of two-point samplers [3, 53], a similarity we find particularly surprising given the stark differences between the two settings – non-commutativity, min-max versus min-min landscape. The key ingredients for the equilibrium convergence rate of Theorem 2 are the two technical results below. The first is a feedback-agnostic "energy inequality" which is tied to the update structure of (MMW) and is stated in terms of the quantum relative entropy function
$$
D (\mathbf {P}, \mathbf {X}) = \operatorname {t r} \left[ \mathbf {P} (\log \mathbf {P} - \log \mathbf {X}) \right] \tag {17}
$$
for  $\mathbf{P},\mathbf{X}\in \mathcal{X}$  with  $\mathbf{X}\succ 0$  .Concretely, we have the following estimate.
Lemma 1. Fix some  $\mathbf{P} \in \mathcal{X}$ , and let  $\mathbf{X}_t, \mathbf{X}_{t+1}$  be two successive iterates of (3MW), without any assumptions for the input sequence  $\hat{\mathbf{V}}_t$ . We then have
$$
D \left(\mathbf {P}, \mathbf {X} _ {t + 1}\right) \leq D \left(\mathbf {P}, \mathbf {X} _ {t}\right) + \gamma_ {t} \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} \left(\mathbf {X} _ {t} - \mathbf {P}\right) \right] + \frac {\gamma_ {t} ^ {2}}{2} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2}. \tag {18}
$$
The proof of Lemma 1 follows established techniques in the theory of (MMW), so we defer a detailed discussion to the appendix. The second result that we will need is tailored to the estimator (2PE) and provides a tight estimate of its moments conditioned on the history  $\mathcal{F}_t = \mathcal{F}(\mathbf{X}_1,\dots ,\mathbf{X}_t)$  of  $\mathbf{X}_t$
Algorithm 1: MMW with bandit feedback
1: Input:  $\mathbf{Y}_1\gets 0$  ; safety parameter  $r_i$  and anchor point  $\mathbf{R}_i,i\in \mathcal{N}$  ; step-size  $\gamma_{t}$  ; sampling radius  $\delta_t$    
2: for  $t = 1,2,\ldots$  do simultaneously for all  $i\in \mathcal{N}$    
3: Set  $\mathbf{X}_{i,t} = \exp (\mathbf{Y}_{i,t}) / \mathrm{tr}[\exp (\mathbf{Y}_{i,t})]$    
4: Sample  $\mathbf{Z}_{i,t}$  uniformly from  $\mathcal{E}_i^{\pm}$    
5: Play  $\mathbf{X}_{i,t}^{(\delta)} + \delta_t\mathbf{Z}_{i,t}$    
6: Observe  $U_{i}(\omega_{t})$    
7: Set  $\hat{\mathbf{V}}_{i,t}\coloneqq D_i / \delta_t\cdot U_i(\omega_t)\mathbf{Z}_{i,t}$    
8: Update  $\mathbf{Y}_{i,t + 1}\leftarrow \mathbf{Y}_{i,t} + \gamma_t\hat{\mathbf{V}}_{i,t}$    
9: end for
Proposition 3. The estimator (2PE) enjoys the conditional bounds
$$
(i) \left\| \mathbb {E} \left[ \hat {\mathbf {V}} _ {t} \mid \mathcal {F} _ {t} \right] - \mathbf {V} \left(\mathbf {X} _ {t}\right) \right\| _ {F} \leq 4 D L \delta_ {t} \quad a n d \quad (i i) \quad \mathbb {E} \left[ \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \mid \mathcal {F} _ {t} \right] \leq 1 6 D ^ {2} G ^ {2} \tag {19}
$$
The defining element in Proposition 3 is that even though the estimator (2PE) is biased, its second moment is bounded as  $\mathcal{O}(1)$ . This is ultimately due to the multilinearity of the players' payoff functions and plays a pivotal role in showing that the duality gap of  $\bar{\mathbf{X}}_t$  under (3MW) is of the same order as under (MMW), because the bias can be controlled with affecting the variance of the estimator. We provide a detailed proof of Lemma 1, Proposition 3, and Theorem 2 in the appendix.
5.2. Learning with bandit feedback. Despite its strong convergence guarantees, a major limiting factor in the applicability of Theorem 2 is that, in many cases, the game's players may only be able to observe their realized payoff observables  $U_{i}(\omega)$ , and their mixed payoffs  $u_{i}(\mathbf{X})$  could be completely inaccessible. In particular, as we described in Section 2, each outcome  $\omega \in \Omega$  of the POVM occurs with probability  $P_{\omega}(\mathbf{X}_t)$  under the strategy profile  $\mathbf{X}_t$ . Accordingly, if this is the only information available to the players, they will need to estimate their individual payoff gradients through the single observation of the (random) scalar  $U_{i}(\omega_{t}) \in \mathbb{R}$ . In view of this, and inspired by previous works on payoff-based learning and zeroth-order optimization [8, 9, 11, 25, 49, 50], we will consider the single-point stochastic approximation approach of [21, 55] which unfolds as follows:
Step 1. Each player draws a sampling direction  $\mathbf{Z}_{i,t} \in \mathcal{E}_i^{\pm}$  uniformly at random.
Step 2. Each player plays  $\mathbf{X}_{i,t}^{(\delta)} + \delta_t\mathbf{Z}_{i,t}$
Step 3. Each player receives  $U_{i}(\omega_{t})$ .
Step 4. Each player approximates  $\mathbf{V}_i(\mathbf{X}_t)$  via the one-point estimator (1PE):
$$
\hat {\mathbf {V}} _ {i, t} := \frac {D _ {i}}{\delta_ {t}} U _ {i} \left(\omega_ {t}\right) \mathbf {Z} _ {i, t} \tag {1PE}
$$
In this case, the players' gradient estimates may be bounded as follows:
Proposition 4. The estimator (1PE) enjoys the conditional bounds
$$
(i) \left\| \mathbb {E} \left[ \hat {\mathbf {V}} _ {t} \mid \mathcal {F} _ {t} \right] - \mathbf {V} \left(\mathbf {X} _ {t}\right) \right\| _ {F} \leq 4 D L \delta_ {t} \quad a n d \quad (i i) \quad \mathbb {E} \left[ \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \mid \mathcal {F} _ {t} \right] \leq 4 D ^ {2} B ^ {2} / \delta_ {t} ^ {2}. \tag {20}
$$
The crucial difference between Propositions 3 and 4 is that the former leads to a gradient estimator with  $\mathcal{O}(1)$  variance and magnitude, whereas the magnitude of the latter is inversely proportional to  $\delta_t$ ; however, since  $\delta_t$  in turn controls the bias of the gradient estimator, we must now resolve a bias-variance dilemma, which was absent in the case of (2PE). This leads to the following variant of Theorem 2 with bandit, realization-based feedback:
Theorem 3. Suppose that each player of a 2-player zero-sum game  $\mathcal{Q}$  follows (3MW) for  $T$  epochs with learning rate  $\gamma$ , sampling radius  $\delta$ , and gradient estimates provided by (1PE). Then the players' empirical frequency of play enjoys the duality gap guarantee
$$
\mathbb {E} \left[ \operatorname {G a p} _ {\mathcal {L}} \left(\bar {\mathbf {X}} _ {T}\right) \right] \leq \frac {H}{\gamma T} + \frac {2 D ^ {2} B ^ {2} \gamma}{\delta^ {2}} + 1 6 D L \delta \tag {21}
$$
where  $H = \log (d_1d_2)$ . In particular, for  $\gamma = \left(\frac{H}{2T}\right)^{3/4} \frac{1}{2D\sqrt{BL}}$  and  $\delta = \left(\frac{H}{2T}\right)^{1/4} \sqrt{\frac{B}{4L}}$ , the players enjoy the equilibrium convergence guarantee:
$$
\mathbb {E} \left[ \operatorname {G a p} _ {\mathcal {L}} \left(\bar {\mathbf {X}} _ {T}\right) \right] \leq \frac {2 ^ {3 / 4} 8 H ^ {1 / 4} D \sqrt {B L}}{T ^ {1 / 4}}. \tag {22}
$$
An important observation here is that the players' equilibrium convergence rate under (3MW) + (1PE) no longer matches the convergence rate of the vanilla MMW algorithm (Theorem 1). The reason for this is the bias-variance trade-off in the estimator (1PE), and is reminiscent of the drop in the rate of regret minimization from  $\mathcal{O}(T^{1/2})$  to  $\mathcal{O}(T^{2/3})$  under (IWE) with bandit feedback and explicit exploration in finite games. A kernel-based approach in the spirit of Bubeck et al. [13] could possibly be used to fill the  $\mathcal{O}(T^{1/4})$  gap between Theorems 1 and 3, but this would come at the cost of a possibly catastrophic dependence on the dimension (which is already quadratic in our setting). This consideration is beyond the scope of our work, but it would constitute an important future direction.
6 Bandit learning in  $N$ -player quantum games
We conclude our paper with an examination of the behavior of the MMW algorithm in general,  $N$ -player quantum games. Here, a major difficulty that arises is that, in stark contrast to the min-max case, the set of the game's equilibria can be disconnected, so any convergence result will have to be, by necessity, local. In addition, because general  $N$ -games do not have the amenable profile of a bilinear min-max problem - they are multilinear, multi-objective problems - it will not be possible to obtain any convergence guarantees for the game's empirical frequency of play (since there is no convex structure to exploit). Instead, we will have to focus squarely on the induced trajectory of play, which carries with it a fair share of complications.
Inspired by the very recent work of [38], we will not constrain our focus to a specific class of games, but to a specific class of equilibria. In particular, we will consider the behavior of MMW-based learning with respect to Nash equilibria  $\mathbf{X}^{*}\in \mathcal{X}$  that satisfy the variational stability condition
$$
\operatorname {t r} \left[ \mathbf {V} (\mathbf {X}) \left(\mathbf {X} - \mathbf {X} ^ {*}\right) \right] <   0 \quad \text {f o r a l l} \mathbf {X} \in \mathcal {U} \backslash \left\{\mathbf {X} ^ {*} \right\}. \tag {VS}
$$
This condition can be traced back to [44], and can be seen as a game-theoretic analogue of first-order stationarity in the context of continuous optimization, or as an equilibrium refinement in the spirit of the seminal concept of evolutionary stability in population games [39, 40].<sup>2</sup> Importantly, as was shown in [38], variationally stable equilibria are the only equilibria that are asymptotically stable under the continuous-time dynamics of the "follow the regularized leader" (FTRL) class of learning policies, so it stands to reason to ask whether they enjoy a similar convergence landscape in the context of bona fide, discrete-time learning with minimal, payoff-based feedback.
Our final result provides an unambiguously positive answer to this question:<sup>3</sup>
Theorem 4. Fix some tolerance level  $\eta \in (0,1)$  and suppose that the players of an  $N$ -player quantum game follow (3MW) with bandit, realization-based feedback, and surrogate gradients provided by the estimator (1PE) with step-size and sampling radius parameters such that
$$
(i) \sum_ {t = 1} ^ {\infty} \gamma_ {t} = \infty , \quad (i i) \sum_ {t = 1} ^ {\infty} \gamma_ {t} \delta_ {t} <   \infty , \quad a n d \quad (i i) \sum_ {t = 1} ^ {\infty} \gamma_ {t} ^ {2} / \delta_ {t} ^ {2} <   \infty . \tag {23}
$$
If  $\mathbf{X}^*$  is variationally stable, there exists a neighborhood of  $\mathcal{U}$  such that
$$
\mathbb {P} \left(\lim  _ {t \rightarrow \infty} \mathbf {X} _ {t} = \mathbf {X} ^ {*}\right) \geq 1 - \eta \quad \text {w h e n e v e r} \mathbf {X} _ {1} \in \mathcal {U}. \tag {24}
$$
It is worth noting that the last-iterate convergence guarantee of Theorem 4 is considerably stronger than the time-averaged variants of Theorems 1–3, and we are not aware of any comparable convergence guarantee for general quantum games. [Trivially, last-iterate convergence implies time-averaged convergence, but the converse, of course, may fail to hold] As such, especially in cases that require to track the trajectory of the system or the players' day-to-day rewards, Theorem 4 provides an important guarantee for the realized sequence of events.
On the other hand, in contrast to Theorem 4, it should be noted that the guarantees of Theorems 1-3 are global. Given that general quantum games may in general possess a large number of disjoint Nash equilibria, this transition from global to local convergence guarantees seems unavoidable. It is, however, an open question whether (VS) could be exploited further in order to deduce the rate of convergence to such equilibria; we leave this as a direction for future research.
2It should be noted here that, if reduced to the simplex, the stability condition (VS) is exactly equivalently to the variational characterization of evolutionarily stable states due to Taylor [56].
3Strictly speaking, the algorithms (3MW) and (1PE) have been stated in the context of 2-player games. The extension to  $N$ -player games is straightforward, so we do not present it here; for the details (which hide no subtleties), see the appendix.
Figure 1: Performance evaluation of the (3MW) with the (2PE) and (1PE) estimators and comparison with the full information (MMW). The solid lines correspond to the mean values of the duality gap of each method, and the shaded regions enclose the area of  $\pm 1$  (sample) standard deviation among the different runs.
In this last section, we provide numerical simulations to validate and explore the performance of (MMW) with payoff-based feedback. Additional experiments can be found in Appendix E.
Game setup. Our testbed is a two-player zero-sum quantum game, which is the quantum analogue of a  $2 \times 2$  min-max game with actions  $\{\alpha_{1}, \alpha_{2}\}$  and  $\{\beta_{1}, \beta_{2}\}$ , and payoff matrix
$$
P = \left( \begin{array}{l l} (4, - 4) & (2, - 2) \\ (- 4, 4) & (- 2, 2) \end{array} \right) \tag {25}
$$
In the quantum regime, the payoff information of the quantum game is encoded in the Hermitian matrices  $\mathbf{W}_1 = \mathrm{diag}(4,2, - 4, - 2)$ , and  $\mathbf{W}_2 = -\mathbf{W}_1$  as per Eq. (3) in Section 2. By elementary considerations, the action profile  $(\alpha_{1},\beta_{2})$  is a strict Nash equilibrium of the classical zero-sum game, which corresponds to the pure quantum state with density matrix profile  $\mathbf{X}^{*} = (\mathbf{X}_{1}^{*},\mathbf{X}_{2}^{*})$  where  $\mathbf{X}_1^* = e_1\otimes e_1$  and  $\mathbf{X}_2^* = e_2\otimes e_2$  in the standard basis in which  $\mathbf{W}_1$  and  $\mathbf{W}_2$  are diagonal.