SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hence, for  $r \leq \sqrt{2} / d$ , and  $\mathbf{Z} = -\frac{1}{\sqrt{2}} (\pmb{\Delta}_{k\ell} + \pmb{\Delta}_{\ell k})$ , we have that  $u^{\dagger}(\mathbf{R} + r\mathbf{Z})u \geq 0$  for all  $u \in \mathbb{C}^d$ .
For  $\mathbf{Z} = \frac{i}{\sqrt{2}} (\Delta_{k\ell} - \Delta_{\ell k})$  and  $u\in \mathbb{C}^d\setminus \{0\}$ , and using (B.11), we have:
$$
\begin{array}{l} u ^ {\dagger} (\mathbf {R} + r \mathbf {Z}) u = \frac {1}{d} \sum_ {j = 1} ^ {d} | u _ {j} | ^ {2} - \frac {r}{\sqrt {2}} 2 \mathrm {I m} (\bar {u} _ {k} u _ {\ell}) \\ = \frac {1}{d} \sum_ {j \neq k, \ell} | u _ {j} | ^ {2} + \frac {1}{d} \left(| u _ {k} | ^ {2} + | u _ {\ell} | ^ {2} - \frac {r d}{\sqrt {2}} 2 \operatorname {I m} \left(\bar {u} _ {k} u _ {\ell}\right)\right) \tag {B.17} \\ \end{array}
$$
If  $\operatorname{Im}(\bar{u}_k u_\ell) < 0$ , we get that  $u^\dagger(\mathbf{R} + r\mathbf{Z})u > 0$ , while if  $\operatorname{Im}(\bar{u}_k u_\ell) \geq 0$  and  $r \leq \sqrt{2}/d$ :
$$
u ^ {\dagger} (\mathbf {R} + r \mathbf {Z}) u \geq \frac {1}{d} \sum_ {j \neq k, \ell} | u _ {j} | ^ {2} + \frac {1}{d} | u _ {k} + i u _ {\ell} | ^ {2} \geq 0 \tag {B.18}
$$
Hence, for  $r \leq \sqrt{2}/2d$ , and  $\mathbf{Z} = \frac{i}{\sqrt{2}} (\Delta_{k\ell} - \Delta_{\ell k})$ , we have that  $u^{\dagger}(\mathbf{R} + r\mathbf{Z})u \geq 0$  for all  $u \in \mathbb{C}^d$ .
For  $\mathbf{Z} = -\frac{i}{\sqrt{2}} (\Delta_{k\ell} - \Delta_{\ell k})$  and  $u\in \mathbb{C}^d\setminus \{0\}$ , and using (B.11), we have:
$$
\begin{array}{l} u ^ {\dagger} (\mathbf {R} + r \mathbf {Z}) u = \frac {1}{d} \sum_ {j = 1} ^ {d} | u _ {j} | ^ {2} + \frac {r}{\sqrt {2}} 2 \mathrm {I m} (\bar {u} _ {k} u _ {\ell}) \\ = \frac {1}{d} \sum_ {j \neq k, \ell} | u _ {j} | ^ {2} + \frac {1}{d} \left(| u _ {k} | ^ {2} + | u _ {\ell} | ^ {2} + \frac {r d}{\sqrt {2}} 2 \operatorname {I m} \left(\bar {u} _ {k} u _ {\ell}\right)\right) \tag {B.19} \\ \end{array}
$$
If  $\mathrm{Im}(\bar{u}_k u_\ell) > 0$ , we get that  $u^\dagger (\mathbf{R} + r\mathbf{Z})u > 0$ , while if  $\mathrm{Im}(\bar{u}_k u_\ell) \leq 0$  and  $r \leq \sqrt{2}/d$ :
$$
u ^ {\dagger} (\mathbf {R} + r \mathbf {Z}) u \geq \frac {1}{d} \sum_ {j \neq k, \ell} | u _ {j} | ^ {2} + \frac {1}{d} | u _ {k} - i u _ {\ell} | ^ {2} \tag {B.20}
$$
Hence, for  $r \leq \sqrt{2} / 2d$ , and  $\mathbf{Z} = -\frac{i}{\sqrt{2}} (\Delta_{k\ell} - \Delta_{\ell k})$ , we have that  $u^{\dagger}(\mathbf{R} + r\mathbf{Z})u \geq 0$  for all  $u \in \mathbb{C}^d$ .
For  $\mathbf{Z} = \frac{1}{\sqrt{j(j + 1)}}\left(\Delta_{11} + \dots + \Delta_{jj} - j\Delta_{(j + 1)(j + 1)}\right)$ , we have:
$$
\begin{array}{l} u ^ {\dagger} (\mathbf {R} + r \mathbf {Z}) u = \frac {1}{d} \sum_ {k = 1} ^ {d} | u _ {k} | ^ {2} + \frac {r}{\sqrt {j (j + 1)}} \sum_ {k = 1} ^ {j} | u _ {k} | ^ {2} - \frac {j r}{\sqrt {j (j + 1)}} | u _ {j + 1} | ^ {2} \\ = \frac {1}{d} \sum_ {k \neq j + 1} | u _ {k} | ^ {2} + \frac {r}{\sqrt {j (j + 1)}} \sum_ {k = 1} ^ {j} | u _ {k} | ^ {2} + \left(\frac {1}{d} - \frac {j r}{\sqrt {j (j + 1)}}\right) | u _ {j + 1} | ^ {2} \tag {B.21} \\ \end{array}
$$
Thus, we need to ensure that
$$
\frac {1}{d} - \frac {j r}{\sqrt {j (j + 1)}} \geq 0 \tag {B.22}
$$
for all  $j = 1, \dots, d - 1$ . Because the function  $x \mapsto \sqrt{x(x + 1)} / x$  is decreasing, it obtains the smallest value from  $x = d - 1$ . Therefore, for  $r \leq 1 / \sqrt{d(d - 1)}$ , we readily obtain that  $u^{\dagger}(\mathbf{R} + r\mathbf{Z})u \geq 0$  for all  $u \in \mathbb{C}^d$ .
For  $\mathbf{Z} = -\frac{1}{\sqrt{j(j + 1)}}\left(\Delta_{11} + \dots + \Delta_{jj} - j\Delta_{(j + 1)(j + 1)}\right)$ , we have:
$$
\begin{array}{l} u ^ {\dagger} (\mathbf {R} + r \mathbf {Z}) u = \frac {1}{d} \sum_ {k = 1} ^ {d} | u _ {k} | ^ {2} - \frac {r}{\sqrt {j (j + 1)}} \sum_ {k = 1} ^ {j} | u _ {k} | ^ {2} + \frac {j r}{\sqrt {j (j + 1)}} | u _ {j + 1} | ^ {2} \\ = \left(\frac {1}{d} - \frac {r}{\sqrt {j (j + 1)}}\right) \sum_ {k = 1} ^ {j} | u _ {k} | ^ {2} + \frac {1}{d} \sum_ {k = j + 1} ^ {d} | u _ {k} | ^ {2} + \frac {j r}{\sqrt {j (j + 1)}} | u _ {j + 1} | ^ {2} \tag {B.23} \\ \end{array}
$$
$$
\frac {1}{d} - \frac {r}{\sqrt {j (j + 1)}} \geq 0 \tag {B.24}
$$
for all  $j = 1, \dots, d - 1$ . Because it holds that
$$
\frac {1}{d} - \frac {r}{\sqrt {j (j + 1)}} \geq \frac {1}{d} - \frac {j r}{\sqrt {j (j + 1)}} \tag {B.25}
$$
we obtain the inequality for free by the previous case, i.e., for  $r \leq 1 / \sqrt{d(d - 1)}$ .
Therefore, for
$$
r = \min  \left\{\frac {1}{\sqrt {d (d - 1)}}, \frac {\sqrt {2}}{d} \right\} \tag {B.26}
$$
we readily obtain that  $u^{\dagger}(\mathbf{R} + r\mathbf{Z})u \geq 0$  for all  $u \in \mathbb{C}^d$ , and our proof is complete.
C Omitted proofs from Section 5
Our aim in this appendix will be to prove the basic guarantees of (3MW) with payoff-based feedback. The structure of this appendix shadows that of Section 5 and is broken into two parts, depending on the specific type of input available to the players. The only point of departure is the energy inequality of Lemma 1, which is common to both algorithms, and which we restate and prove below:
Proof. By the definition of  $D$ , it is easy to see that for  $\mathbf{P} \in \mathcal{X}$  and  $\mathbf{X}, \mathbf{X}' \in \mathrm{ri}(\mathcal{X})$ , we have
$$
D (\mathbf {P}, \mathbf {X} ^ {\prime}) = D (\mathbf {P}, \mathbf {X}) + D (\mathbf {X}, \mathbf {X} ^ {\prime}) + \operatorname {t r} \left[ \left(\log \mathbf {X} ^ {\prime} - \log \mathbf {X}\right) (\mathbf {X} - \mathbf {P}) \right] \tag {C.1}
$$
Since  $\nabla h(\mathbf{X}) = \log \mathbf{X} + \mathbf{I}$ , the above equality can be written as:
$$
D (\mathbf {P}, \mathbf {X} ^ {\prime}) = D (\mathbf {P}, \mathbf {X}) + D (\mathbf {X}, \mathbf {X} ^ {\prime}) + \operatorname {t r} \left[ \left(\nabla h \left(\mathbf {X} ^ {\prime}\right) - \nabla h (\mathbf {X})\right) (\mathbf {X} - \mathbf {P}) \right] \tag {C.2}
$$
Setting  $\mathbf{X}$  as  $\mathbf{X}_{t + 1}$ , and  $\mathbf{X}'$  as  $\mathbf{X}_t$ , and invoking the easily verifiable fact that  $\nabla h(\mathbf{X}_{t + 1}) - \nabla h(\mathbf{X}_t) = \gamma_t\hat{\mathbf{V}}_t$ , we get
$$
D (\mathbf {P}, \mathbf {X} _ {t}) = D (\mathbf {P}, \mathbf {X} _ {t + 1}) + D (\mathbf {X} _ {t + 1}, \mathbf {X} _ {t}) - \gamma_ {t} \operatorname {t r} [ \hat {\mathbf {V}} _ {t} (\mathbf {X} _ {t + 1} - \mathbf {P}) ] \tag {C.3}
$$
$$
\begin{array}{l} D (\mathbf {P}, \mathbf {X} _ {t + 1}) = D (\mathbf {P}, \mathbf {X} _ {t}) - D (\mathbf {X} _ {t + 1}, \mathbf {X} _ {t}) + \gamma_ {t} \operatorname {t r} [ \hat {\mathbf {V}} _ {t} (\mathbf {X} _ {t + 1} - \mathbf {P}) ] \\ \leq D \left(\mathbf {P}, \mathbf {X} _ {t}\right) - \frac {1}{2} \left\| \mathbf {X} _ {t + 1} - \mathbf {X} _ {t} \right\| _ {F} ^ {2} + \gamma_ {t} \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} \left(\mathbf {X} _ {t + 1} - \mathbf {P}\right) \right] \\ = D \left(\mathbf {P}, \mathbf {X} _ {t}\right) - \frac {1}{2} \left\| \mathbf {X} _ {t + 1} - \mathbf {X} _ {t} \right\| _ {F} ^ {2} + \gamma_ {t} \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} \left(\mathbf {X} _ {t} - \mathbf {P}\right) \right] + \gamma_ {t} \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} \left(\mathbf {X} _ {t + 1} - \mathbf {X} _ {t}\right) \right] \\ \leq D \left(\mathbf {P}, \mathbf {X} _ {t}\right) + \gamma_ {t} \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} \left(\mathbf {X} _ {t} - \mathbf {P}\right) \right] + \frac {\gamma_ {t} ^ {2}}{2} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \tag {C.4} \\ \end{array}
$$
where the first inequality holds due to Lemma A.2, and in the last step we used that  $\| \cdot \| _F$  is an inner-product norm on  $\mathcal{X}$ , so
$$
\frac {1}{2} \left\| \mathbf {X} _ {t + 1} - \mathbf {X} _ {t} \right\| _ {F} ^ {2} + \frac {\gamma_ {t} ^ {2}}{2} \left\| \hat {\mathbf {V}} _ {t} \right\| _ {F} ^ {2} \geq \gamma_ {t} \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} \left(\mathbf {X} _ {t + 1} - \mathbf {X} _ {t}\right) \right] \tag {C.5}
$$
This concludes our proof.
With this template inequality in hand, we proceed with the guarantees of (3MW) in the next sections.
C.1. Learning with mixed payoff observations. We begin with the statistics of the 2-point sampler (2PE), which we restate below:
Proof. We prove each part separately.
(i) Let  $\Xi_{i,t}^{(+)}$  and  $\Xi_{i,t}^{(-)}$  be defined for all players  $i\in \{1,2\}$  as
$$
\begin{array}{l} \Xi_ {i, t} ^ {(+)} = \left(\mathbf {X} _ {i, t} ^ {(\delta)} + s _ {i, t} \delta_ {t} \mathbf {Z} _ {i, t}\right) - \mathbf {X} _ {i, t} \\ = s _ {i, t} \delta_ {t} \mathbf {Z} _ {i, t} + \frac {\delta_ {t}}{r _ {i}} \left(\mathbf {R} _ {i} - \mathbf {X} _ {i, t}\right) = \delta_ {t} \left[ s _ {i, t} \mathbf {Z} _ {i, t} + \frac {1}{r _ {i}} \left(\mathbf {R} _ {i} - \mathbf {X} _ {i, t}\right) \right] \tag {C.6} \\ \end{array}
$$
$$
\Xi_ {i, t} ^ {(-)} = \left(\mathbf {X} _ {i, t} ^ {(\delta)} - s _ {i, t} \delta_ {t} \mathbf {Z} _ {i, t}\right) - \mathbf {X} _ {i, t} = \delta_ {t} \left[ - s _ {i, t} \mathbf {Z} _ {i, t} + \frac {1}{r _ {i}} \left(\mathbf {R} _ {i} - \mathbf {X} _ {i, t}\right) \right] \tag {C.7}
$$
Taking a first-order Taylor expansion of  $u_{i}$ , we obtain:
$$
u _ {i} \left(\mathbf {X} _ {t} ^ {(\delta)} + s _ {t} \delta_ {t} \mathbf {Z} _ {t}\right) = u _ {i} \left(\mathbf {X} _ {t}\right) + \sum_ {j \in \mathcal {N}} \operatorname {t r} \left[ \nabla_ {\mathbf {X} _ {j} ^ {\top}} u _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \boldsymbol {\Xi} _ {j, t} ^ {(+)} \right] + R _ {2} \left(\boldsymbol {\Xi} _ {t} ^ {(+)}\right) \tag {C.8a}
$$
$$
u _ {i} \left(\mathbf {X} _ {t} ^ {(\delta)} - s _ {t} \delta_ {t} \mathbf {Z} _ {t}\right) = u _ {i} \left(\mathbf {X} _ {t}\right) + \sum_ {j \in \mathcal {N}} \operatorname {t r} \left[ \nabla_ {\mathbf {X} _ {j} ^ {\top}} u _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \boldsymbol {\Xi} _ {j, t} ^ {(-)} \right] + R _ {2} \left(\boldsymbol {\Xi} _ {t} ^ {(-)}\right) \tag {C.8b}
$$
where  $R_{2}(\cdot)$  is the 2nd order Taylor remainder. Now, for  $j \neq i \in \mathcal{N}$ , since  $s_{i,t}$  is zero-mean and independent of any other process:
$$
\mathbb {E} \left[ \operatorname {t r} \left[ \nabla_ {\mathbf {X} _ {j} ^ {\top}} u _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \left(\Xi_ {j, t} ^ {(+)} - \Xi_ {j, t} ^ {(-)}\right) \right] s _ {i, t} \mathbf {Z} _ {i, t} \mid \mathcal {F} _ {t} \right] = 0 \tag {C.9}
$$
and using that  $\Xi_{i,t}^{(+)} - \Xi_{i,t}^{(-)} = 2s_{i,t}\delta_t\mathbf{Z}_{i,t}$ , we have:
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {t r} \left[ \nabla_ {\mathbf {X} _ {i} ^ {\top}} u _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \left(\Xi_ {i, t} ^ {(+)} - \Xi_ {i, t} ^ {(-)}\right) \right] s _ {i, t} \mathbf {Z} _ {i, t} \mid \mathcal {F} _ {t} \right] = \mathbb {E} \left[ \operatorname {t r} \left[ \mathbf {V} _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \left(2 s _ {i, t} \delta_ {t} \mathbf {Z} _ {i, t}\right) \right] s _ {i, t} \mathbf {Z} _ {i, t} \mid \mathcal {F} _ {t} \right] \\ = 2 \delta_ {t} \mathbb {E} \left[ \operatorname {t r} \left[ \mathbf {V} _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \mathbf {Z} _ {i, t} \right] s _ {i, t} ^ {2} \mathbf {Z} _ {i, t} \mid \mathcal {F} _ {t} \right] \\ = 2 \delta_ {t} \mathbb {E} \left[ \operatorname {t r} \left[ \mathbf {V} _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \mathbf {Z} _ {i, t} \right] \mathbf {Z} _ {i, t} \mid \mathcal {F} _ {t} \right] \\ = \frac {2 \delta_ {t}}{D _ {i}} \sum_ {W \in \mathcal {E} _ {i}} \operatorname {t r} \left[ \mathbf {V} _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} W \right] W \\ = \frac {2 \delta_ {t}}{D _ {i}} \operatorname {p r o j} _ {\mathcal {E} _ {i}} \left(\mathbf {V} _ {i} \left(\mathbf {X} _ {t}\right)\right) = \frac {2 \delta_ {t}}{D _ {i}} \mathbf {V} _ {i} \left(\mathbf {X} _ {t}\right) \quad (C. 1 0) \\ \end{array}
$$
where in the last step, with a slight abuse of notation, we identify  $\mathrm{proj}_{\mathcal{E}_i}(\mathbf{V}_i(\mathbf{X}_t))$  with  $\mathbf{V}_i(\mathbf{X}_t)$ . The reason for this is that we apply the differential operator  $\mathbf{V}_i(\mathbf{X}_t)$  only on elements of  $\mathcal{X}_i$ , and thus, we can ignore the component of  $\mathbf{V}_i(\mathbf{X}_t)$  that is orthogonal to  $\mathrm{span}(\mathcal{E}_i)$ .
$$
\left| R _ {2} \left(\Xi_ {t} ^ {(+)}\right) \right| \leq \frac {L}{2} \| \Xi_ {t} ^ {(+)} \| _ {F} ^ {2} \leq L \delta_ {t} ^ {2} \tag {C.11}
$$
and similarly, we get the same bound for  $|R_2(\Xi_t^{(-)})|$ . Therefore, in light of the above, we obtain the bound:
$$
\left\| \mathbb {E} \left[ \hat {\mathbf {V}} _ {i, t} \mid \mathcal {F} _ {t} \right] - \mathbf {V} _ {i} \left(\mathbf {X} _ {t}\right) \right\| _ {F} \leq \frac {1}{2} D _ {i} L \delta_ {t} \tag {C.12a}
$$
and, hence
$$
\left\| \mathbb {E} \left[ \hat {\mathbf {V}} _ {t} \mid \mathcal {F} _ {t} \right] - \mathbf {V} \left(\mathbf {X} _ {t}\right) \right\| _ {F} \leq \frac {\sqrt {2}}{2} D L \delta_ {t} \tag {C.12b}
$$
(ii) By the definition of  $\hat{\mathbf{V}}_{i,t}$ , we have:
$$
\begin{array}{l} \| \hat {\mathbf {V}} _ {i, t} \| _ {F} = \frac {D _ {i}}{2 \delta_ {t}} \bigg | u _ {i} (\mathbf {X} _ {t} ^ {(\delta)} + s _ {t} \delta_ {t} \mathbf {Z} _ {t}) - u _ {i} (\mathbf {X} _ {t} ^ {(\delta)} - s _ {t} \delta_ {t} \mathbf {Z} _ {t}) \bigg | \| s _ {i, t} \mathbf {Z} _ {i, t} \| _ {F} \\ \leq \frac {D _ {i}}{2 \delta_ {t}} G \| 2 s _ {t} \delta_ {t} \mathbf {Z} _ {t} \| _ {F} \leq \sqrt {2} D _ {i} G \tag {C.13} \\ \end{array}
$$
and therefore, we readily obtain that:
$$
\mathbb {E} \left[ \| \hat {\mathbf {V}} _ {i, t} \| _ {F} ^ {2} \mid \mathcal {F} _ {t} \right] \leq 2 D _ {i} ^ {2} G ^ {2} \tag {C.14}
$$
$$
\mathbb {E} \left[ \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \mid \mathcal {F} _ {t} \right] \leq 4 D ^ {2} G ^ {2} \tag {C.15}
$$
With all these technical elements in place, we are finally in a position to prove our convergence result for (3MW) run with 2-point gradient estimators. As before, we restate our result below for convenience:
Proof. Let  $\mathbf{X}^* \in \mathcal{X}$  be a NE point. By Lemma 1 for  $\mathbf{P} = \mathbf{X}^*$ , and setting  $F_t \coloneqq D(\mathbf{X}^*, \mathbf{X}_t)$  for all  $t = 1, 2, \ldots$ , we have
$$
F _ {t + 1} \leq F _ {t} + \gamma_ {t} \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} ^ {\dagger} \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] + \frac {\gamma_ {t} ^ {2}}{2} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \tag {C.16}
$$
or, equivalently
$$
\left. \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} ^ {\dagger} \left(\mathbf {X} ^ {*} - \mathbf {X} _ {t}\right) \right] \leq \frac {1}{\gamma_ {t}} \left(F _ {t} - F _ {t + 1}\right) + \frac {\gamma_ {t}}{2} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \right. \tag {C.17}
$$
Summing over the whole sequence  $t = 1,\ldots ,T$  , we get:
$$
\sum_ {t = 1} ^ {T} \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} ^ {\dagger} \left(\mathbf {X} ^ {*} - \mathbf {X} _ {t}\right) \right] \leq \sum_ {t = 1} ^ {T} \frac {1}{\gamma_ {t}} \left(F _ {t} - F _ {t + 1}\right) + \frac {1}{2} \sum_ {t = 1} ^ {T} \gamma_ {t} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \tag {C.18}
$$
which can be rewritten by setting  $\gamma_0 = \infty$ , as:
$$
\sum_ {t = 1} ^ {T} \operatorname {t r} \left[ \hat {\mathbf {V}} _ {t} ^ {\dagger} \left(\mathbf {X} ^ {*} - \mathbf {X} _ {t}\right) \right] \leq \sum_ {t = 1} ^ {T} F _ {t} \left(\frac {1}{\gamma_ {t}} - \frac {1}{\gamma_ {t - 1}}\right) + \frac {1}{2} \sum_ {t = 1} ^ {T} \gamma_ {t} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \tag {C.19}
$$
Decomposing  $\hat{\mathbf{V}}_t$  as
$$
\hat {\mathbf {V}} _ {t} = \mathbf {V} \left(\mathbf {X} _ {t}\right) + b _ {t} + U _ {t} \tag {C.20}
$$
(i)  $b_{t} = \mathbb{E}\left[\hat{\mathbf{V}}_{t}\mid \mathcal{F}_{t}\right] - \mathbf{V}(\mathbf{X}_{t})$
(ii)  $U_{t} = \hat{\mathbf{V}}_{t} - \mathbb{E}\left[\hat{\mathbf{V}}_{t}\mid \mathcal{F}_{t}\right]$
equation (D.18) becomes:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \operatorname {t r} \left[ \mathbf {V} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \left(\mathbf {X} ^ {*} - \mathbf {X} _ {t}\right) \right] \leq \sum_ {t = 1} ^ {T} F _ {t} \left(\frac {1}{\gamma_ {t}} - \frac {1}{\gamma_ {t - 1}}\right) + \frac {1}{2} \sum_ {t = 1} ^ {T} \gamma_ {t} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \\ + \sum_ {t = 1} ^ {T} \mathrm {t r} [ b _ {t} ^ {\dagger} (\mathbf {X} _ {t} - \mathbf {X} ^ {*}) ] + \sum_ {t = 1} ^ {T} \mathrm {t r} [ U _ {t} ^ {\dagger} (\mathbf {X} _ {t} - \mathbf {X} ^ {*}) ] \\ \leq \sum_ {t = 1} ^ {T} F _ {t} \left(\frac {1}{\gamma_ {t}} - \frac {1}{\gamma_ {t - 1}}\right) + \frac {1}{2} \sum_ {t = 1} ^ {T} \gamma_ {t} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \\ + 4 \sum_ {t = 1} ^ {T} \| b _ {t} \| _ {F} + \sum_ {t = 1} ^ {T} \operatorname {t r} \left[ U _ {t} ^ {\dagger} \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] \tag {C.21} \\ \end{array}
$$
The left-hand side (LHS) of (D.20) gives:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \operatorname {t r} \left[ \mathbf {V} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \left(\mathbf {X} ^ {*} - \mathbf {X} _ {t}\right) \right] = \sum_ {t = 1} ^ {T} \operatorname {t r} \left[ \mathbf {V} _ {1} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \left(\mathbf {X} _ {1} ^ {*} - \mathbf {X} _ {1, t}\right) \right] + \sum_ {t = 1} ^ {T} \operatorname {t r} \left[ \mathbf {V} _ {2} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \left(\mathbf {X} _ {2} ^ {*} - \mathbf {X} _ {2, t}\right) \right] \\ = \sum_ {t = 1} ^ {T} \left(u _ {1} \left(\mathbf {X} _ {1} ^ {*}, \mathbf {X} _ {2, t}\right) - u _ {1} \left(\mathbf {X} _ {t}\right)\right) + \sum_ {t = 1} ^ {T} \left(u _ {2} \left(\mathbf {X} _ {1, t}, \mathbf {X} _ {2} ^ {*}\right) - u _ {2} \left(\mathbf {X} _ {t}\right)\right) \\ = \sum_ {t = 1} ^ {T} \left(\mathcal {L} \left(\mathbf {X} _ {1} ^ {*}, \mathbf {X} _ {2, t}\right) - \mathcal {L} \left(\mathbf {X} _ {1, t}, \mathbf {X} _ {2} ^ {*}\right)\right) \tag {C.22} \\ \end{array}
$$
Hence, dividing by  $T$ , we get:
$$
\mathcal {L} \left(\mathbf {X} _ {1} ^ {*}, \bar {\mathbf {X}} _ {2, T}\right) - \mathcal {L} \left(\bar {\mathbf {X}} _ {1, T}, \mathbf {X} _ {2} ^ {*}\right) \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \operatorname {t r} \left[ \mathbf {V} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \left(\mathbf {X} ^ {*} - \mathbf {X} _ {t}\right) \right] \tag {C.23}
$$
$$
\begin{array}{l} \operatorname {G a p} _ {\mathcal {L}} \left(\bar {\mathbf {X}} _ {T}\right) \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \operatorname {t r} \left[ \mathbf {V} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \left(\mathbf {X} ^ {*} - \mathbf {X} _ {t}\right) \right] \\ \leq \frac {1}{T} \sum_ {t = 1} ^ {T} F _ {t} \left(\frac {1}{\gamma_ {t}} - \frac {1}{\gamma_ {t - 1}}\right) + \frac {1}{2 T} \sum_ {t = 1} ^ {T} \gamma_ {t} \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \\ + \frac {4}{T} \sum_ {t = 1} ^ {T} \| b _ {t} \| _ {F} + \frac {1}{T} \sum_ {t = 1} ^ {T} \operatorname {t r} \left[ U _ {t} ^ {\dagger} \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] \tag {C.24} \\ \end{array}
$$
Now, we focus on the right-hand side (RHS) of (D.20). Specifically, we have:
$$
\mathbb {E} \left[ \operatorname {t r} \left[ U _ {t} ^ {\dagger} \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] \right] = \mathbb {E} \left[ \mathbb {E} \left[ \operatorname {t r} \left[ U _ {t} ^ {\dagger} \left(\mathbf {X} _ {t} - \mathbf {X} ^ {*}\right) \right] \mid \mathcal {F} _ {t} \right] \right] = 0 \tag {C.25}
$$
since  $\mathbf{X}_t$  is  $\mathcal{F}_t$ -measurable and  $\mathbb{E}[U_t\mid \mathcal{F}_t] = 0$ .
Moreover, by Proposition 3, we have:
$$
\left\| b _ {i, t} \right\| _ {F} = \left\| \mathbb {E} \left[ \hat {\mathbf {V}} _ {i, t} \mid \mathcal {F} _ {t} \right] - \mathbf {V} _ {i} (\mathbf {X} _ {t}) \right\| _ {F} \leq 2 D _ {i} L \delta_ {t} \tag {C.26a}
$$
$$
\mathbb {E} \left[ \left\| \hat {\mathbf {V}} _ {i, t} \right\| _ {F} ^ {2} \right] = \mathbb {E} \left[ \mathbb {E} \left[ \left\| \hat {\mathbf {V}} _ {i, t} \right\| _ {F} ^ {2} \mid \mathcal {F} _ {t} \right] \right] \leq 4 D _ {i} ^ {2} G ^ {2} \tag {C.26b}
$$
Hence, taking expectation in (D.20), we obtain:
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {G a p} _ {\mathcal {L}} \left(\bar {\mathbf {X}} _ {T}\right) \right] \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ F _ {t} \right] \left(\frac {1}{\gamma_ {t}} - \frac {1}{\gamma_ {t - 1}}\right) + \frac {1}{2 T} \sum_ {t = 1} ^ {T} \gamma_ {t} \mathbb {E} \left[ \| \hat {\mathbf {V}} _ {t} \| _ {F} ^ {2} \right] + \frac {4}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \| b _ {t} \| _ {F} \right] \\ \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} [ F _ {t} ] \left(\frac {1}{\gamma_ {t}} - \frac {1}{\gamma_ {t - 1}}\right) + \frac {8 D ^ {2} G ^ {2}}{T} \sum_ {t = 1} ^ {T} \gamma_ {t} + \frac {1 6 D L}{T} \sum_ {t = 1} ^ {T} \delta_ {t} \tag {C.27} \\ \end{array}
$$
Setting  $\gamma_{t} = \gamma$  and  $\delta_t = \delta$ , we obtain:
$$
\mathbb {E} \left[ \operatorname {G a p} _ {\mathcal {L}} \left(\bar {\mathbf {X}} _ {T}\right) \right] \leq \frac {F _ {1}}{\gamma T} + 8 D ^ {2} G ^ {2} \gamma + 1 6 D L \delta \tag {C.28}
$$
and finally, noting that
$$
F _ {1} = D \left(\mathbf {X} ^ {*}, \mathbf {X} _ {1}\right) \leq \log \left(d _ {1} d _ {2}\right) \tag {C.29}
$$
we get:
$$
\mathbb {E} \left[ \operatorname {G a p} _ {\mathcal {L}} \left(\bar {\mathbf {X}} _ {T}\right) \right] \leq \frac {H}{\gamma T} + 8 D ^ {2} G ^ {2} \gamma + 1 6 D L \delta \tag {C.30}
$$
for  $H = \log (d_1d_2)$ . Hence, after tuning  $\gamma$  to optimize this last expression, our result follows by setting  $\gamma = \sqrt{\frac{H}{8TD^2G^2}}$  and  $\delta = \sqrt{\frac{G^2H}{8L^2T}}$ .
C.2. Learning with bandit feedback. We now proceed with the more arduous task of proving the bona fide, bandit guarantees of (3MW) with 1-point, stochastic, payoff-based feedback. The key difference with our previous analysis lies in the different statistical properties of the 1-point estimator (1PE). The relevant result that we will need is restated below:
(i) Let  $\Xi_{i,t}$  be defined for all players  $i\in \mathcal{N}$
$$
\boldsymbol {\Xi} _ {i, t} = \mathbf {X} _ {i, t} ^ {(\delta)} - \mathbf {X} _ {i, t} = \delta_ {t} \mathbf {Z} _ {i, t} + \frac {\delta_ {t}}{r _ {i}} \left(\mathbf {R} _ {i} - \mathbf {X} _ {i, t}\right) = \delta_ {t} \left[ \mathbf {Z} _ {i, t} + \frac {1}{r _ {i}} \left(\mathbf {R} _ {i} - \mathbf {X} _ {i, t}\right) \right] \tag {C.31}
$$
$$
u _ {i} \left(\mathbf {X} _ {t} ^ {(\delta)} + \delta_ {t} \mathbf {Z} _ {t}\right) = u _ {i} \left(\mathbf {X} _ {t}\right) + \sum_ {j \in \mathcal {N}} \operatorname {t r} \left[ \nabla_ {\mathbf {X} _ {j} ^ {\top}} u _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \boldsymbol {\Xi} _ {j, t} \right] + R _ {2} \left(\boldsymbol {\Xi} _ {t}\right) \tag {C.32}
$$
Since  $\mathbb{E}[U_i(\omega_t) \mid \mathcal{F}_t, \mathbf{Z}_t] = u(\mathbf{X}_t^{(\delta)} + \delta_t \mathbf{Z}_t)$ , combining it with (D.3), we readily get:
$$
\begin{array}{l} \mathbb {E} \left[ \hat {\mathbf {V}} _ {i, t} \mid \mathcal {F} _ {t}, \mathbf {Z} _ {t} \right] = \frac {D _ {i}}{\delta_ {t}} u _ {i} \left(\mathbf {X} _ {t} ^ {(\delta)} + \delta_ {t} \mathbf {Z} _ {t}\right) \mathbf {Z} _ {i, t} (C.33) \\ = \frac {D _ {i}}{\delta_ {t}} u _ {i} \left(\mathbf {X} _ {t}\right) \mathbf {Z} _ {i, t} + \frac {D _ {i}}{\delta_ {t}} \sum_ {j \in \mathcal {N}} \operatorname {t r} \left[ \nabla_ {\mathbf {X} _ {j} ^ {\top}} u _ {i} \left(\mathbf {X} _ {t}\right) ^ {\dagger} \boldsymbol {\Xi} _ {j, t} \right] \mathbf {Z} _ {i, t} + \frac {D _ {i}}{\delta_ {t}} R _ {2} \left(\boldsymbol {\Xi} _ {t}\right) \mathbf {Z} _ {i, t} (C.34) \\ \end{array}
$$
Now, because  $\mathbb{E}\big[\mathbf{Z}_{i,t} \mid \mathcal{F}_t\big] = 0$  and  $\mathbf{Z}_{i,t}$  is sampled independent of any other process, we have:
$$
\mathbb {E} \left[ u _ {i} \left(\mathbf {X} _ {t}\right) \mathbf {Z} _ {i, t} \mid \mathcal {F} _ {t} \right] = u _ {i} \left(\mathbf {X} _ {t}\right) \mathbb {E} \left[ \mathbf {Z} _ {i, t} \mid \mathcal {F} _ {t} \right] = 0 \tag {C.35}
$$
and for  $j\neq i\in \mathcal{N}$