SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In quantum mechanics, a  $d$ -dimensional quantum state is described by a unit vector  $v = (v_0, v_1, \ldots, v_{d-1})^{\intercal}$ , usually denoted as  $|v\rangle$  with the Dirac symbol  $|\cdot\rangle$ , in a complex Hilbert space  $\mathbb{C}^d$ . The computational basis of  $\mathbb{C}^d$  is defined as  $\{|i\rangle\}_{i=0}^{d-1}$ , where  $|i\rangle = (0, \ldots, 0, 1, 0, \ldots, 0)^{\intercal}$  with the  $i$ -th (0-indexed) entry being 1 and other entries being 0. The inner product of quantum states  $|v\rangle$  and  $|w\rangle$  is defined by  $\langle v | w \rangle = \sum_{i=0}^{d-1} v_i^* w_i$ , where  $z^*$  denotes the conjugate of complex number  $z$ . The norm of  $|v\rangle$  is defined by  $\| |v\rangle \| = \sqrt{\langle v | v \rangle}$ . The tensor product of quantum states  $|v\rangle \in \mathbb{C}^{d_1}$  and  $|w\rangle \in \mathbb{C}^{d_2}$  is defined by  $|v\rangle \otimes |w\rangle = (v_0 w_0, v_0 w_1, \ldots, v_{d_1 - 1} w_{d_2 - 1})^{\intercal} \in \mathbb{C}^{d_1 d_2}$ , denoted as  $|v\rangle |w\rangle$  for short.
A quantum bit (qubit for short) is a quantum state  $|\psi \rangle$  in  $\mathbb{C}^2$ , which can be written as  $|\psi \rangle = \alpha |0\rangle + \beta |1\rangle$  with  $|\alpha|^2 + |\beta|^2 = 1$ . An  $n$ -qubit quantum state is in the tensor product space of  $n$  Hilbert spaces  $\mathbb{C}^2$ , i.e.,  $(\mathbb{C}^2)^{\otimes n} = \mathbb{C}^{2^n}$  with the computational basis  $\{|0\rangle, |1\rangle, \ldots, |2^n - 1\rangle\}$ . To obtain classical information from an  $n$ -qubit quantum state  $|v\rangle$ , we measure  $|v\rangle$  on the computational basis and obtain outcome  $i$  with probability  $p(i) = |\langle i|v\rangle|^2$  for every  $0 \leq i < 2^n$ . The evolution of a quantum state  $|v\rangle$  is described by a unitary transformation  $U \colon |v\rangle \mapsto U|v\rangle$  such that  $UU^\dagger = U^\dagger U = I$ , where  $U^\dagger$  is the Hermitian conjugate of  $U$ , and  $I$  is the identity operator. A quantum gate is a unitary transformation that acts on 1 or 2 qubits, and a quantum circuit is a sequence of quantum gates.
Throughout this paper, we assume the quantum oracle  $\mathcal{O}_{\mathbf{A}}$  for a matrix  $\mathbf{A} \in \mathbb{R}^{m \times n}$ , which is a unitary operator such that for every row index  $i \in [m]$  and column index  $j \in [n]$ ,  $\mathcal{O}_{\mathbf{A}}|i\rangle |j\rangle |0\rangle = |i\rangle |j\rangle |A_{i,j}\rangle$ . Intuitively, the oracle  $\mathcal{O}_{\mathbf{A}}$  reads the entry  $A_{i,j}$  and stores it in the third register; this is potentially stronger than the classical counterpart when the query is a linear combination of basis vectors, e.g.,  $\sum_{k} \alpha_{k}|i_{k}\rangle|j_{k}\rangle$  with  $\sum_{k} |\alpha_{k}|^{2} = 1$ . This is known as the superposition principle in quantum computing.
Note that this input model for matrices is commonly used in quantum algorithms, e.g., linear system solvers [28] and semidefinite programming solvers [7, 50, 8, 49].
A quantum (query) algorithm  $\mathcal{A}$  is a quantum circuit that consists of unitary operators  $G_{1}, G_{2}, \ldots, G_{T}$ , each of which is either a quantum gate or a quantum oracle. The quantum time complexity of  $\mathcal{A}$  is measured by the number  $T$  of quantum gates and quantum oracles in  $\mathcal{A}$ . The execution of  $\mathcal{A}$  on  $n$  qubits starts with quantum state  $|0\rangle^{\otimes n}$ , then it performs unitary operators  $G_{1}, G_{2}, \ldots, G_{T}$  on the quantum state in this order, resulting in the quantum state  $|\phi\rangle = G_{T} \ldots G_{2} G_{1}|0\rangle^{\otimes n}$ . Finally, we measure  $|\phi\rangle$  on the computational basis  $|i\rangle$  for  $0 \leq i < 2^n$ , giving a classical output  $i$  with probability  $|\langle i|\phi\rangle|^2$ .
3 Quantum Algorithm for Online Zero-Sum Games by Sample-Based Optimistic Multiplicative Weight Update
Now, we present our quantum algorithm for finding an approximate Nash equilibrium for zero-sum games (Algorithm 1). This algorithm is a modification of the optimistic multiplicative weight algorithm [46], in which we use stochastic gradients to estimate true gradients. This modification utilizes the quantum advantage of Gibbs sampling (as will be shown in Section 4). It is the source of quantum speedups in the algorithm and also the reason that we call the algorithm sample-based. To this end, we first give the definition of Gibbs sampling oracles.
Definition 3.1 (Approximate Gibbs sampling oracle). Let  $p \in \mathbb{R}^n$  be an  $n$ -dimensional vector,  $\epsilon > 0$  be the approximate error. We let  $\mathcal{O}_p^{\mathrm{Gibbs}}(k,\epsilon)$  denote the oracle which produces  $k$  independent samples from a distribution that is  $\epsilon$ -close to the Gibbs distribution with parameter  $p$  in total variation distance. Here, for a random variable  $j$  taking value in  $[n]$  following the Gibbs distribution with parameter  $p$ , we have  $\mathbf{Pr}[j = l] = \exp(p_l) / \sum_{i=1}^{n} \exp(p_i)$ .
Algorithm 1 Sample-Based Optimistic Multiplicative Weight Update for Matrix Games
Input:  $\mathbf{A}\in \mathbb{R}^{m\times n}$  , additive approximation  $\varepsilon$  , approximate Gibbs sampling oracle  $\mathcal{O}^{\mathrm{Gibbs}}$  with error  $\epsilon_{\mathrm{G}}$  total episode  $T$  , learning rate  $\lambda \in (0,\sqrt{3} /6)$    
Output:  $(\hat{u},\hat{v})$  as the approximate Nash equilibrium of the matrix game A. 1: Initialize  $\hat{u}\gets \mathbf{0}_m,\hat{v}\gets \mathbf{0}_n,x_1\gets \mathbf{0}_m,y_1\gets \mathbf{0}_n.$  2: Set  $g_{1}\leftarrow x_{1},h_{1}\leftarrow y_{1}$    
3: for  $t = 1,\dots ,T$  do   
4: Get  $T$  independent samples  $i_1^t,i_2^t,\ldots ,i_T^t$  from the Gibbs sampling oracle  $\mathcal{O}_{-\mathbf{A}h_t}^{\mathrm{Gibbs}}(T,\epsilon_{\mathrm{G}}).$    
5: Choose the action  $\zeta_t = \sum_{N = 1}^T e_{i_N^t} / T$    
6: Update  $x_{t + 1}\leftarrow x_t + \lambda \zeta_t,g_{t + 1}\leftarrow x_{t + 1} + \lambda \zeta_t.\hat{u}\leftarrow \hat{u} +\frac{1}{T}\zeta_t.$    
7: Get  $T$  independent samples  $j_1^t,j_2^t,\ldots ,j_T^t$  from the Gibbs sampling oracle  $\mathcal{O}_{\mathbf{A}^\top g_t}^{\mathrm{Gibbs}}(T,\epsilon_{\mathrm{G}}).$    
8: Choose the action  $\eta_t = \sum_{N = 1}^T e_{j_N^t} / T$    
9: Update  $y_{t + 1}\leftarrow y_t + \lambda \eta_t,h_{t + 1}\leftarrow y_{t + 1} + \lambda \eta_t,\hat{v}\leftarrow \hat{v} +\frac{1}{T}\eta_t.$    
10: end for   
11: return  $(\hat{u},\hat{v})$
Suppose the zero-sum game is represented by matrix  $\mathbf{A} \in \mathbb{R}^{m \times n}$  with  $\|\mathbf{A}\| \leq 1$ . Our sample-based optimistic multiplicative weight update algorithm is given in Algorithm 1. Algorithm 1 is inspired by the classical optimistic follow-the-regularized-leader algorithm (see Appendix A for more information). In that classical algorithm, the update terms are essentially  $\mathbb{E}[\zeta_t]$  and  $\mathbb{E}[\eta_t]$ , which are computed deterministically by matrix arithmetic operations during the update. In contrast, we do this probabilistically by sampling from the Gibbs distributions and  $\zeta_t$  and  $\eta_t$  in Line 5 and Line 10 are the corresponding averages of the samples. For this to work, we need to bound the expectation of the total regret (see Appendix B) based on the RVU property (Definition A.2). Technically, the  $l_1$  variances of  $\zeta_t$  and  $\eta_t$  turn out to be significant in the analysis. To reduce the variances, we need multiple independent samples identically distributed from Gibbs distributions (see Line 4 and Line 7 in Algorithm 1). Because of the randomness from Gibbs sampling oracles, the total regret  $\mathcal{R}(T) + \mathcal{R}'(T)$  of Algorithm 1 is a random variable. Nevertheless, we can bound the total regret by  $O(\log(mn))$  with high probability as follows.
Theorem 3.2. Let  $\epsilon_{\mathrm{G}} = 1 / T$ . After  $T$  rounds of playing the zero-sum game A by Algorithm 1, the total regret will be bounded by
$$
\mathcal {R} (T) + \mathcal {R} ^ {\prime} (T) \leq 1 4 4 \lambda + \frac {3 \log (m n)}{\lambda} + 1 2,
$$
with probability at least  $2/3$ . Then, for any constant  $\lambda \in (0, \sqrt{3}/6)$ , the total regret is  $O(\log(mn))$ . Moreover, if we choose  $T = \widetilde{\Theta}(1/\varepsilon)$ , then Algorithm 1 will return an  $\varepsilon$ -approximate Nash equilibrium of the game  $\mathbf{A}$ .
The proof of Theorem 3.2 is deferred to Appendix B.2. Combining Theorem 3.2 and our fast quantum multi-Gibbs sampler in Theorem 4.2 (which will be developed in the next section), we obtain a quantum algorithm for finding an  $\varepsilon$ -approximate Nash equilibrium of zero-sum games.
Corollary 3.3. If we choose  $T = \widetilde{\Theta}(1 / \varepsilon)$  and use quantum multi-Gibbs sampling (Algorithm 2) with  $\epsilon_{\mathrm{G}} = 1 / T$ , then Algorithm 1 will return an  $\varepsilon$ -approximate Nash equilibrium in quantum time  $\widetilde{O}(\sqrt{m + n} / \varepsilon^{2.5})$ .
4 Fast Quantum Multi-Gibbs Sampling
In Algorithm 1, the vectors  $h_t$  and  $g_t$  updated in each round are used to generate independent samples from Gibbs distributions  $\mathcal{O}_{-\mathbf{A}h_t}^{\mathrm{Gibbs}}$  and  $\mathcal{O}_{\mathbf{A}^\top g_t}^{\mathrm{Gibbs}}$ . Here,  $h_t$  and  $g_t$  are supposed to be stored in classical registers. To allow quantum speedups, we store  $h_t$  and  $g_t$  in quantum-read classical-write random access memory (QRAM) [21], which is commonly used in prior work [49, 6]. Specifically, a QRAM can store/modify an array  $a_1, a_2, \ldots, a_\ell$  of classical data and provide quantum (read-only) access to them, i.e., a unitary operator  $U_{\mathrm{QRAM}}$  is given such that  $U_{\mathrm{QRAM}}: |i\rangle |0\rangle \mapsto |i\rangle |a_i\rangle$ . Without loss of generality (see Remark 4.3), suppose we have quantum oracle  $\mathcal{O}_{\mathbf{A}}$  for  $\mathbf{A} \in \mathbb{R}^{n \times n}$  with  $A_{i,j} \in [0,1]$ , and QRAM access to a vector  $z \in \mathbb{R}^n$  with  $z_i \geq 0$ . We also need the polynomial approximation of the exponential function for applying the QSVT technique [20]:
Lemma 4.1 (Polynomial approximation, Lemma 7 of [48]). Let  $\beta \geq 1$  and  $\epsilon_{P} \in (0,1/2)$ . There is a classically efficiently computable polynomial  $P_{\beta} \in \mathbb{R}[x]$  of degree  $O(\beta \log (\epsilon_{P}^{-1}))$  such that  $\left|P_{\beta}(x)\right| \leq 1$  for  $x \in [-1,1]$ , and  $\max_{x \in [-1,0]} \left|P_{\beta}(x) - \frac{1}{4} \exp(\beta x)\right| \leq \epsilon_{P}$ .
Then, we can produce multiple samples from  $\mathcal{O}_{\mathbf{Az}}^{\mathrm{Gibbs}}$  efficiently by Algorithm 2 on a quantum computer. Algorithm 2 is inspired by Hamoudi [26] about preparing multiple samples of a quantum state, with quantum access to its amplitudes. However, we do not have access to the exact values of the amplitudes, which are  $(\mathbf{Az})_i$  in our case. To resolve this issue, we develop consistent quantum amplitude estimation (see Appendix C) to estimate  $(\mathbf{Az})_i$  with a unique answer (Line 1). After having prepared an initial quantum state  $|u_{\mathrm{guess}}\rangle$ , we use quantum singular value decomposition [20] to correct the tail amplitudes (Line 6), and finally obtain the desired quantum state  $|\tilde{u}_{\mathrm{Gibbs}}\rangle$  by quantum amplitude amplification [9] (Line 7). We have the following (see Appendix D for its proof):
Theorem 4.2 (Fast quantum multi-Gibbs sampling). For  $k \in [n]$ , if we set  $\epsilon_{P} = \Theta(k\epsilon_{G}^{2}/n)$ , then Algorithm 2 will produce  $k$  independent and identical distributed samples from a distribution that is  $\epsilon_{G}$ -close to  $\mathcal{O}_{\mathbf{A}z}^{\mathrm{Gibbs}}$  in total variation distance, in quantum time  $\widetilde{O}(\beta \sqrt{nk})$ .
Remark 4.3. If  $\mathbf{A} \in \mathbb{R}^{m \times n}$  is not a square matrix, then by adding  $0$ 's we can always enlarge  $\mathbf{A}$  to an  $(m + n)$ -dimensional square matrix. For  $\mathcal{O}_{-A h_t}^{\mathrm{Gibbs}}$  as required in Algorithm 1, we note that  $\mathcal{O}_{(1 - A)h_t}^{\mathrm{Gibbs}}$  indicates the same distribution as  $\mathcal{O}_{-A h_t}^{\mathrm{Gibbs}}$ , where  $\mathbf{1}$  has the same size as  $\mathbf{A}$  with all entries being  $1$ . From the above discussion, we can always convert  $\mathbf{A}$  to another matrix satisfying our assumption, i.e., with entries in the range  $[0,1]$ .
Remark 4.4. The description of the unitary operator  $U^{\mathrm{exp}}$  defined by the polynomial  $P_{2\beta}$  can be classically computed to precision  $\epsilon_P$  in time  $\widetilde{O}(\beta^3)$  by Haah [25], which is  $\widetilde{O}(1 / \varepsilon^3)$  in our case as  $\beta \leq \lambda T = \widetilde{\Theta}(1 / \varepsilon)$  is required in Corollary 3.3. This extra cost can be neglected because  $\widetilde{O}(\sqrt{m + n} / \varepsilon^{2.5})$  dominates the complexity whenever  $\varepsilon = \Omega((m + n)^{-1})$ .
Remark 4.5. Our multi-Gibbs sampler is based on maximum finding and consistent amplitude estimation, with a guaranteed worst-case performance in each round. In comparison, the dynamic Gibbs sampler in [6] maintains a hint vector, resulting in an amortized time complexity per round.
Algorithm 2 Quantum Multi-Gibbs Sampling  $\mathcal{O}_{\mathbf{Az}}^{\mathrm{Gibbs}}(k,\epsilon_{\mathrm{G}})$
Input: Quantum oracle  $\mathcal{O}_{\mathbf{A}}$  for  $\mathbf{A} \in \mathbb{R}^{n \times n}$ , QRAM access to  $z \in \mathbb{R}^n$  with  $\|z\|_1 \leq \beta$ , polynomial  $P_{2\beta}$  with parameters  $\epsilon_P$  by Lemma 4.1, number  $k$  of samples. We write  $u = \mathbf{A}z$ .
Output:  $k$  independent samples  $i_1, i_2, \ldots, i_k$ .
1: Obtain  $\mathcal{O}_{\tilde{u}}\colon |i\rangle |0\rangle \mapsto |i\rangle |\tilde{u}_i\rangle$  by consistent quantum amplitude estimation such that  $u_{i}\leq \tilde{u}_{i}\leq u_{i} + 1$
2: Find the set  $S \subseteq [n]$  of indexes of the  $k$  largest  $\tilde{u}_i$  by quantum  $k$ -maximum finding, with access to  $\mathcal{O}_{\tilde{u}}$ .
3: Obtain  $\tilde{u}_i$  for all  $i\in S$  from  $\mathcal{O}_{\tilde{u}}$ , then compute  $\tilde{u}^{*} = \min_{i\in S}\tilde{u}_{i}$  and  $W = \sum_{i\in S}\exp (\tilde{u}_i) + (n - k)\exp (\tilde{u}^*)$ .
4: for  $\ell = 1,\dots ,k$  do
5: Prepare the quantum state  $|u_{\mathrm{guess}} \rangle = \sum_{i \in S} \sqrt{\frac{\exp(\tilde{u}_i)}{W}} |i\rangle + \sum_{i \notin S} \sqrt{\frac{\exp(\tilde{u}^*)}{W}} |i\rangle$ .
6: Obtain unitary  $U^{\mathrm{exp}}$  such that  $\langle 0|\otimes aU^{\mathrm{exp}}|0\rangle^{\otimes a} = \mathrm{diag}\big(P_{2\beta}(u - \max \{\tilde{u},\tilde{u}^{*}\})\big) / 4\beta$  by QSVT.
7: Post-select  $|\tilde{u}_{\mathrm{Gibbs}}\rangle \propto \langle 0|^{\otimes a}U^{\mathrm{exp}}|u_{\mathrm{guess}}\rangle |0\rangle^{\otimes a}$  by quantum amplitude amplification.
8: Let  $i_{\ell}$  be the measurement outcome of  $|\tilde{u}_{\mathrm{Gibbs}}\rangle$  in the computational basis.
9: end for
0: return  $i_1, i_2, \ldots, i_k$ .
In our paper, we propose the first quantum online algorithm for zero-sum games with near-optimal regret. This is achieved by developing a sample-based stochastic version of the optimistic multiplicative weight update method [46]. Our core technical contribution is a fast multi-Gibbs sampling, which may have potential applications in other quantum computing scenarios.
Our result naturally gives rise to some further open questions. For instance: Can we improve the dependence on  $\varepsilon$  for the time complexity? And can we further explore the combination of optimistic learning and quantum computing into broader applications? Now that many heuristic quantum approaches for different machine learning problems have been realized, e.g. in Havlček et al. [29], Saggio et al. [44], Harrigan et al. [27], can fast quantum algorithms for zero-sum games be realized in the near future?
We would like to thank Kean Chen, Wang Fang, Ji Guan, Junyi Liu, Xinzhao Wang, Chenyi Zhang, and Zhicheng Zhang for helpful discussions. MG would like to thank Mingsheng Ying for valuable suggestions.
Minbo Gao was supported by the National Key R&D Program of China (2018YFA0306701) and the National Natural Science Foundation of China (61832015). Zhengfeng Ji was supported by a startup fund from Tsinghua University, and the Department of Computer Science and Technology, Tsinghua Univeristy. Tongyang Li was supported by a startup fund from Peking University, and the Advanced Institute of Information Technology, Peking University. Qisheng Wang was supported by the MEXT Quantum Leap Flagship Program (MEXT Q-LEAP) grants No. JPMXS0120319794.
[1] S. Aaronson, X. Chen, E. Hazan, S. Kale, and A. Nayak. Online learning of quantum states. In Advances in Neural Information Processing Systems, volume 31, 2018. URL https://proceedings.neurips.cc/paper/2018/file/cla3d34711ab5d85335331ca0e57f067-Paper.pdf.
[2] A. Ambainis. Variable time amplitude amplification and quantum algorithms for linear algebra problems. In Proceedings of the 29th Symposium on Theoretical Aspects of Computer
[48] J. van Apeldoorn and A. Gilyén. Quantum algorithms for zero-sum games, 2019. URL http://arxiv.org/abs/1904.03180.
[49] J. van Apeldonorn and A. Gilyén. Improvements in quantum SDP-solving with applications. In Proceedings of the 46th International Colloquium on Automata, Languages, and Programming, volume 132, pages 99:1-99:15, 2019. URL https://doi.org/10.4230/LIPICS.ICALP.2019.99.
[50] J. van Apeldonoorn, A. Gilyén, S. Gribling, and R. de Wolf. Quantum SDP-solvers: Better upper and lower bounds. In Proceedings of the 58th Annual Symposium on Foundations of Computer Science, pages 403-414, 2017. URL https://doi.org/10.1109/FOCS.2017.44.
[51] J. von Neumann. Zur theorie der gesellschaftsspiele. Mathematische Annalen, 100(1):295-320, 1928.
[52] F. Yang, J. Jiang, J. Zhang, and X. Sun. Revisiting online quantum state learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6607-6614, 2020.
A Revisit of Optimistic Online Learning
In this section, we briefly review some important properties in the classical optimistic online learning algorithms. Some of the propositions in this section will be frequently used in the proof of the regret bound.
For convenience, we will use  $\psi(\cdot)$  to denote the negative entropy function, i.e.,  $\psi: \Delta_n \to \mathbb{R}$ ,  $\psi(p) = \sum_{i=1}^{n} p_i \log p_i$ . Note that log stands for the natural logarithm function with base  $e$ .
For a vector norm  $\| \cdot \|$  , its dual norm is defined as:
$$
\left\| y \right\| _ {*} = \max  _ {x} \left\{\langle x, y \rangle : \| x \| \leq 1 \right\}.
$$
Proposition A.1. Let  $L$  be a vector in  $n$ -dimensional space. If  $p^* = \operatorname*{arg\min}_{p \in \Delta_n} \{\langle p, L \rangle + \psi(p)\}$ , then
$p^*$  can be written as:
$$
p ^ {*} = \frac {\exp (- L)}{\| \exp (- L) \| _ {1}}
$$
Proof. Write  $L = (L_{1}, L_{2}, \ldots, L_{n})$ . By definition, we know that  $p^{*}$  is the solution to the following convex optimization problem:
$$
\underset {p _ {1}, p _ {2}, \dots , p _ {n}} {\text {m i n i m i z e}} \quad \sum_ {i = 1} ^ {n} p _ {i} L _ {i} + \sum_ {i = 1} ^ {n} p _ {i} \log p _ {i}
$$
$$
\text {s u b j e c t} \quad \sum_ {i = 1} ^ {n} p _ {i} = 1,
$$
$$
\forall i, p _ {i} \geq 0
$$
The Lagrangian is
$$
\mathcal {L} (p, u, v) = \sum_ {i = 1} ^ {n} p _ {i} L _ {i} + \sum_ {i = 1} ^ {n} p _ {i} \log p _ {i} - \sum_ {i = 1} ^ {n} u _ {i} p _ {i} + v \left(\sum_ {i = 1} ^ {n} p _ {i} - 1\right)
$$
From KKT conditions, we know that the stationarity is:
$$
L _ {i} + 1 + \log p _ {i} - u _ {i} + v = 0. \tag {1}
$$
The complementary slackness is:
$$
u _ {i} p _ {i} = 0.
$$
The primal feasibility is
$$
\forall i, p _ {i} \geq 0; \sum_ {i = 1} ^ {n} p _ {i} = 1.
$$
The dual feasibility is
$$
u _ {i} \geq 0.
$$
If  $u_{i} \neq 0$  then  $p_{i} = 0$ , from stationarity we know  $u_{i} = -\infty$ , but that violates the dual feasibility. So we can conclude that  $u_{i} = 0$  for all  $i \in [n]$ , thus  $p_{i} \propto \exp(-L_{i})$  and the result follows.
Now we present a generalized version of the optimistic multiplicative weight algorithm called optimisitically follow the regularized leader (Opt-FTRL) in Algorithm 3. In the algorithm,  $m_t$  has the same meaning as  $m(t)$  for notation consistency.
Algorithm 3 Optimistic follow-the-regularized-leader
Input: The closed convex domain  $\mathcal{X}$
Output: Step size  $\lambda$ , loss gradient prediction  $m$ .
Initialize  $L_0\gets 0$  ,choose appropriate  $m_{1}$
$$
\begin{array}{l} \text {C h o o s e} x _ {t} = \arg \min  \left\{\lambda \langle L _ {t - 1} + m _ {t}, x \rangle + \psi (x) \right\}. \\ x \in \mathcal {X} \\ \end{array}
$$
Observe loss  $l_{t}$ , update  $L_{t} = L_{t - 1} + l_{t}$ .
Compute  $m_{t+1}$  using observations till now.
Now we study a crucial property that leads to the fast convergence of the algorithm, called the Regret bounded by Variation in Utilities (RVU in short). For simplicity, we only consider the linear loss function  $l_{t}(x) = \langle l_{t}, x \rangle$ . (There is a little abuse of notation here.)
Definition A.2 (Regret bounded by Variations in Utilities (RVU), Definition 3 in Syrgkanis et al. [46]). Consider an online learning algorithm  $\mathcal{A}$  with regret  $\mathcal{R}(T) = o(T)$ , we say that it has the property of regret bound by variation in utilities if for any linear loss sequence  $l_{1}, l_{2}, \ldots, l_{T}$ , there exists parameters  $\alpha > 0, 0 < \beta \leq \gamma$  such that the algorithm output decisions  $x_{1}, x_{2}, \ldots, x_{T}, x_{T+1}$  that satisfy:
$$
\sum_ {i = 1} ^ {T} \left\langle l _ {i}, x _ {i} \right\rangle - \min _ {x \in \mathcal {X}} \sum_ {i = 1} ^ {T} \left\langle l _ {i}, x \right\rangle \leq \alpha + \beta \sum_ {i = 1} ^ {T - 1} \left\| l _ {i + 1} - l _ {i} \right\| _ {*} ^ {2} - \gamma \sum_ {i = 1} ^ {T - 1} \left\| x _ {i + 1} - x _ {i} \right\| ^ {2},
$$
where  $\| \cdot \|_{*}$  is the dual norm of  $\| \cdot \|$ .
We do not choose the norm to be any specific one here. In fact, Syrgkanis et al. [46] have already shown that the above optimistic follow-the-regularized-leader algorithm has the RVU property with respect to any norm  $\|\cdot\|$  in which the negative entropy function  $\psi$  is 1-strongly convex. So, from Pinsker's inequality, for  $l_{2}$  norms the following result holds:
Proposition A.3 (Proposition 7 in Syrgkanis et al. [46]). If we choose  $m_t = l_{t-1}$  in the optimistic follow-the-regularized-leader algorithm with step size  $\lambda \leq 1/2$ , then it has the regret bound by variation in utilities property with the parameters  $\alpha = \log n / \lambda$ ,  $\beta = \lambda$  and  $\gamma = 1/(4\lambda)$ , where  $n$  is the dimension of  $\mathcal{X}$ .
B Regret Bound and Time Complexity of Our Algorithm
B.1 Ideal Samplers
We assume that after the execution of our algorithm, the sequences we get are  $\{(x_{t},y_{t})\}_{t = 1}^{T + 1}$  and  $\{(g_t,h_t)\}_{t = 1}^{T + 1}$ , respectively. We denote  $u_{t}\coloneqq \frac{\exp(-\mathbf{A}h_{t})}{\|\exp(-\mathbf{A}h_{t})\|_{1}}$  and  $v_{t}\coloneqq \frac{\exp(\mathbf{A}^{\top}g_{t})}{\|\exp(\mathbf{A}^{\top}g_{t})\|_{1}}$  to be the corresponding Gibbs distribution, we will first assume that the Gibbs oracle in our algorithm has no error (i.e.  $\epsilon_G = 0$ ) until Theorem B.5 is proved.
Observation B.1. The sequence  $\{u_t\}_{t=1}^T$  can be seen as the decision result of applying optimistic FTRL algorithm to the linear loss function  $\mathbf{A}\eta_t$  with linear prediction function  $\mathbf{A}\eta_{t-1}$ , and similarly for  $\{v_t\}_{t=1}^{T+1}$  with the loss function  $-\mathbf{A}^\top \zeta_t$ , the prediction function  $-\mathbf{A}^\top \zeta_{t-1}$ .
Proof. By symmetry, we only consider  $u_{t}$ . Since  $u_{t} = \frac{\exp(-\mathbf{A}h_{t})}{\|\exp(-\mathbf{A}h_{t})\|_{1}}$ , from Proposition A.1 we can write
$$
u_{t} = \operatorname *{arg  min}_{u\in \Delta_{m}}\{\langle \mathbf{A}h_{t},u\rangle +\psi (u)\} .
$$
Then we notice the iteration of Algorithm 1 gives
$$
h _ {t} = \lambda \left(\sum_ {i = 1} ^ {t - 1} \eta_ {i}\right) + \lambda \eta_ {t - 1}.
$$