SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minbo Gao
State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, China and University of Chinese Academy of Sciences, Beijing, China gmb17@tsinghua.org.cn
Zhengfeng Ji
Department of Computer Science and Technology, Tsinghua University, Beijing, China jizhengfeng@tsinghua.edu.cn
Tongyang Li
Center on Frontiers of Computing Studies, and School of Computer Science, Peking University, Beijing, China tongyangli@pku.edu.cn
Qisheng Wang
Graduate School of Mathematics, Nagoya University, Nagoya, Japan QishengWang1994@gmail.com
We propose the first online quantum algorithm for solving zero-sum games with  $\widetilde{O}(1)$  regret under the game setting. Moreover, our quantum algorithm computes an  $\varepsilon$ -approximate Nash equilibrium of an  $m \times n$  matrix zero-sum game in quantum time  $\widetilde{O}(\sqrt{m + n} / \varepsilon^{2.5})$ . Our algorithm uses standard quantum inputs and generates classical outputs with succinct descriptions, facilitating end-to-end applications. Technically, our online quantum algorithm "quantizes" classical algorithms based on the optimistic multiplicative weight update method. At the heart of our algorithm is a fast quantum multi-sampling procedure for the Gibbs sampling problem, which may be of independent interest.
Nash equilibrium is one of the most important concepts in game theory. It characterizes and predicts rational agents' behaviors in non-cooperative games, finding a vast host of applications ranging from analyzing wars [45] and designing auctions [35], to optimizing networks [43].
It was shown in Daskalakis et al. [16], Chen and Deng [12] that finding a Nash equilibrium is PPAD-hard for general games. Nevertheless, computing the Nash equilibrium for specific types of games, such as zero-sum games, is particularly interesting. A zero-sum game requires that the utility of one player is the opposite of the other's, a condition that often appears in, for example, chess games. Von Neumann's minimax theorem [51] promises that every finite two-player zero-sum game has optimal mixed strategies.
1 Throughout this paper,  $\widetilde{O} (\cdot)$  suppresses polylogarithmic factors such as  $\log (n)$  and  $\log (1 / \varepsilon)$ , and  $O^{*}(\cdot)$  hides quasi-polylogarithmic factors such as  $n^{o(1)}$  and  $(1 / \varepsilon)^{o(1)}$ .
Zero-Sum Games. For a two-player zero-sum game represented by an  $m \times n$  matrix  $\mathbf{A}$ , the Nash equilibrium is the solution pair  $(x, y)$  to the following min-max problem:
$$
\min  _ {x \in \Delta_ {m}} \max  _ {y \in \Delta_ {n}} x ^ {\mathsf {T}} \mathbf {A} y,
$$
where  $x \in \Delta_m$  and  $y \in \Delta_n$  are  $m-$  and  $n$ -dimensional probability distributions, respectively. Usually, we are satisfied with an approximate Nash equilibrium rather than an exact one. An  $\varepsilon$ -approximate Nash equilibrium is a solution pair  $(x,y)$  such that:
$$
\max  _ {y ^ {\prime} \in \Delta_ {n}} x ^ {\mathsf {T}} \mathbf {A} y ^ {\prime} - \min  _ {x ^ {\prime} \in \Delta_ {m}} x ^ {\prime \mathsf {T}} \mathbf {A} y \leq \varepsilon .
$$
Online Learning. Since the matrix  $\mathbf{A}$  of the zero-sum game usually has a large dimension in practice, it is common that we trade accuracy for space and time efficiency. Thus, online learning becomes increasingly significant in these scenarios. Online learning studies the situation when data is only available in sequential order and aims at making good decisions in this setup. In evaluating online learning algorithms, regret is an important criterion that measures how good an algorithm is compared with the optimal static loss (see more details in Section 2.3).
The idea of the online learning algorithms for zero-sum games stems from repeated play in game theory, e.g., fictitious play [11]. Specifically, we simulate the actions of two players for multiple rounds. In each round, players make decisions using a no-regret learning algorithm, considering the opponent's previous actions. For example, a famous algorithm of this type was proposed in Grigoriadis and Khachiyan [22] inspired by the exponential Hedge algorithm. The algorithm has regret  $\widetilde{O} (\sqrt{T})$  and  $T$  rounds, establishing the convergence rate of  $\widetilde{O}(1 / \sqrt{T})$ .
It takes about two decades before the next improvement in Daskalakis et al. [17] to happen, where the authors proposed a strongly-uncoupled algorithm, achieving  $\widetilde{O}(1)$  total regret if both players use the algorithm. They used the technique of minimizing non-smooth functions using smoothed approximations proposed in Nesterov [37], and this technique was later developed in Nesterov [38], Nemirovski [36] for broader classes of problems. Later, it was found in Syrgkanis et al. [46] that the optimistic multiplicative weight algorithm also leads to  $\widetilde{O}(1)$  total regret with regret bounded by variation in utilities; this algorithm was recently extended to correlated equilibria in multi-player general-sum games in Anagnostides et al. [3]. It was proved in Hsieh et al. [30] that optimistic mirror descent with a time-varying learning rate can also achieve  $\widetilde{O}(1)$  total regret for multi-players. Our quantum algorithm follows the overall idea of the optimistic multiplicative weight update and the regret bounded by variation methods [46].
Quantum Computing and Learning. Quantum computing has been rapidly advancing in recent years. Specifically, many machine learning problems are known to have significant quantum speedups, e.g., support vector machines [42], principal component analysis [34], classification [31, 33], etc. The combination of quantum computing and online learning has recently become a popular topic. For instance, online learning tools have been applied to solving semidefinite programs (SDPs) with quantum speedup in the problem dimension and the number of constraints [7, 50, 8, 49]. In addition, as an important quantum information task, the online version of quantum state learning has been systematically developed with good theoretical and empirical guarantees [1, 52, 14, 13].
For finding the Nash equilibrium of zero-sum games, a quantum algorithm was proposed in van Apeldonorn and Gilyén [48] by "quantizing" the classical algorithm in Grigoriadis and Khachiyan [22], achieving a quadratic speedup in the dimension parameters  $m$  and  $n$ . At the same time, quantum algorithms for training linear and kernel-based classifiers were proposed in Li et al. [33], which have similar problem formulations to zero-sum games. Recently, an improved quantum algorithm for zero-sum games was proposed in Bouland et al. [6] using dynamic Gibbs sampling. All of the above quantum algorithms are based on the multiplicative weight update method, and as a consequence, they all share the  $O(\sqrt{T})$  regret bound.
2The complexity given in Bouland et al. [6] is  $\widetilde{O} (\sqrt{m + n} /\varepsilon^{2.5} + 1 / \varepsilon^3)$  , wherein the former term  $\sqrt{m + n} /\varepsilon^{2.5}$  dominates the complexity. See Footnote 4 and Remark 4.4 for discussions.
3Here, we require that the input matrix  $\mathbf{A}$  satisfies  $\| \mathbf{A}\| \leq 1$  , while other works require  $|A_{i,j}|\leq 1$
Table 1: Online Algorithms for  $\varepsilon$  -Approximate Nash Equilibria of Zero-Sum Games.
Approach	Type	Regret	Update Cost Per Round	Classical/Quantum Time Complexity
[22]	Classical	O(√T)	O(m+n)	O((m+n)/ε2)
[48]	Quantum	O(√T)	O(√m+n/ε)	O(√m+n/ε3)
[6]	Quantum	O(√T)	O(√m+n/ε0.5)	O(√m+n/ε2.5)2
[46]	Classical	O(1)	O(mn)	O(mn/ε)
[10]	Classical	O(1)	O(√mn(m+n))	O(mn+√mn(m+n)/ε)
Our result3	Quantum	O(1)	O(√m+n/ε1.5)	O(√m+n/ε2.5)
1.1 Main Result
Our result in this paper establishes a positive answer to the following open question: Does there exist a learning algorithm with  $\widetilde{O}(1)$  regret allowing quantum speedups?
Inspired by the optimistic follow-the-regularized-leader algorithm proposed in Syrgkanis et al. [46], we propose a sample-based quantum online learning algorithm for zero-sum games with  $O(\log (mn))$  total regret, which is near-optimal. If we run this algorithm for  $T$  rounds, it will compute an  $\tilde{O}(1 / T)$ -approximate Nash equilibrium with high probability, achieving a quadratic speedup in dimension parameters  $m$  and  $n$ . Formally, we have the following quantum online learning algorithm:
Theorem 1.1 (Online learning for zero-sum games). Suppose  $T \leq \widetilde{O}(m + n)$ . There is a quantum online algorithm for zero-sum game  $\mathbf{A} \in \mathbb{R}^{m \times n}$  with  $\| \mathbf{A} \| \leq 1$  such that it achieves a total regret of  $O(\log(mn))$  with high probability after  $T$  rounds, while each round takes quantum time  $\widetilde{O}(T^{1.5}\sqrt{m + n})$ .
Our algorithm does not need to read all the entries of the input matrix  $\mathbf{A}$  at once. Instead, we assume that our algorithm can query its entries when necessary. The input model is described as follows:
Classically, given any  $i\in [m],j\in [n]$  , the entry  $A_{i,j}$  can be accessed in  $\widetilde{O} (1)$  time.
- Quantumly, we assume that the entry  $A_{i,j}$  can be accessed in  $\widetilde{O}(1)$  time coherently.
This is the standard quantum input model for zero-sum games adopted in previous literature [33, 48, 6]. See more details in Section 2.4.
In addition, same as prior works [48, 6], our algorithm outputs purely classical vectors with succinct descriptions because they are sparse (with at most  $T^2$  nonzero entries). Overall, using standard quantum inputs and generating classical outputs significantly facilitate end-to-end applications of our algorithm in the near term.
As a direct corollary, we can find an  $\varepsilon$ -approximate Nash equilibrium by taking  $T = \widetilde{O}(1 / \varepsilon)$ , resulting in a quantum speedup stated as follows. A detailed comparison to previous literature is presented in Table 1.
Corollary 1.2 (Computing Nash equilibrium). There is a quantum online algorithm for zero-sum game  $\mathbf{A} \in \mathbb{R}^{m \times n}$  with  $\|A\| \leq 1$  that, with high probability, computes an  $\varepsilon$ -approximate Nash equilibrium in quantum time  $\widetilde{O}(\sqrt{m + n} / \varepsilon^{2.5})$ .<sup>4</sup>
Quantum Lower Bounds. In the full version of [33], they showed a lower bound  $\Omega (\sqrt{m + n})$  for the quantum query complexity of computing an  $\varepsilon$ -approximate Nash equilibrium of zero-sum games for constant  $\varepsilon$ . Therefore, our algorithm is tight in terms of  $m$  and  $n$ .
4In fact, a condition of  $\varepsilon = \Omega((m + n)^{-1})$  is required in our quantum algorithm. Nevertheless, our claim still holds because when  $\varepsilon = O((m + n)^{-1})$ , we can directly apply the classical algorithm in Grigoriadis and Khachiyan [22] with time complexity  $\widetilde{O}((m + n) / \varepsilon^2) \leq \widetilde{O}(\sqrt{m + n} / \varepsilon^{2.5})$ .
1.2 Our Techniques
Our quantum online algorithm is a stochastic modification of the optimistic multiplicative weight update proposed by Syrgkanis et al. [46]. We choose to "quantize" the optimistic online algorithm because it has a better convergence rate for zero-sum games than general multiplicative weight update algorithms. During the update of classical online algorithms, the update term (gradient vector) is computed in linear time by arithmetic operations. However, we observe that it is not necessary to know the exact gradient vector. This motivates us to apply stochastic gradient methods for updates so that certain fast quantum samplers can be utilized here and bring quantum speedups.
Specifically, in our quantum online algorithm, we need to establish an upper bound on the expectation of the total regret of our stochastic update rule, and also deal with errors that may appear from noisy samplers. To reduce the variance of the gradient estimator, we need multiple samples (from Gibbs distributions) at a time. To this end, we develop a fast quantum multi-Gibbs sampler that produces multiple samples by preparing and measuring quantum states.
Sample-Based Optimistic Multiplicative Weight Update. Optimistic online learning adds a "prediction loss" term to the cumulative loss for regularized minimization, giving a faster convergence rate than the non-optimistic versions for zero-sum games. Arora et al. [4] surveyed the use of the multiplicative weight update method in various domains, but little was known for the optimistic learning method at that time. Daskalakis et al. [17] proposed an extragradient method that largely resembles the optimistic multiplicative weight. Syrgkanis et al. [46] gave a characterization of this update rule—RVU (Regret bounded by Variation in Utilities) property, which is very useful in proving regret bounds. Subsequently, the optimistic learning method is applied to other areas, including training GANs [18] and multi-agent learning [40].
However, when implementing the optimistic learning methods, we face a fundamental difficulty—we cannot directly access data from quantum states without measurement. To resolve this issue, we get samples from the desired distribution and use them to estimate the actual gradient. This idea is commonly seen in previous literature on quantum SDP solvers [7, 50, 8, 49]. Then we prove the regret bound (see Theorem 3.2) of our algorithm by showing that it has a property similar to the RVU property [46]. Moreover, we need multiple samples to obtain a small "variance" of the stochastic gradient (by taking the average of the samples), to ensure that the expected regret is bounded. Our fast quantum multi-Gibbs sampler produces the required samples and ensures further quantum speedups. In a nutshell, we give an algorithm (Algorithm 1) which modifies the optimistic multiplicative weight algorithm in Syrgkanis et al. [46] to fit the quantum implementation. This is the first quantum algorithm that implements optimistic online learning to the best of our knowledge.
Fast Quantum Multi-Gibbs Sampling. The key to our sample-based approach is to obtain multiple samples from the Gibbs distribution after a common preprocessing step. For a vector  $p \in \mathbb{R}^n$  with  $\max_{i \in [n]} |p_i| \leq \beta$ , a sample from the Gibbs distribution with respect to  $p$  is a random variable  $j \in [n]$  such that  $\mathbf{Pr}[j = l] = \frac{\exp(p_l)}{\sum_{i=1}^{n} \exp(p_i)}$ . Gibbs sampling on a quantum computer was first studied in Poulin and Wocjan [41], and was later used as a subroutine in quantum SDP solvers [7, 50, 8, 49]. However, the aforementioned quantum Gibbs samplers produce one sample from an  $n$ -dimensional Gibbs distribution in quantum time  $\widetilde{O}(\beta \sqrt{n})$ ; thus, we can produce  $k$  samples in quantum time  $\widetilde{O}(\beta k\sqrt{n})$ . Inspired by the recent work Hamoudi [26] about preparing multiple samples of quantum states, we develop a fast quantum Gibbs sampler (Theorem 4.2) which produces  $k$  samples from a Gibbs distribution in quantum time  $\widetilde{O}(\beta \sqrt{nk})$ . Our quantum multi-Gibbs sampling may have potential applications in sample-based approaches for optimization tasks that require multiple samples.
Technically, the main idea is based on quantum rejection sampling [24, 39], where the target quantum state  $|u\rangle$  is obtained by post-selection from a quantum state  $|u_{\mathrm{guess}}\rangle$  that is easy to prepare (see Section 4). The algorithm has the following steps (Here we assume  $\beta = O(1)$  for simplicity): To bring  $|u_{\mathrm{guess}}\rangle$  closer to the target  $|u\rangle$ , we find the  $k$  (approximately) most dominant amplitudes of  $|u\rangle$  by quantum  $k$ -maximum finding [19] in quantum time  $\widetilde{O}(\sqrt{nk})$ . In quantum  $k$ -maximum finding, we need to estimate the amplitudes of  $|u\rangle$  and compare them coherently, which requires the estimation should be consistent. To address this issue, we develop consistent quantum amplitude estimation (see Appendix C) for our purpose based on consistent phase estimation [2, 47], which is of independent interest. Then, we correct the tail amplitudes of  $|u_{\mathrm{guess}}\rangle$  by quantum singular
value transformation [20], resulting in an (approximate) target state with amplitude  $\Omega (\sqrt{k / n})$  (see Appendix D for details). Finally, we post-select the target state by quantum amplitude amplification [9] in quantum time  $\widetilde{O} (\sqrt{n / k})$ . This follows that  $k$  samples of the target quantum state can be obtained in  $k\cdot \widetilde{O} (\sqrt{n / k}) = \widetilde{O} (\sqrt{n k})$  time.
We believe that our technique can be extended to a wide range of distributions whose mass function is monotonic and satisfies certain Lipschitz conditions.
2.1 General Mathematical Notations
For convenience, we use  $[n]$  to denote the set  $\{1,2,\ldots,n\}$ . We use  $e_i$  to denote a vector whose  $i$ -th coordinate is 1 and other coordinates are 0. For a vector  $v\in \mathbb{R}^n$ ,  $v_{i}$  is the  $i$ -th coordinate of  $v$ . For a function  $f\colon \mathbb{R}\to \mathbb{R}$ , we write  $f(v)$  to denote the result of  $f$  applied to its coordinates, i.e.,  $f(v) = (f(v_{1}),f(v_{2}),\dots ,f(v_{n}))$ . We use  $\Delta_{n}$  to represent the set of  $n$ -dimensional probability distributions, i.e.,  $\Delta_n\coloneqq \{v\in \mathbb{R}^n:\sum_{i = 1}^n v_i = 1,\forall i\in [n],v_i\geq 0\}$ . Here the  $i$ -th coordinate represents the probability of event  $i$  takes place. We use  $\| \cdot \|$  for vector norms. The  $l_{1}$  norm  $\| \cdot \| _1$  for a vector  $v\in \mathbb{R}^n$  is defined as  $\| v\| _1\coloneqq \sum_{i = 1}^n |v_i|$ . For two  $n$ -dimensional probability distributions  $p,q\in \Delta_n$ , their total variance distance is defined as:  $d_{\mathrm{TV}}(p,q) = \frac{1}{2}\| p - q\| _1 = \frac{1}{2}\sum_{i = 1}^n |p_i - q_i|$ . We will use  $\mathbf{A}\in \mathbb{R}^{m\times n}$  to denote a matrix with  $m$  rows and  $n$  columns.  $A_{i,j}$  is the entry of  $\mathbf{A}$  in the  $i$ -th row and  $j$ -th column.  $\| \mathbf{A}\|$  denotes the operator norm of matrix  $\mathbf{A}$ . For a vector  $v\in \mathbb{R}^n$ , we use  $\mathrm{diag}(v)$  to denote the diagonal matrix in  $\mathbb{R}^{n\times n}$  whose diagonal entries are coordinates of  $v$  with the same order.
2.2 Game Theory
Let us consider two players, Alice and Bob, playing a zero-sum game represented by a matrix  $\mathbf{A} \in \mathbb{R}^{m \times n}$  with entries  $A_{i,j}$ , where  $[m]$  is the labeled action set of Alice and  $[n]$  of Bob. Usually, Alice is called the row player and Bob is called the column player.  $A_{i,j}$  is the payoff to Bob and  $-A_{i,j}$  is the payoff to Alice when Alice plays action  $i$  and Bob plays action  $j$ . Both players want to maximize their payoff when they consider their opponent's strategy.
Consider the situation that both players' actions are the best responses to each other. In this case, we call the actions form a Nash equilibrium. A famous minimax theorem by von Neumann [51] states that we can exchange the order of the min and max operation and thus the value of the Nash equilibrium of the game can be properly defined. To be more exact, we have:  $\min_{x\in \Delta_m}\max_{y\in \Delta_n}x^{\top}\mathbf{A}y = \max_{y\in \Delta_n}\min_{x\in \Delta_m}x^{\top}\mathbf{A}y$ . Here the value of the minimax optimization problem is called the value of the game. Our goal is to find an approximate Nash equilibrium for this problem. More formally, we need two probabilistic vectors  $x\in \Delta_m,y\in \Delta_n$  such that the following holds:  $\max_{y'\in \Delta_n}x^{\top}\mathbf{A}y' - \min_{x'\in \Delta_m}x'^{\top}\mathbf{A}y\leq \varepsilon$ . We will call such a pair  $(x,y)$  an  $\varepsilon$ -approximate Nash Equilibrium for the two-player zero-sum game  $\mathbf{A}$ . Computing an  $\varepsilon$ -approximate Nash for a zero-sum game is not as hard as for a general game. For a general game, approximately computing its Nash equilibrium is PPAD-hard [16, 12].
Here, we emphasize an important observation that will be used throughout our paper: we can add a constant or multiply by a positive constant on all  $\mathbf{A}$ 's entries without changing the solution pair of the Nash equilibrium. This is because, in the definition of the Nash equilibrium, the best response is always in comparison with other possible actions, so only the order of the utility matters. Thus without loss of generality, we can always rescale  $\mathbf{A}$  to  $(\mathbf{A} + c\mathbf{1}) / 2$  where  $\mathbf{1}$  is an  $m \times n$  matrix with all entries being 1 with  $c$  being the largest absolute value of  $\mathbf{A}$ 's entries, and let  $\mathbf{A}' = \mathbf{A} / \| \mathbf{A}\|$  to guarantee that  $\mathbf{A}$  has non-negative entries and has operator norm no more than 1.
2.3 Online Learning
2.3.1 Notions of online learning
In general, online learning focuses on making decisions in an uncertain situation, where a decider is not aware of the current loss and is required to make decisions based on observations of previous
losses. To be more exact, we fix the number of rounds  $T$  and judge the performance of the algorithm (in the following of this subsection we use "decider" with the same meaning as the "algorithm") in these  $T$  rounds. Assume that the decoder is allowed to choose actions in the domain  $\mathcal{X}$ , usually a convex subset of a finite-dimensional Euclidean space. Let  $t \in [T]$  denote the current number of rounds. At round  $t$ , the decoder chooses an action  $x_{t} \in \mathcal{X}$ . (The action may depend on the decoder's previous observation of the loss functions  $l_{i}$  for all  $i \in [t - 1]$ .) Then the decoder will get the loss  $l_{t}(x_{t})$  for the current round. We assume that the decoder can also observe the full information of  $l_{t}$ , i.e., the formula of the function. We judge the performance of the algorithm by its regret:  $\mathcal{R}(T) = \sum_{i=1}^{T} l_{t}(x_{t}) - \min_{x \in \mathcal{X}} \sum_{i=1}^{T} l_{t}(x)$ . Intuitively, the regret shows how far the total loss in  $T$  rounds caused by the algorithm is from the optimal static loss.
2.3.2 Online learning in zero-sum games
To demonstrate how to compute the approximate Nash equilibrium using online learning algorithms, we present a useful proposition here. It states that from any sublinear regret learning algorithm  $\mathcal{A}$  with regret  $\mathcal{R}(T)$ , we can find an  $O(\mathcal{R}(T) / T)$ -approximate Nash equilibrium of the zero-sum game in  $T$  rounds.
To be more precise, let us consider the following procedure. Let  $\mathbf{A}$  be the matrix for the two-player zero-sum game. The algorithm starts with some initial strategies  $u_0 \in \Delta_m$ ,  $v_0 \in \Delta_n$  for the two players. Then at each round  $t$ , for each player, it makes decisions with previous observations of the opponent's strategy. In particular, the row player is required to choose his/her action  $u_t \in \Delta_m$  after considering the previous loss functions  $g_i(x) = v_t^\top \mathbf{A}^\top x$  for  $i \in [t-1]$ . Similarly, the column player chooses his/her action  $v_t \in \Delta_n$  with respect to the previous loss functions  $h_i(y) = -u_t^\top \mathbf{A}y$  for  $i \in [t-1]$ . After both players choose their actions at this round  $t$ , they will receive their loss functions  $g_t(x) := v_t^\top \mathbf{A}^\top x$  and  $h_t(y) = -u_t^\top \mathbf{A}y$ , respectively.
Suppose after  $T$  rounds, the regret of the row player with respect to the loss functions  $g_{t}(x)$  is  $\mathcal{R}(T)$ , and the regret of the column player with loss functions  $h_t(y)$  is  $\mathcal{R}'(T)$ . We can write the total regret  $\mathcal{R}(T) + \mathcal{R}'(T)$  explicitly:  $\mathcal{R}(T) + \mathcal{R}'(T) = T(\max_{v\in \Delta_n}\langle v,\mathbf{A}^{\intercal}\hat{u}\rangle -\min_{u\in \Delta_m}\langle u,\mathbf{A}\hat{v}\rangle)$ , where the average strategy is defined as  $\hat{u} = \sum_{i = 1}^{T}u_{i} / T$ ,  $\hat{v} = \sum_{i = 1}^{T}v_{i} / T$ . This pair is a good approximation of the Nash equilibrium for the game  $\mathbf{A}$  if the regret is  $o(T)$ .
2.4 Quantum Computing