Title: Logarithmic-Regret Quantum Learning Algorithms for Zero-Sum Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: Online learning

quantum computing

zero-sum games

optimistic multiplicative weight update
Abstract: We propose the first online quantum algorithm for zero-sum games with $\widetilde O(1)$ regret under the game setting. Moreover, our quantum algorithm computes an $\varepsilon$-approximate Nash equilibrium of an $m \times n$ matrix zero-sum game in quantum time $\widetilde O(\sqrt{m+n}/\varepsilon^{2.5})$. Our algorithm uses standard quantum inputs and generates classical outputs with succinct descriptions, facilitating end-to-end applications. Technically, our online quantum algorithm "quantizes" classical algorithms based on the optimistic multiplicative weight update method. At the heart of our algorithm is a fast quantum multi-sampling procedure for the Gibbs sampling problem, which may be of independent interest.

Logarithmic-Regret Quantum Learning Algorithms for Zero-Sum Games

Abstract

We propose the first online quantum algorithm for solving zero-sum games with ˜ O ( 1 ) regret under the game setting. 1 Moreover, our quantum algorithm computes an ε -approximate Nash equilibrium of an m × n matrix zero-sum game in quantum time ˜ O ( √ m + n / ε 2.5 ) . Our algorithm uses standard quantum inputs and generates classical outputs with succinct descriptions, facilitating end-to-end applications. Technically, our online quantum algorithm 'quantizes' classical algorithms based on the optimistic multiplicative weight update method. At the heart of our algorithm is a fast quantum multi-sampling procedure for the Gibbs sampling problem, which may be of independent interest.

1 Introduction

Nash equilibrium is one of the most important concepts in game theory. It characterizes and predicts rational agents' behaviors in non-cooperative games, finding a vast host of applications ranging from analyzing wars [45] and designing auctions [35], to optimizing networks [43].

It was shown in Daskalakis et al. [16], Chen and Deng [12] that finding a Nash equilibrium is PPAD -hard for general games. Nevertheless, computing the Nash equilibrium for specific types of games, such as zero-sum games, is particularly interesting. A zero-sum game requires that the utility of one player is the opposite of the other's, a condition that often appears in, for example, chess games. Von Neumann's minimax theorem [51] promises that every finite two-player zero-sum game has optimal mixed strategies.

Zhengfeng Ji

Department of Computer Science and Technology, Tsinghua University, Beijing, China jizhengfeng@tsinghua.edu.cn

Qisheng Wang

Zero-Sum Games. For a two-player zero-sum game represented by an m × n matrix A , the Nash equilibrium is the solution pair ( x , y ) to the following min-max problem:

where x ∈ ∆ m and y ∈ ∆ n are m - and n -dimensional probability distributions, respectively. Usually, we are satisfied with an approximate Nash equilibrium rather than an exact one. An ε -approximate Nash equilibrium is a solution pair ( x , y ) such that:

Online Learning. Since the matrix A of the zero-sum game usually has a large dimension in practice, it is common that we trade accuracy for space and time efficiency. Thus, online learning becomes increasingly significant in these scenarios. Online learning studies the situation when data is only available in sequential order and aims at making good decisions in this setup. In evaluating online learning algorithms, regret is an important criterion that measures how good an algorithm is compared with the optimal static loss (see more details in Section 2.3).

The idea of the online learning algorithms for zero-sum games stems from repeated play in game theory, e.g., fictitious play [11]. Specifically, we simulate the actions of two players for multiple rounds. In each round, players make decisions using a no-regret learning algorithm, considering the opponent's previous actions. For example, a famous algorithm of this type was proposed in Grigoriadis and Khachiyan [22] inspired by the exponential Hedge algorithm. The algorithm has regret ˜ O ( √ T ) and T rounds, establishing the convergence rate of ˜ O ( 1/ √ T ) .

It takes about two decades before the next improvement in Daskalakis et al. [17] to happen, where the authors proposed a strongly-uncoupled algorithm, achieving ˜ O ( 1 ) total regret if both players use the algorithm. They used the technique of minimizing non-smooth functions using smoothed approximations proposed in Nesterov [37], and this technique was later developed in Nesterov [38], Nemirovski [36] for broader classes of problems. Later, it was found in Syrgkanis et al. [46] that the optimistic multiplicative weight algorithm also leads to ˜ O ( 1 ) total regret with regret bounded by variation in utilities; this algorithm was recently extended to correlated equilibria in multi-player general-sum games in Anagnostides et al. [3]. It was proved in Hsieh et al. [30] that optimistic mirror descent with a time-varying learning rate can also achieve ˜ O ( 1 ) total regret for multi-players. Our quantum algorithm follows the overall idea of the optimistic multiplicative weight update and the regret bounded by variation methods [46].

Quantum Computing and Learning. Quantum computing has been rapidly advancing in recent years. Specifically, many machine learning problems are known to have significant quantum speedups, e.g., support vector machines [42], principal component analysis [34], classification [31, 33], etc. The combination of quantum computing and online learning has recently become a popular topic. For instance, online learning tools have been applied to solving semidefinite programs (SDPs) with quantum speedup in the problem dimension and the number of constraints [7, 50, 8, 49]. In addition, as an important quantum information task, the online version of quantum state learning has been systematically developed with good theoretical and empirical guarantees [1, 52, 14, 13].

For finding the Nash equilibrium of zero-sum games, a quantum algorithm was proposed in van Apeldoorn and Gilyén [48] by 'quantizing' the classical algorithm in Grigoriadis and Khachiyan [22], achieving a quadratic speedup in the dimension parameters m and n . At the same time, quantum algorithms for training linear and kernel-based classifiers were proposed in Li et al. [33], which have similar problem formulations to zero-sum games. Recently, an improved quantum algorithm for zero-sum games was proposed in Bouland et al. [6] using dynamic Gibbs sampling. All of the above quantum algorithms are based on the multiplicative weight update method, and as a consequence, they all share the O ( √ T ) regret bound.

√

1.1 Main Result

Our result in this paper establishes a positive answer to the following open question: Does there exist a learning algorithm with ˜ O ( 1 ) regret allowing quantum speedups?

Inspired by the optimistic follow-the-regularized-leader algorithm proposed in Syrgkanis et al. [46], we propose a sample-based quantum online learning algorithm for zero-sum games with O ( log ( mn )) total regret, which is near-optimal. If we run this algorithm for T rounds, it will compute an ˜ O ( 1/ T ) -approximate Nash equilibrium with high probability, achieving a quadratic speedup in dimension parameters m and n . Formally, we have the following quantum online learning algorithm:

Theorem 1.1 (Online learning for zero-sum games) . Suppose T ≤ ˜ O ( m + n ) . There is a quantum online algorithm for zero-sum game A ∈ R m × n with ∥ A ∥ ≤ 1 such that it achieves a total regret of O ( log ( mn )) with high probability after T rounds, while each round takes quantum time ˜ O ( T 1.5 √ m + n ) .

Our algorithm does not need to read all the entries of the input matrix A at once. Instead, we assume that our algorithm can query its entries when necessary. The input model is described as follows:

Classically, given any i ∈ [ m ] , j ∈ [ n ] , the entry Ai , j can be accessed in ˜ O ( 1 ) time.

Quantumly, we assume that the entry Ai , j can be accessed in ˜ O ( 1 ) time coherently .

This is the standard quantum input model for zero-sum games adopted in previous literature [33, 48, 6]. See more details in Section 2.4.

In addition, same as prior works [48, 6], our algorithm outputs purely classical vectors with succinct descriptions because they are sparse (with at most T 2 nonzero entries). Overall, using standard quantum inputs and generating classical outputs significantly facilitate end-to-end applications of our algorithm in the near term.

As a direct corollary, we can find an ε -approximate Nash equilibrium by taking T = ˜ O ( 1/ ε ) , resulting in a quantum speedup stated as follows. A detailed comparison to previous literature is presented in Table 1.

Corollary 1.2 (Computing Nash equilibrium) . There is a quantum online algorithm for zero-sum game A ∈ R m × n with ∥ A ∥ ≤ 1 that, with high probability, computes an ε -approximate Nash equilibrium in quantum time ˜ O ( √ m + n / ε 2.5 ) . 4

√

Quantum Lower Bounds. In the full version of [33], they showed a lower bound Ω ( m + n ) for the quantum query complexity of computing an ε -approximate Nash equilibrium of zero-sum games for constant ε . Therefore, our algorithm is tight in terms of m and n .

1.2 Our Techniques

Our quantum online algorithm is a stochastic modification of the optimistic multiplicative weight update proposed by Syrgkanis et al. [46]. We choose to 'quantize' the optimistic online algorithm because it has a better convergence rate for zero-sum games than general multiplicative weight update algorithms. During the update of classical online algorithms, the update term (gradient vector) is computed in linear time by arithmetic operations. However, we observe that it is not necessary to know the exact gradient vector. This motivates us to apply stochastic gradient methods for updates so that certain fast quantum samplers can be utilized here and bring quantum speedups.

Specifically, in our quantum online algorithm, we need to establish an upper bound on the expectation of the total regret of our stochastic update rule, and also deal with errors that may appear from noisy samplers. To reduce the variance of the gradient estimator, we need multiple samples (from Gibbs distributions) at a time. To this end, we develop a fast quantum multi-Gibbs sampler that produces multiple samples by preparing and measuring quantum states.

Sample-Based Optimistic Multiplicative Weight Update. Optimistic online learning adds a 'prediction loss' term to the cumulative loss for regularized minimization, giving a faster convergence rate than the non-optimistic versions for zero-sum games. Arora et al. [4] surveyed the use of the multiplicative weight update method in various domains, but little was known for the optimistic learning method at that time. Daskalakis et al. [17] proposed an extragradient method that largely resembles the optimistic multiplicative weight. Syrgkanis et al. [46] gave a characterization of this update rule-RVU (Regret bounded by Variation in Utilities) property, which is very useful in proving regret bounds. Subsequently, the optimistic learning method is applied to other areas, including training GANs [18] and multi-agent learning [40].

However, when implementing the optimistic learning methods, we face a fundamental difficulty-we cannot directly access data from quantum states without measurement. To resolve this issue, we get samples from the desired distribution and use them to estimate the actual gradient. This idea is commonly seen in previous literature on quantum SDP solvers [7, 50, 8, 49]. Then we prove the regret bound (see Theorem 3.2) of our algorithm by showing that it has a property similar to the RVU property [46]. Moreover, we need multiple samples to obtain a small 'variance' of the stochastic gradient (by taking the average of the samples), to ensure that the expected regret is bounded. Our fast quantum multi-Gibbs sampler produces the required samples and ensures further quantum speedups. In a nutshell, we give an algorithm (Algorithm 1) which modifies the optimistic multiplicative weight algorithm in Syrgkanis et al. [46] to fit the quantum implementation. This is the first quantum algorithm that implements optimistic online learning to the best of our knowledge.

Fast Quantum Multi-Gibbs Sampling. The key to our sample-based approach is to obtain multiple samples from the Gibbs distribution after a common preprocessing step. For a vector p ∈ R n with max i ∈ [ n ] | pi | ≤ β , a sample from the Gibbs distribution with respect to p is a random variable j ∈ [ n ]

such that Pr [ j = l ] = exp ( p l ) ∑ n i = 1 exp ( p i ) . Gibbs sampling on a quantum computer was first studied in Poulin and Wocjan [41], and was later used as a subroutine in quantum SDP solvers [7, 50, 8, 49]. However, the aforementioned quantum Gibbs samplers produce one sample from an n -dimensional Gibbs distribution in quantum time ˜ O ( β √ n ) ; thus, we can produce k samples in quantum time ˜ O ( β k √ n ) . Inspired by the recent work Hamoudi [26] about preparing multiple samples of quantum states, we develop a fast quantum Gibbs sampler (Theorem 4.2) which produces k samples from a Gibbs distribution in quantum time ˜ O ( β √ nk ) . Our quantum multi-Gibbs sampling may have potential applications in sample-based approaches for optimization tasks that require multiple samples.

Technically, the main idea is based on quantum rejection sampling [24, 39], where the target quantum state | u ⟩ is obtained by post-selection from a quantum state | u guess ⟩ that is easy to prepare (see Section 4). The algorithm has the following steps (Here we assume β = O ( 1 ) for simplicity): To bring | u guess ⟩ closer to the target | u ⟩ , we find the k (approximately) most dominant amplitudes of | u ⟩ by quantum k -maximum finding [19] in quantum time ˜ O ( √ nk ) . In quantum k -maximum finding, we need to estimate the amplitudes of | u ⟩ and compare them coherently, which requires the estimation should be consistent. To address this issue, we develop consistent quantum amplitude estimation (see Appendix C) for our purpose based on consistent phase estimation [2, 47], which is of independent interest. Then, we correct the tail amplitudes of | u guess ⟩ by quantum singular

value transformation [20], resulting in an (approximate) target state with amplitude Ω ( √ k / n ) (see Appendix D for details). Finally, we post-select the target state by quantum amplitude amplification [9] in quantum time ˜ O ( √ n / k ) . This follows that k samples of the target quantum state can be obtained in k · ˜ O ( √ n / k ) = ˜ O ( √ nk ) time.

We believe that our technique can be extended to a wide range of distributions whose mass function is monotonic and satisfies certain Lipschitz conditions.

2 Preliminaries

2.1 General Mathematical Notations

For convenience, we use [ n ] to denote the set { 1, 2, . . . , n } . We use e i to denote a vector whose i -th coordinate is 1 and other coordinates are 0 . For a vector v ∈ R n , vi is the i -th coordinate of v . For a function f : R → R , we write f ( v ) to denote the result of f applied to its coordinates, i.e., f ( v ) = ( f ( v 1 ) , f ( v 2 ) , . . . , f ( vn )) . We use ∆ n to represent the set of n -dimensional probability distributions, i.e., ∆ n : = { v ∈ R n : ∑ n i = 1 vi = 1, ∀ i ∈ [ n ] , vi ≥ 0 } . Here the i -th coordinate represents the probability of event i takes place. We use ∥·∥ for vector norms. The l 1 norm ∥·∥ 1 for a vector v ∈ R n is defined as ∥ v ∥ 1 : = ∑ n i = 1 | vi | . For two n -dimensional probability distributions p , q ∈ ∆ n , their total variance distance is defined as: d TV ( p , q ) = 1 2 ∥ p -q ∥ 1 = 1 2 ∑ n i = 1 | pi -qi | . We will use A ∈ R m × n to denote a matrix with m rows and n columns. Ai , j is the entry of A in the i -th row and j -th column. ∥ A ∥ denotes the operator norm of matrix A . For a vector v ∈ R n , we use diag ( v ) to denote the diagonal matrix in R n × n whose diagonal entries are coordinates of v with the same order.

2.2 Game Theory

Let us consider two players, Alice and Bob, playing a zero-sum game represented by a matrix A ∈ R m × n with entries Ai , j , where [ m ] is the labeled action set of Alice and [ n ] of Bob. Usually, Alice is called the row player and Bob is called the column player. Ai , j is the payoff to Bob and -Ai , j is the payoff to Alice when Alice plays action i and Bob plays action j . Both players want to maximize their payoff when they consider their opponent's strategy.

Consider the situation that both players' actions are the best responses to each other. In this case, we call the actions form a Nash equilibrium . Afamous minimax theorem by von Neumann [51] states that we can exchange the order of the min and max operation and thus the value of the Nash equilibrium of the game can be properly defined. To be more exact, we have: min x ∈ ∆ m max y ∈ ∆ n x ⊺ A y = max y ∈ ∆ n min x ∈ ∆ m x ⊺ A y . Here the value of the minimax optimization problem is called the value of the game . Our goal is to find an approximate Nash equilibrium for this problem. More formally, we need two probabilistic vectors x ∈ ∆ m , y ∈ ∆ n such that the following holds: max y ′ ∈ ∆ n x ⊺ A y ′ -min x ′ ∈ ∆ m x ′ ⊺ A y ≤ ε . We will call such a pair ( x , y ) an ε -approximate Nash Equilibrium for the two-player zero-sum game A . Computing an ε -approximate Nash for a zero-sum game is not as hard as for a general game. For a general game, approximately computing its Nash equilibrium is PPAD -hard [16, 12].

Here, we emphasize an important observation that will be used throughout our paper: we can add a constant or multiply by a positive constant on all A 's entries without changing the solution pair of the Nash equilibrium. This is because, in the definition of the Nash equilibrium, the best response is always in comparison with other possible actions, so only the order of the utility matters. Thus without loss of generality, we can always rescale A to ( A + c 1 ) /2 where 1 is an m × n matrix with all entries being 1 with c being the largest absolute value of A 's entries, and let A ′ = A / ∥ A ∥ to guarantee that A has non-negative entries and has operator norm no more than 1 .

2.3 Online Learning

2.3.1 Notions of online learning

In general, online learning focuses on making decisions in an uncertain situation, where a decider is not aware of the current loss and is required to make decisions based on observations of previous

losses. To be more exact, we fix the number of rounds T and judge the performance of the algorithm (in the following of this subsection we use 'decider' with the same meaning as the 'algorithm') in these T rounds. Assume that the decider is allowed to choose actions in the domain X , usually a convex subset of a finite-dimensional Euclidean space. Let t ∈ [ T ] denote the current number of rounds. At round t , the decider chooses an action xt ∈ X . (The action may depend on the decider's previous observation of the loss functions l i for all i ∈ [ t -1 ] .) Then the decider will get the loss l t ( xt ) for the current round. We assume that the decider can also observe the full information of l t , i.e., the formula of the function. We judge the performance of the algorithm by its regret : R ( T ) = ∑ T i = 1 l t ( xt ) -min x ∈ X ∑ T i = 1 l t ( x ) . Intuitively, the regret shows how far the total loss in T rounds caused by the algorithm is from the optimal static loss.

2.3.2 Online learning in zero-sum games

To demonstrate how to compute the approximate Nash equilibrium using online learning algorithms, we present a useful proposition here. It states that from any sublinear regret learning algorithm A with regret R ( T ) , we can find an O ( R ( T ) / T ) -approximate Nash equilibrium of the zero-sum game in T rounds.

To be more precise, let us consider the following procedure. Let A be the matrix for the two-player zero-sum game. The algorithm starts with some initial strategies u 0 ∈ ∆ m , v 0 ∈ ∆ n for the two players. Then at each round t , for each player, it makes decisions with previous observations of the opponent's strategy. In particular, the row player is required to choose his/her action ut ∈ ∆ m after considering the previous loss functions gi ( x ) = v ⊺ i A ⊺ x for i ∈ [ t -1 ] . Similarly, the column player chooses his/her action vt ∈ ∆ n with respect to the previous loss functions hi ( y ) = -u ⊺ i A y for i ∈ [ t -1 ] . After both players choose their actions at this round t , they will receive their loss functions gt ( x ) : = v ⊺ t A ⊺ x and ht ( y ) = -u ⊺ t A y , respectively.

Suppose after T rounds, the regret of the row player with respect to the loss functions gt ( x ) is R ( T ) , and the regret of the column player with loss functions ht ( y ) is R ′ ( T ) . We can write the total regret R ( T ) + R ′ ( T ) explicitly: R ( T ) + R ′ ( T ) = T ( max v ∈ ∆ n ⟨ v , A ⊺ ˆ u ⟩ -min u ∈ ∆ m ⟨ u , A ˆ v ⟩ ) , where the average strategy is defined as ˆ u = ∑ T i = 1 ui / T , ˆ v = ∑ T i = 1 vi / T . This pair is a good approximation of the Nash equilibrium for the game A if the regret is o ( T ) .

2.4 Quantum Computing

In quantum mechanics, a d -dimensional quantum state is described by a unit vector v = ( v 0, v 1 , . . . , vd -1 ) ⊺ , usually denoted as | v ⟩ with the Dirac symbol |·⟩ , in a complex Hilbert space C d . The computational basis of C d is defined as {| i ⟩} d -1 i = 0 , where | i ⟩ = ( 0, . . . , 0, 1, 0, . . . , 0 ) ⊺ with the i -th (0-indexed) entry being 1 and other entries being 0 . The inner product of quantum states | v ⟩ and | w ⟩ is defined by ⟨ v | w ⟩ = ∑ d -1 i = 0 v ∗ i wi , where z ∗ denotes the conjugate of complex number z . The norm of | v ⟩ is defined by ∥| v ⟩∥ = √ ⟨ v | v ⟩ . The tensor product of quantum states | v ⟩ ∈ C d 1 and | w ⟩ ∈ C d 2 is defined by | v ⟩ ⊗ | w ⟩ = ( v 0 w 0, v 0 w 1 , . . . , vd 1 -1 wd 2 -1 ) ⊺ ∈ C d 1 d 2 , denoted as | v ⟩| w ⟩ for short.

A quantum bit (qubit for short) is a quantum state | ψ ⟩ in C 2 , which can be written as | ψ ⟩ = α | 0 ⟩ + β | 1 ⟩ with | α | 2 + | β | 2 = 1 . An n -qubit quantum state is in the tensor product space of n Hilbert spaces C 2 , i.e., ( C 2 ) ⊗ n = C 2 n with the computational basis {| 0 ⟩ , | 1 ⟩ , . . . , | 2 n -1 ⟩} . To obtain classical information from an n -qubit quantum state | v ⟩ , we measure | v ⟩ on the computational basis and obtain outcome i with probability p ( i ) = |⟨ i | v ⟩| 2 for every 0 ≤ i < 2 n . The evolution of a quantum state | v ⟩ is described by a unitary transformation U : | v ⟩ ↦→ U | v ⟩ such that UU † = U † U = I , where U † is the Hermitian conjugate of U , and I is the identity operator. A quantum gate is a unitary transformation that acts on 1 or 2 qubits, and a quantum circuit is a sequence of quantum gates.

Throughout this paper, we assume the quantum oracle O A for a matrix A ∈ R m × n , which is a unitary operator such that for every row index i ∈ [ m ] and column index j ∈ [ n ] , O A | i ⟩| j ⟩| 0 ⟩ = | i ⟩| j ⟩| Ai , j ⟩ . Intuitively, the oracle O A reads the entry Ai , j and stores it in the third register; this is potentially stronger than the classical counterpart when the query is a linear combination of basis vectors, e.g., ∑ k α k | i k ⟩| j k ⟩ with ∑ k | α k | 2 = 1 . This is known as the superposition principle in quantum computing.

Note that this input model for matrices is commonly used in quantum algorithms, e.g., linear system solvers [28] and semidefinite programming solvers [7, 50, 8, 49].

A quantum (query) algorithm A is a quantum circuit that consists a sequence of unitary operators G 1 , G 2, . . . , GT , each of which is either a quantum gate or a quantum oracle. The quantum time complexity of A is measured by the number T of quantum gates and quantum oracles in A . The execution of A on n qubits starts with quantum state | 0 ⟩ ⊗ n , then it performs unitary operators G 1 , G 2, . . . , GT on the quantum state in this order, resulting in the quantum state | ϕ ⟩ = GT . . . G 2 G 1 | 0 ⟩ ⊗ n . Finally, we measure | ϕ ⟩ on the computational basis | i ⟩ for 0 ≤ i < 2 n , giving a classical output i with probability |⟨ i | ϕ ⟩| 2 .

3 Quantum Algorithm for Online Zero-Sum Games by Sample-Based Optimistic Multiplicative Weight Update

Now, we present our quantum algorithm for finding an approximate Nash equilibrium for zerosum games (Algorithm 1). This algorithm is a modification of the optimistic multiplicative weight algorithm [46], in which we use stochastic gradients to estimate true gradients. This modification utilizes the quantum advantage of Gibbs sampling (as will be shown in Section 4). It is the source of quantum speedups in the algorithm and also the reason that we call the algorithm sample-based . To this end, we first give the definition of Gibbs sampling oracles.

Definition 3.1 (Approximate Gibbs sampling oracle) . Let p ∈ R n be an n -dimensional vector, ϵ > 0 be the approximate error. We let O Gibbs p ( k , ϵ ) denote the oracle which produces k independent samples from a distribution that is ϵ -close to the Gibbs distribution with parameter p in total variation distance. Here, for a random variable j taking value in [ n ] following the Gibbs distribution with parameter p , we have Pr [ j = l ] = exp ( pl ) / ∑ n i = 1 exp ( pi ) .

Algorithm 1 Sample-Based Optimistic Multiplicative Weight Update for Matrix Games

Input: A ∈ R m × n , additive approximation ε , approximate Gibbs sampling oracle O Gibbs with error √

Output: ( ˆ u , ˆ v ) as the approximate Nash equilibrium of the matrix game A .

ϵ G , total episode T , learning rate λ ∈ ( 0, 3/6 ) .

1: Initialize ˆ u ← 0 m , ˆ v ← 0 n , x 1 ← 0 m , y 1 ← 0 n .

3: for t = 1, . . . , T do

2: Set g 1 ← x 1 , h 1 ← y 1 .

4: Get T independent samples i t 1 , i t 2 , . . . , i t T from the Gibbs sampling oracle O Gibbs -A ht ( T , ϵ G ) .

6: Update xt + 1 ← xt + λζ t , gt + 1 ← xt + 1 + λζ t . ˆ u ← ˆ u + 1 T ζ t .

5: Choose the action ζ t = ∑ T N = 1 e i t N / T .

7: Get T independent samples j t 1 , j t 2 , . . . , j t T from the Gibbs sampling oracle O Gibbs A ⊺ gt ( T , ϵ G ) .

8: Choose the action η t = ∑ T N = 1 e j t N / T .

10: end for

9: Update yt + 1 ← yt + λη t , ht + 1 ← yt + 1 + λη t , ˆ v ← ˆ v + 1 T η t .

11: return ( ˆ u , ˆ v ) .

Suppose the zero-sum game is represented by matrix A ∈ R m × n with ∥ A ∥ ≤ 1 . Our samplebased optimistic multiplicative weight update algorithm is given in Algorithm 1. Algorithm 1 is inspired by the classical optimistic follow-the-regularized-leader algorithm (see Appendix A for more information). In that classical algorithm, the update terms are essentially E [ ζ t ] and E [ η t ] , which are computed deterministically by matrix arithmetic operations during the update. In contrast, we do this probabilistically by sampling from the Gibbs distributions and ζ t and η t in Line 5 and Line 10 are the corresponding averages of the samples. For this to work, we need to bound the expectation of the total regret (see Appendix B) based on the RVU property (Definition A.2). Technically, the l 1 variances of ζ t and η t turn out to be significant in the analysis. To reduce the variances, we need multiple independent samples identically distributed from Gibbs distributions (see Line 4 and Line 7 in Algorithm 1). Because of the randomness from Gibbs sampling oracles, the total regret R ( T ) + R ′ ( T ) of Algorithm 1 is a random variable. Nevertheless, we can bound the total regret by O ( log ( mn )) with high probability as follows.

Theorem 3.2. Let ϵ G = 1/ T . After T rounds of playing the zero-sum game A by Algorithm 1, the total regret will be bounded by

with probability at least 2/3 . Then, for any constant λ ∈ ( 0, 3/6 ) , the total regret is O ( log ( mn )) . Moreover, if we choose T = ˜ Θ ( 1/ ε ) , then Algorithm 1 will return an ε -approximate Nash equilibrium of the game A .

The proof of Theorem 3.2 is deferred to Appendix B.2. Combining Theorem 3.2 and our fast quantum multi-Gibbs sampler in Theorem 4.2 (which will be developed in the next section), we obtain a quantum algorithm for finding an ε -approximate Nash equilibrium of zero-sum games.

Corollary 3.3. If we choose T = ˜ Θ ( 1/ ε ) and use quantum multi-Gibbs sampling (Algorithm 2) with ϵ G = 1/ T , then Algorithm 1 will return an ε -approximate Nash equilibrium in quantum time ˜ O ( √ m + n / ε 2.5 ) .

4 Fast Quantum Multi-Gibbs Sampling

In Algorithm 1, the vectors ht and gt updated in each round are used to generate independent samples from Gibbs distributions O Gibbs -A ht and O Gibbs A ⊺ gt . Here, ht and gt are supposed to be stored in classical registers. To allow quantum speedups, we store ht and gt in quantum-read classical-write random access memory (QRAM) [21], which is commonly used in prior work [49, 6]. Specifically, a QRAM can store/modify an array a 1 , a 2, . . . , a ℓ of classical data and provide quantum (read-only) access to them, i.e., a unitary operator U QRAM is given such that U QRAM : | i ⟩| 0 ⟩ ↦→ | i ⟩| ai ⟩ . Without loss of generality (see Remark 4.3), suppose we have quantum oracle O A for A ∈ R n × n with Ai , j ∈ [ 0, 1 ] , and QRAM access to a vector z ∈ R n with zi ≥ 0 . We also need the polynomial approximation of the exponential function for applying the QSVT technique [20]:

Lemma 4.1 (Polynomial approximation, Lemma 7 of [48]) . Let β ≥ 1 and ϵ P ∈ ( 0, 1/2 ) . There is a classically efficiently computable polynomial P β ∈ R [ x ] of degree O ( β log ( ϵ -1 P )) such that ∣ ∣ P β ( x ) ∣ ∣ ≤ 1 for x ∈ [ -1, 1 ] , and max x ∈ [ -1,0 ] ∣ ∣ ∣ P β ( x ) -1 4 exp ( β x ) ∣ ∣ ∣ ≤ ϵ P .

Then, we can produce multiple samples from O Gibbs A z efficiently by Algorithm 2 on a quantum computer. Algorithm 2 is inspired by Hamoudi [26] about preparing multiple samples of a quantum state, with quantum access to its amplitudes. However, we do not have access to the exact values of the amplitudes, which are ( A z ) i in our case. To resolve this issue, we develop consistent quantum amplitude estimation (see Appendix C) to estimate ( A z ) i with a unique answer (Line 1). After having prepared an initial quantum state | u guess ⟩ , we use quantum singular value decomposition [20] to correct the tail amplitudes (Line 6), and finally obtain the desired quantum state | ˜ u Gibbs ⟩ by quantum amplitude amplification [9] (Line 7). We have the following (see Appendix D for its proof):

Theorem 4.2 (Fast quantum multi-Gibbs sampling) . For k ∈ [ n ] , if we set ϵ P = Θ ( k ϵ 2 G / n ) , then Algorithm 2 will produce k independent and identical distributed samples from a distribution that is ϵ G -close to O Gibbs A z in total variation distance, in quantum time ˜ O ( β √ nk ) .

Remark 4.3 . If A ∈ R m × n is not a square matrix, then by adding 0 's we can always enlarge A to an ( m + n ) -dimensional square matrix. For O Gibbs -A ht as required in Algorithm 1, we note that O Gibbs ( 1 -A ) ht indicates the same distribution as O Gibbs -A ht , where 1 has the same size as A with all entries being 1 . From the above discussion, we can always convert A to another matrix satisfying our assumption, i.e., with entries in the range [ 0, 1 ] .

Remark 4.4 . The description of the unitary operator U exp defined by the polynomial P 2 β can be classically computed to precision ϵ P in time ˜ O ( β 3 ) by Haah [25], which is ˜ O ( 1/ ε 3 ) in our case as β ≤ λ T = ˜ Θ ( 1/ ε ) is required in Corollary 3.3. This extra cost can be neglected because ˜ O ( √ m + n / ε 2.5 ) dominates the complexity whenever ε = Ω (( m + n ) -1 ) .

Remark 4.5 . Our multi-Gibbs sampler is based on maximum finding and consistent amplitude estimation, with a guaranteed worst-case performance in each round. In comparison, the dynamic Gibbs sampler in [6] maintains a hint vector, resulting in an amortized time complexity per round.

Algorithm 2 Quantum Multi-Gibbs Sampling O Gibbs A z ( k , ϵ G )

Input: Quantum oracle O A for A ∈ R n × n , QRAM access to z ∈ R n with ∥ z ∥ 1 ≤ β , polynomial P 2 β with parameters ϵ P by Lemma 4.1, number k of samples. We write u = A z .

Output: k independent samples i 1 , i 2, . . . , i k .

2: Find the set S ⊆ [ n ] of indexes of the k largest ˜ ui by quantum k -maximum finding, with access to O ˜ u .

1: Obtain O ˜ u : | i ⟩| 0 ⟩ ↦→ | i ⟩| ˜ ui ⟩ by consistent quantum amplitude estimation such that ui ≤ ˜ ui ≤ ui + 1 .

3: Obtain ˜ ui for all i ∈ S from O ˜ u , then compute ˜ u ∗ = min i ∈ S ˜ ui and W = ∑ i ∈ S exp ( ˜ ui ) + ( n -k ) exp ( ˜ u ∗ ) .

4: for ℓ = 1, . . . , k do

5: Prepare the quantum state | u guess ⟩ = ∑ i ∈ S √ exp ( ˜ ui ) W | i ⟩ + ∑ i / ∈ S √ exp ( ˜ u ∗ ) W | i ⟩ .

7: Post-select | ˜ u Gibbs ⟩ ∝ ⟨ 0 | ⊗ a U exp | u guess ⟩| 0 ⟩ ⊗ a by quantum amplitude amplification.

6: Obtain unitary U exp such that ⟨ 0 | ⊗ a U exp | 0 ⟩ ⊗ a = diag ( P 2 β ( u -max { ˜ u , ˜ u ∗ } ) ) /4 β by QSVT.

8: Let i ℓ be the measurement outcome of | ˜ u Gibbs ⟩ in the computational basis.

9: end for

10: return i 1 , i 2, . . . , i k .

5 Discussion

In our paper, we propose the first quantum online algorithm for zero-sum games with near-optimal regret. This is achieved by developing a sample-based stochastic version of the optimistic multiplicative weight update method [46]. Our core technical contribution is a fast multi-Gibbs sampling, which may have potential applications in other quantum computing scenarios.

Our result naturally gives rise to some further open questions. For instance: Can we improve the dependence on ε for the time complexity? And can we further explore the combination of optimistic learning and quantum computing into broader applications? Now that many heuristic quantum approaches for different machine learning problems have been realized, e.g.in Havlíˇ cek et al. [29], Saggio et al. [44], Harrigan et al. [27], can fast quantum algorithms for zero-sum games be realized in the near future?

Acknowledgments

We would like to thank Kean Chen, Wang Fang, Ji Guan, Junyi Liu, Xinzhao Wang, Chenyi Zhang, and Zhicheng Zhang for helpful discussions. MG would like to thank Mingsheng Ying for valuable suggestions.

Minbo Gao was supported by the National Key R&D Program of China (2018YFA0306701) and the National Natural Science Foundation of China (61832015). Zhengfeng Ji was supported by a startup fund from Tsinghua University, and the Department of Computer Science and Technology, Tsinghua Univeristy. Tongyang Li was supported by a startup fund from Peking University, and the Advanced Institute of Information Technology, Peking University. Qisheng Wang was supported by the MEXT Quantum Leap Flagship Program (MEXT Q-LEAP) grants No. JPMXS0120319794.

References

S. Aaronson, X. Chen, E. Hazan, S. Kale, and A. Nayak. Online learning of quantum states. In Advances in Neural Information Processing Systems , volume 31, 2018. URL https://proceedings.neurips.cc/paper/2018/file/ c1a3d34711ab5d85335331ca0e57f067-Paper.pdf .

A. Ambainis. Variable time amplitude amplification and quantum algorithms for linear algebra problems. In Proceedings of the 29th Symposium on Theoretical Aspects of Computer

Science , volume 14, pages 636-647, 2012. URL https://hal.archives-ouvertes. fr/hal-00678197 .

I. Anagnostides, C. Daskalakis, G. Farina, M. Fishelson, N. Golowich, and T. Sandholm. Nearoptimal no-regret learning for correlated equilibria in multi-player general-sum games. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 736-749, 2022. URL https://doi.org/10.1145/3519935.3520031 .

S. Arora, E. Hazan, and S. Kale. The multiplicative weights update method: a meta-algorithm and applications. Theory of Computing , 8(6):121-164, 2012. URL https://doi.org/10. 4086/toc.2012.v008a006 .

D. W. Berry, A. M. Childs, R. Cleve, R. Kothari, and R. D. Somma. Simulating Hamiltonian dynamics with a truncated Taylor series. Physical Review Letters , 114(9):090502, 2015. URL https://doi.org/10.1103/PhysRevLett.114.090502 .

A. Bouland, Y. M. Getachew, Y. Jin, A. Sidford, and K. Tian. Quantum speedups for zerosum games via improved dynamic Gibbs sampling. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 2932-2952, 2023. URL https:// proceedings.mlr.press/v202/bouland23a.html .

F. G. Brandão and K. M. Svore. Quantum speed-ups for solving semidefinite programs. In Proceedings of the 58th Annual Symposium on Foundations of Computer Science , pages 415426, 2017. URL https://doi.org/10.1109/FOCS.2017.45 .

F. G. S. L. Brandão, A. Kalev, T. Li, C. Y .-Y . Lin, K. M. Svore, and X. Wu. Quantum SDP solvers: Large speed-ups, optimality, and applications to quantum learning. In Proceedings of the 46th International Colloquium on Automata, Languages, and Programming , volume 132, pages 27:1-27:14, 2019. URL https://doi.org/10.4230/LIPIcs.ICALP.2019.27 .

G. Brassard, P. Høyer, M. Mosca, and A. Tapp. Quantum amplitude amplification and estimation. Quantum Computation and Information , 305:53-74, 2002. URL https://doi.org/10. 1090/conm/305/05215 .

Y. Carmon, Y. Jin, A. Sidford, and K. Tian. Variance reduction for matrix games. In Advances in Neural Information Processing Systems , volume 32, pages 1137711388, 2019. URL https://proceedings.neurips.cc/paper/2019/file/ 6c442e0e996fa84f344a14927703a8c1-Paper.pdf .

N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006. URL https://doi.org/10.1017/CBO9780511546921 .

X. Chen and X. Deng. Settling the complexity of two-player nash equilibrium. In Proceedings of the 47th Annual Symposium on Foundations of Computer Science , pages 261-272, 2006. URL https://doi.org/10.1109/FOCS.2006.69 .

X. Chen, E. Hazan, T. Li, Z. Lu, X. Wang, and R. Yang. Adaptive online learning of quantum states, 2022. URL https://arxiv.org/abs/2206.00220 .

Y. Chen and X. Wang. More practical and adaptive algorithms for online quantum state learning, 2020. URL https://arxiv.org/abs/2006.01013 .

A. M. Childs and N. Wiebe. Hamiltonian simulation using linear combinations of unitary operations. Quantum Information and Computation , 12(11-12):901-924, 2012. URL https: //doi.org/10.26421/QIC12.11-12-1 .

C. Daskalakis, P. W. Goldberg, and C. H. Papadimitriou. The complexity of computing a nash equilibrium. SIAM Journal on Computing , 39(1):195-259, 2009. URL http://epubs. siam.org/doi/10.1137/070699652 .

C. Daskalakis, A. Deckelbaum, and A. Kim. Near-optimal no-regret algorithms for zero-sum games. Games and Economic Behavior , 92:327-348, 2015. URL https://doi.org/10. 1016/j.geb.2014.01.003 .

C. Daskalakis, A. Ilyas, V . Syrgkanis, and H. Zeng. Training gans with optimism. In Proceedings of the 6th International Conference on Learning Representations , 2018.

C. Dürr, M. Heiligman, P. Høyer, and M. Mhalla. Quantum query complexity of some graph problems. SIAM Journal on Computing , 35(6):1310-1328, 2006. URL http://doi.org/ 10.1137/050644719 .

A. Gilyén, Y. Su, G. H. Low, and N. Wiebe. Quantum singular value transformation and beyond: Exponential improvements for quantum matrix arithmetics. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , pages 193-204, 2019. URL https://doi.org/10.1145/3313276.3316366 .

V. Giovannetti, S. Lloyd, and L. Maccone. Quantum random access memory. Physical Review Letters , 100(16):160501, 2008. URL https://doi.org/10.1103/PhysRevLett. 100.160501 .

M. D. Grigoriadis and L. G. Khachiyan. A sublinear-time randomized approximation algorithm for matrix games. Operations Research Letters , 18(2):53-58, 1995. URL https://doi. org/10.1016/0167-6377(95)00032-0 .

L. Grover and T. Rudolph. Creating superpositions that correspond to efficiently integrable probability distributions, 2002. URL http://arxiv.org/abs/quant-ph/0208112 .

L. K. Grover. Synthesis of quantum superpositions by quantum computation. Physical Review Letters , 85(6):1334, 2000. URL https://doi.org/10.1103/PhysRevLett.85. 1334 .

J. Haah. Product decomposition of periodic functions in quantum signal processing. Quantum , 3:190, 2019. URL https://doi.org/10.22331/q-2019-10-07-190 .

Y. Hamoudi. Preparing many copies of a quantum state in the black-box model. Physical Review A , 105(6):062440, 2022. URL https://doi.org/10.1103/PhysRevA.105. 062440 .

M. P. Harrigan, K. J. Sung, M. Neeley, K. J. Satzinger, F. Arute, K. Arya, J. Atalaya, J. C. Bardin, R. Barends, S. Boixo, M. Broughton, B. B. Buckley, D. A. Buell, B. Burkett, N. Bushnell, Y. Chen, Z. Chen, Ben Chiaro, R. Collins, W. Courtney, S. Demura, A. Dunsworth, D. Eppens, A. Fowler, B. Foxen, C. Gidney, M. Giustina, R. Graff, S. Habegger, A. Ho, S. Hong, T. Huang, L. B. Ioffe, S. V . Isakov, E. Jeffrey, Z. Jiang, C. Jones, D. Kafri, K. Kechedzhi, J. Kelly, S. Kim, P. V. Klimov, A. N. Korotkov, F. Kostritsa, D. Landhuis, P. Laptev, M. Lindmark, M. Leib, O. Martin, J. M. Martinis, J. R. McClean, M. McEwen, A. Megrant, X. Mi, M. Mohseni, W. Mruczkiewicz, J. Mutus, O. Naaman, C. Neill, F. Neukart, M. Y. Niu, T. E. O'Brien, B. O'Gorman, E. Ostby, A. Petukhov, H. Putterman, C. Quintana, P. Roushan, N. C. Rubin, D. Sank, A. Skolik, V. Smelyanskiy, D. Strain, M. Streif, M. Szalay, A. Vainsencher, T. White, Z. J. Yao, P. Yeh, A. Zalcman, L. Zhou, H. Neven, D. Bacon, E. Lucero, E. Farhi, and R. Babbush. Quantum approximate optimization of non-planar graph problems on a planar superconducting processor. Nature Physics , 17(3):332-336, 2021. URL https://doi.org/10.1038/ s41567-020-01105-y .

A. W. Harrow, A. Hassidim, and S. Lloyd. Quantum algorithm for linear systems of equations. Physical Review Letters , 103(15):150502, 2009. URL https://doi.org/10.1103/ PhysRevLett.103.150502 .

V. Havlíˇ cek, A. D. Córcoles, K. Temme, A. W. Harrow, A. Kandala, J. M. Chow, and J. M. Gambetta. Supervised learning with quantum-enhanced feature spaces. Nature , 567(7747): 209-212, 2019. URL https://doi.org/10.1038/s41586-019-0980-2 .

Y.-G. Hsieh, K. Antonakopoulos, and P. Mertikopoulos. Adaptive learning in continuous games: Optimal regret bounds and convergence to nash equilibrium. In Proceedings of the 34th Conference on Learning Theory , volume 134, pages 2388-2422, 2021. URL https: //proceedings.mlr.press/v134/hsieh21a.html .

A. Kapoor, N. Wiebe, and K. Svore. Quantum perceptron models. In Advances in Neural Information Processing Systems , volume 29, 2016. URL https://proceedings.neurips. cc/paper/2016/file/d47268e9db2e9aa3827bba3afb7ff94a-Paper.pdf .

I. Kerenidis and A. Prakash. Quantum recommendation systems. In Proceedings of the 8th Innovations in Theoretical Computer Science Conference , volume 67, pages 49:1-49:21, 2017. URL https://doi.org/10.4230/LIPIcs.ITCS.2017.49 .

T. Li, S. Chakrabarti, and X. Wu. Sublinear quantum algorithms for training linear and kernelbased classifiers. In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 3815-3824, 2019. URL https://proceedings.mlr.press/v97/ li19b.html .

S. Lloyd, M. Mohseni, and P. Rebentrost. Quantum principal component analysis. Nature Physics , 10(9):631-633, 2014. URL https://doi.org/10.1038/nphys3029 .

R. B. Myerson. Optimal auction design. Mathematics of Operations Research , 6(1):58-73, 1981. URL https://doi.org/10.1287/moor.6.1.58 .

A. Nemirovski. Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM Journal on Optimization , 15(1):229-251, 2004. URL https://doi.org/10.1137/ S1052623403425629 .

Y. Nesterov. Smooth minimization of non-smooth functions. Mathematical Programming , 103 (1):127-152, may 2005. URL https://doi.org/10.1007/s10107-004-0552-5 .

Y. Nesterov. Excessive gap technique in nonsmooth convex minimization. SIAM Journal on Optimization , 16(1):235-249, 2005. URL https://doi.org/10.1137/ S1052623403422285 .

M. Ozols, M. Roetteler, and J. Roland. Quantum rejection sampling. ACM Transactions on Computation Theory , 5(3):11:1-33, 2013. URL https://doi.org/10.1145/2493252. 2493256 .

S. Pattathil, K. Zhang, and A. Ozdaglar. Symmetric (optimistic) natural policy gradient for multi-agent learning with parameter convergence, 2022. URL https://arxiv.org/abs/ 2210.12812 .

D. Poulin and P. Wocjan. Sampling from the thermal quantum Gibbs state and evaluating partition functions with a quantum computer. Physical Review Letters , 103(22):220502, 2009. URL https://doi.org/10.1103/PhysRevLett.103.220502 .

P. Rebentrost, M. Mohseni, and S. Lloyd. Quantum support vector machine for big data classification. Physical Review Letters , 113:130503, Sep 2014. URL https://doi.org/ 10.1103/PhysRevLett.113.130503 .

T. Roughgarden and E. Tardos. How bad is selfish routing? Journal of the ACM , 49(2):236-259, mar 2002. URL https://doi.org/10.1145/506147.506153 .

V. Saggio, B. E. Asenbeck, A. Hamann, T. Strömberg, P. Schiansky, V. Dunjko, N. Friis, N. C. Harris, M. Hochberg, D. Englund, S. Wölk, H. J. Briegel, and P. Walther. Experimental quantum speed-up in reinforcement learning agents. Nature , 591(7849):229-233, 2021. URL https://doi.org/10.1038/s41586-021-03242-7 .

T. C. Schelling. The Strategy of Conflict: with a new Preface by the Author . Harvard university press, 1980.

V. Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire. Fast convergence of regularized learning in games. In Advances in Neural Information Processing Systems , pages 2989-2997, 2015.

A. Ta-Shma. Inverting well conditioned matrices in quantum logspace. In Proceedings of the 45th Annual ACM Symposium on Theory of Computing , pages 881-890, 2013. URL https://doi.org/10.1145/2488608.2488720 .

J. van Apeldoorn and A. Gilyén. Quantum algorithms for zero-sum games, 2019. URL http://arxiv.org/abs/1904.03180 .

J. van Apeldoorn and A. Gilyén. Improvements in quantum SDP-solving with applications. In Proceedings of the 46th International Colloquium on Automata, Languages, and Programming , volume 132, pages 99:1-99:15, 2019. URL https://doi.org/10.4230/LIPIcs. ICALP.2019.99 .

J. van Apeldoorn, A. Gilyén, S. Gribling, and R. de Wolf. Quantum SDP-solvers: Better upper and lower bounds. In Proceedings of the 58th Annual Symposium on Foundations of Computer Science , pages 403-414, 2017. URL https://doi.org/10.1109/FOCS.2017.44 .

J. von Neumann. Zur theorie der gesellschaftsspiele. Mathematische Annalen , 100(1):295-320, 1928.

F. Yang, J. Jiang, J. Zhang, and X. Sun. Revisiting online quantum state learning. Proceedings of the AAAI Conference on Artificial Intelligence , 34(04):6607-6614, 2020.

A Revisit of Optimistic Online Learning

In this section, we briefly review some important properties in the classical optimistic online learning algorithms. Some of the propositions in this section will be frequently used in the proof of the regret bound.

For convenience, we will use ψ ( · ) to denote the negative entropy function, i.e., ψ : ∆ n → R , ψ ( p ) = ∑ n i = 1 pi log pi . Note that log stands for the natural logarithm function with base e .

For a vector norm ∥·∥ , its dual norm is defined as:

Proposition A.1. Let L be a vector in n -dimensional space. If p ∗ = arg min p ∈ ∆ n {⟨ p , L ⟩ + ψ ( p ) } , then

p ∗ can be written as:

and vice versa.

Proof. Write L = ( L 1 , L 2, . . . , Ln ) . By definition, we know that p ∗ is the solution to the following convex optimization problem:

The Lagrangian is

From KKT conditions, we know that the stationarity is:

The complementary slackness is:

The primal feasibility is

The dual feasibility is

̸

If ui = 0 then pi = 0 , from stationarity we know ui = -∞ , but that violates the dual feasibility. So we can conclude that ui = 0 for all i ∈ [ n ] , thus pi ∝ exp ( -Li ) and the result follows.

Now we present a generalized version of the optimistic multiplicative weight algorithm called optimisitically follow the regularized leader (Opt-FTRL) in Algorithm 3. In the algorithm, mt has the same meaning as m ( t ) for notation consistency.

Algorithm 3 Optimistic follow-the-regularized-leader

Input: The closed convex domain X .

Output: Step size λ , loss gradient prediction m .

Initialize L 0 ← 0 , choose appropriate m 1 .

Observe loss l t , update Lt = Lt - 1 + l t .

Compute

mt + 1 using observations till now.

end for

Now we study a crucial property that leads to the fast convergence of the algorithm, called the Regret bounded by Variation in Utilities (RVU in short). For simplicity, we only consider the linear loss function l t ( x ) = ⟨ l t , x ⟩ . (There is a little abuse of notation here.)

Definition A.2 (Regret bounded by Variations in Utilities (RVU), Definition 3 in Syrgkanis et al. [46]) . Consider an online learning algorithm A with regret R ( T ) = o ( T ) , we say that it has the property of regret bound by variation in utilities if for any linear loss sequence l 1 , l 2, . . . , lT , there exists parameters α > 0, 0 < β ≤ γ such that the algorithm output decisions x 1 , x 2, . . . , xT , xT + 1 that satisfy:

where ∥·∥ ∗ is the dual norm of ∥·∥ .

We do not choose the norm to be any specific one here. In fact, Syrgkanis et al. [46] have already shown that the above optimistic follow-the-regularized-leader algorithm has the RVU property with respect to any norm ∥·∥ in which the negative entropy function ψ is 1 -strongly convex. So, from Pinsker's inequality, for l 2 norms the following result holds:

Proposition A.3 (Proposition 7 in Syrgkanis et al. [46]) . If we choose mt = l t -1 in the optimistic follow-the-regularized-leader algorithm with step size λ ≤ 1/2 , then it has the regret bound by variation in utilities property with the parameters α = log n / λ , β = λ and γ = 1/ ( 4 λ ) , where n is the dimension of X .

B Regret Bound and Time Complexity of Our Algorithm

B.1 Ideal Samplers

We assume that after the execution of our algorithm, the sequences we get are { ( xt , yt ) } T + 1 t = 1 and { ( gt , ht ) } T + 1 t = 1 , respectively. We denote ut : = exp ( -A ht ) ∥ exp ( -A ht ) ∥ 1 and vt : = exp ( A ⊺ gt ) ∥ exp ( A ⊺ gt ) ∥ 1 to be the corresponding Gibbs distribution, we will first assume that the Gibbs oracle in our algorithm has no error (i.e. ϵ G = 0 ) until Theorem B.5 is proved.

Observation B.1. The sequence { ut } T t = 1 can be seen as the decision result of applying optimistic FTRL algorithm to the linear loss function A η t with linear prediction function A η t -1 , and similarly for { vt } T + 1 t = 1 with the loss function -A ⊺ ζ t , the prediction function -A ⊺ ζ t -1 .

Proof. By symmetry, we only consider ut . Since ut = exp ( -A ht ) ∥ exp ( -A ht ) ∥ 1 , from Proposition A.1 we can write

Then we notice the iteration of Algorithm 1 gives

So from the definition of the Algorithm 3, we know that our observation holds.

This observation, together with Proposition A.3, gives the following inequalities. For any u ∈ ∆ m , v ∈ ∆ n , we have:

However, we find that the loss function is slightly different from what we expect.

Let us consider the difference qt : = A ( vt -η t ) and pt : = -A ⊺ ( ut -ζ t ) , we have the decomposition of the regret:

Notice that E [ qt ] = E [ pt ] = 0 , we have:

Lemma B.2.

Proof. By symmetry, we only prove the case for u . It suffices to prove that for every t , E [ ⟨ ut -u , qt ⟩ ] = 0 . Since u is fixed, E [ ⟨ u , qt ⟩ ] = ⟨ u , E [ qt ] ⟩ = 0 .

Now consider E [ ⟨ ut , qt ⟩ ] , notice that given η 1 , . . . , η t -1 then ut is a constant. We have:

Now we are going to bound the term ∑ T -1 ∥ A ( η t + 1 -η t ) ∥ 2 .

t = 1 Lemma B.3.

Proof. Recall that by rescaling we have ∥ A ∥ ≤ 1 . Hence,

Write η t + 1 -η t = ( η t + 1 -vt + 1 ) + ( vt + 1 -vt ) + ( vt -η t ) . Using the triangle inequality of the l 1 norm and the Cauchy inequality ( a + b + c ) 2 ≤ 3 ( a 2 + b 2 + c 2 ) , we get

Similarly, we have:

Observing that in our algorithm we collect T independent and identically distributed samples and take their average, we have:

Combining the result above, we just get the desired equation.

We also need the following lemma to guarantee that the sum of the regret is always non-negative. Lemma B.4. The sum of the regrets of two players in Algorithm 1 is always non-negative. In other words:

Proof.

The last step is because and

Combining the result above, we finally have the following theorem.

Theorem B.5. Suppose that in our Algorithm 1, we choose the episode T = ˜ Θ ( 1/ ε ) , and choose a constant learning rate λ that satisfies λ < √ 3/6 . Then with probability at least 2/3 the total regret of the algorithm is ˜ O ( 1 ) . To be more clear, we have:

and so our algorithm returns an ε -approximate Nash equilibrium.

Proof. Adding the inequalities (2) and (3) together, we get

Taking expectation, and using the inequalities (6) we have

Similarly we can prove

So, taking expectations of Equation (8), and using the above inequalities and the Lemma B.2, we get

Using the fact that we have

By Lemma B.4, we know that the regret is always non-negative. So applying Markov's inequality, we know with probability at least 2/3 , the following inequality holds:

B.2 Samplers with Errors

Theorem B.6 (Restatement of Theorem 3.2) . Suppose that in our Algorithm 1, we choose the episode T = ˜ O ( 1/ ε ) , and choose a constant learning rate λ that satisfies 0 < λ < √ 3/6 . The quantum implementation of the oracle in the algorithm will return T independent and identically distributed samples from a distribution that is ϵ G -close to the desired distribution in total variational distance in quantum time T Q G .

Then with probability at least 2/3 the total regret of the algorithm is ˜ O ( 1 + ϵ G/ ε ) and the algorithm returns an ˜ O ( ε + ϵ G ) -approximate Nash equilibrium in quantum time ˜ O ( T Q G / ε ) .

Proof. We will follow similar steps of proof for Theorem B.5. Since the sampling is not from the ideal distribution, we must bound the terms where η t and ζ t take place.

Notice that in this case, we have

So for the term qt in Lemma B.2 we now have the bound:

where the last step is by Hölder's inequality.

Then for the other term, we have

So following the similar steps of proof for Theorem B.5, and using the above bounds, we can get

Again using linearity of expectation and Markov's inequality, we conclude that with probability at least 2/3

C Consistent Quantum Amplitude Estimation

Theorem C.1 (Consistent phase estimation, [2, 47]) . Suppose U is a unitary operator. For every positive reals ϵ , δ , there is a quantum algorithm (a unitary quantum circuit) A such that, on input O ( log ( ϵ -1 )) -bit random string s , it holds that

f ( s , θ ) is a function of s and θ such that | f ( s , θ ) -θ | < δ ,

with time complexity ˜ O ( δ -1 ) · poly ( ϵ -1 ) .

Theorem C.2 (Consistent quantum amplitude estimation) . Suppose U is a unitary operator such that

where p ∈ [ 0, 1 ] and | ϕ 0 ⟩ and | ϕ 1 ⟩ are normalized pure quantum states. Then for every positive reals ϵ , δ , there is a quantum algorithm that, on input O ( log ( ϵ -1 )) -bit random string s , outputs f ( s , p ) ∈ [ 0, 1 ] such that

with time complexity ˜ O ( δ -1 ) · poly ( ϵ -1 ) .

Proof. Suppose U is a unitary operator such that

Let

Similar to the analysis in Brassard et al. [9], we have

where sin 2 ( θ p ) = p ( 0 ≤ θ p < π /2 ), and

Note that | ψ ±⟩ AB are eigenvectors of Q , i.e., Q | ψ ±⟩ AB = exp ( ± i2 θ p ) | ψ ±⟩ AB .

Now applying the algorithm A of consistent phase estimation of Q by Theorem C.1 on state U | 0 ⟩ A | 0 ⟩ B ⊗| 0 ⟩ C (with an O ( log ( ϵ -1 )) -bit random string s ), we obtain

Since each of | ψ ±⟩ AB is an eigenvector of Q , it holds that, with probability ≥ 1 -ϵ ,

which implies that A ( U | 0 ⟩ A | 0 ⟩ B ⊗| 0 ⟩ C ) is O ( √ ϵ ) -close to

in trace distance, where ∣ ∣ f ( s , ± 2 θ p ) ∓ 2 θ p ∣ ∣ < δ . Measuring register C , we denote the outcome as γ , which will be either f ( s , 2 θ p ) or f ( s , -2 θ p ) . Finally, output sin 2 ( γ /2 ) as the estimate of p (which is consistent). Since sin 2 ( · ) is even and 2 -Lipschitz, the additive error is bounded by

Note that A makes ˜ O ( δ -1 ) · poly ( ϵ -1 ) queries to Q , thus our consistent amplitude estimation has quantum time complexity ˜ O ( δ -1 ) · poly ( ϵ -1 ) .

Theorem C.3 (Error-Reduced Consistent quantum amplitude estimation) . Suppose U is a unitary operator such that

where p ∈ [ 0, 1 ] and | ϕ 0 ⟩ and | ϕ 1 ⟩ are normalized pure quantum states. Then for every positive integers r and positive real δ , there is a quantum algorithm that, on input O ( r ) -bit random string s , outputs f ∗ ( s , p ) ∈ [ 0, 1 ] such that

with time complexity ˜ O ( δ -1 ) · poly ( r ) .

Proof. Consider that we divide the input random string s into r strings s 1 , s 2, . . . , sr of length O ( 1 ) . For each i ∈ [ r ] , we use Theorem C.2 with input string s i and parameter ϵ = 1/10 . So we get, for each i ∈ [ r ] ,

Now we set f ∗ ( s , p ) to be the median of the estimations f ( s i , p ) for i ∈ [ r ] . We claim it satisfies the desired property. To show that, we define random variables Xi for i ∈ [ r ] as follows:

Noticing E [ ∑ r i = 1 Xi ] ≥ 9 r /10 , and by Chernoff bound, we have:

Thus with probability at least 1 -exp ( -8 r /45 ) , we know that at least half of the estimations fall into the interval [ p -δ , p + δ ] , and then f ∗ ( s , p ) returns a correct answer.

D Details and Proofs of Fast Quantum Multi-Gibbs Sampling

We present the detailed version of the fast quantum multi-Gibss sampling. Here, we use the shorthand O Gibbs p = O Gibbs p ( 1, 0 ) , and it also means the distribution of the sample.

We first define the notion of amplitude-encoding (a unitary operator that encodes a vector in its amplitudes).

Definition D.1 (Amplitude-encoding) . A unitary operator V is said to be a β -amplitude-encoding of a vector u ∈ R n with non-negative entries, if

for all i ∈ [ n ] .

Then, as shown in Algorithm 4, we can construct a quantum multi-Gibbs sampler for a vector u if an amplitude-encoding of the vector u is given. To complete the proof of Theorem 4.2, we only have to construct an amplitude-encoding of A z (see Appendix D.2 for details).

Algorithm 4 Quantum Multi-Gibbs Sampling implementing O Gibbs u ( k , ϵ G )

Input: Sample count k , a β -amplitude-encoding V of vector u ∈ R n , polynomial P 2 β ∈ R [ x ] that satisfies Lemma 4.1 with parameter ϵ P = k ϵ 2 G /300 n .

Output: k independent samples i 1 , i 2, . . . , i k .

1: Obtain O ˜ u : | i ⟩| 0 ⟩ ↦→ | i ⟩| ˜ ui ⟩ using ˜ O ( β ) queries to V , where ui ≤ ˜ ui ≤ ui + 1 , by consistent quantum amplitude estimation (Theorem C.3).

2: Find the k largest ˜ ui 's by quantum k -maximum finding (Theorem D.3) and let S be the set of their indexes. This can be done with ˜ O ( √ nk ) queries to O ˜ u .

3: Compute ˜ u ∗ = min i ∈ S ˜ ui , and W = ( n -k ) exp ( ˜ u ∗ ) + ∑ i ∈ S exp ( ˜ ui ) .

4: for ℓ = 1, . . . , k do

5: Prepare the quantum state

6: Obtain Uu = ( V † CAB ⊗ ID )( VDAB ⊗ IC ) being a block-encoding of diag ( u ) / β . Similarly, obtain U max ˜ u being a block-encoding of diag ( max { ˜ u , ˜ u ∗ } ) /2 β .

7: Obtain U -being a block-encoding of diag ( u -max { ˜ u , ˜ u ∗ } ) /4 β by the LCU (LinearCombination-of-Unitaries) technique (Theorem D.6), using O ( 1 ) queries to Uu and U max ˜ u .

8: Obtain U exp being a block-encoding of P 2 β ( diag ( u -max { ˜ u , ˜ u ∗ } ) /4 β ) by the QSVT technique (Theorem D.7), using O ( β log ( ϵ -1 P )) queries to U -.

9: Post-select | ˜ u post ⟩ = ⟨ 0 | ⊗ a U exp | u guess ⟩| 0 ⟩ ⊗ a by quantum amplitude amplification (Theorem D.8), and obtain | ˜ u Gibbs ⟩ = | ˜ u post ⟩ / ∥| ˜ u post ⟩∥ . (Suppose U exp has a ancilla qubits.)

10: Measure | ˜ u Gibbs ⟩ in the computational basis and let i ℓ ∈ [ n ] be the outcome.

11: end for

12: Return i 1 , i 2, . . . , i k .

D.1 Useful Theorems

Theorem D.2 (Quantum state preparation, [23, 32]) . There is a data structure implemented on QRAM maintaining an array a 1 , a 2, . . . , a ℓ of positive numbers that supports the following operations.

Initialization: For any value c , set ai ← c for all i ∈ [ ℓ ] .

Assignment: For any index i and value c , set ai ← c .

State Preparation: Prepare a quantum state

Each operation costs polylog ( ℓ ) time.

Theorem D.3 (Quantum k -maximum finding, Theorem 6 of Dürr et al. [19]) . Given k ∈ [ n ] and quantum oracle O u for an array u 1 , u 2, . . . , un , i.e., for every i ∈ [ n ] ,

there is a quantum algorithm that, with probability ≥ 0.99 , finds a set S ⊆ [ n ] of cardinality | S | = k such that ui ≥ uj for every i ∈ S and j / ∈ S , using O ( √ nk ) queries to O u .

We now recall the definition of block-encoding, a crucial concept in quantum singular value transformation [20], which is used in line 9 to 12 in Algorithm 4.

Definition D.4 (Block-encoding) . Suppose A is a linear operator on b qubits, α , ϵ ≥ 0 and a is a positive integer. A ( b + a ) -qubit unitary operator U is said to be an ( α , ϵ ) -block-encoding of A , if

Definition D.5 (State Preparation Pair, Definition 28 of Gilyén et al. [20]) . Let y ∈ R n be a vector, specially in this context the number of coordinates starts from 0 . Suppose ∥ y ∥ 1 ≤ β . Let ϵ be a positive real. We call a pair of unitaries ( PL , PR ) acting on b qubits a ( β , ϵ ) -state-preparation pair for y if

and for j ∈ [ 2 b ] , j ≥ m , we require c ∗ j dj = 0 .

We now state a theorem about linear combination of unitary operators, introduced by Berry et al. [5] and Childs and Wiebe [15]. The following form is from Gilyén et al. [20]. Again we restrict ourselves to the case of real linear combinations.

Theorem D.6 (Linear Combination of Unitaries, Lemma 29 of Gilyén et al. [20]) . Let ϵ be a positive real number and y ∈ R n be a vector as in Definition D.5 with ( β , ϵ 1 ) state preparation pair ( PL , PR ) . Let { Aj } m -1 j = 0 be a set of linear operators on s qubits, and forall j , we have Uj as an ( α , ϵ 2 ) -block-encoding of Aj acting on a + s qubits. Let

Then we can implement a ( αβ , αϵ 1 + αβϵ 2 ) -block-encoding of A = ∑ m -1 j = 0 yj A j , with one query from P † L , PR , and W .

Theorem D.7 (Eigenvalue transformation, Theorem 31 of Gilyén et al. [20]) . Suppose U is an ( α , ϵ ) -block-encoding of an Hermitian operator A . For every δ > 0 and real polynomial P ∈ R [ x ] of degree d such that | P ( x ) | ≤ 1/2 for all x ∈ [ -1, 1 ] , there is an efficiently computable quantum circuit ˜ U , which is a ( 1, 4 d √ ϵ / α + δ ) -block-encoding of P ( A / α ) , using O ( d ) queries to U .

such that:

Finally, for quantum amplitude amplification without knowing the exact value of the amplitude, we need the following theorem:

Theorem D.8 (Quantum amplitude amplification, Theorem 3 of Brassard et al. [9]) . Suppose U is a unitary operator such that

where p ∈ [ 0, 1 ] is unknown and | ϕ 0 ⟩ and | ϕ 1 ⟩ are normalized pure quantum states. There is a quantum algorithm that outputs | 0 ⟩ A | ϕ 0 ⟩ B with probability ≥ 0.99 , using O ( 1/ √ p ) queries to U .

D.2 Main Proof

We generalize Theorem 4.2 as follows.

Theorem D.9. Algorithm 4 will produce k independent and identical distributed samples from a distribution that is ϵ G -close to O Gibbs u in total variation distance, in quantum time ˜ O ( β √ nk ) .

It is immediate to show Theorem 4.2 from Theorem D.9 by constructing a β -amplitude-encoding V of A z . To see this, let u = A z , then ui = ( A z ) i ∈ [ 0, β ] . By Theorem D.2, we can implement a unitary operator U QRAM z such that

Using two queries to O A , we can construct a unitary operator O ′ A such that

Let

It can be verified (see Proposition D.10) that

and thus ⟨ 0 | C ⟨ 0 | EV | 0 ⟩ C | 0 ⟩ E | i ⟩ A | 0 ⟩ B = √ ui / β | i ⟩ A | ψ i ⟩ B for some | ψ i ⟩ . Therefore, V is a β -amplitude-encoding of A z .

Now, we will show Theorem D.9 in the following.

Proof of Theorem D.9. Now we start to describe our algorithm. By our consistent quantum amplitude estimation (Theorem C.3), we choose an O ( r ) -bit random string s , then we can obtain a quantum algorithm O ˆ u such that, with probability 1 -O ( exp ( -r )) , for every i ∈ [ n ] , it computes f ∗ ( s , ui / β ) with ˜ O ( δ -1 ) · poly ( r ) queries to V , where f ∗ ( s , p ) is a function that only depends on s and p , and it holds that

for every p ∈ [ -1, 1 ] . Here, r , δ are parameters to be determined. Note that

so when applying consistent quantum amplitude estimation, we just use a controlled-XOR gate conditioned on the index and with A the target system, before every query to V .

By quantum k -maximum finding algorithm (Theorem D.3), we can find a set S ⊆ [ n ] with | S | = k such that f ∗ ( s , ui / β ) ≥ f ∗ ( s , uj / β ) for every i ∈ S and j / ∈ S with probability 0.99 -O ( √ nk exp ( -r ) ) , using O ( √ nk ) queries to O ˆ u . To obtain a constant probability, it is sufficient to choose r = Θ ( log ( n )) .

For each i ∈ S , again applying our consistent quantum amplitude estimation (Theorem C.3), we can obtain the value of f ∗ ( s , ui / β ) with probability 1 -O ( exp ( -r )) , using ˜ O ( δ -1 ) · poly ( r ) queries to V ; then we set

for all i ∈ S , with success probability 1 -O ( k exp ( -r )) and using ˜ O ( k δ -1 ) · poly ( r ) queries to V in total. It can be seen that | ˆ ui -ui | ≤ βδ for every i ∈ S .

Let ˜ ui = ˆ ui + βδ , and then we store ˜ ui for all i ∈ S in the data structure as in Theorem D.2 (which costs O ( k ) QRAM operations). Then, we calculate

by classical computation in ˜ O ( k ) time, where

By Theorem D.2, we can prepare the quantum state

in ˜ O ( 1 ) time.

Now we introduce another system D , and then let

It can be shown (see Proposition D.11) that Uu is a ( 1, 0 ) -block-encoding of diag ( u ) / β . By QRAM access to ˜ ui , we can implement a unitary operator

in ˜ O ( 1 ) time by noting that max { ˜ ui , ˜ u ∗ } = ˜ ui if i ∈ S and ˜ u ∗ otherwise. We introduce one-qubit system C , and let

It can be shown that U max ˜ u is a ( 1, 0 ) -block-encoding of diag ( max { ˜ u , ˜ u ∗ } ) /2 β . Applying the LCU technique (Theorem D.6), we can obtain a unitary operator U -that is a ( 1, 0 ) -block-encoding of diag ( u -max { ˜ u , ˜ u ∗ } ) /4 β , using O ( 1 ) queries to Uu and U max ˜ u . By the QSVT technique (Theorem D.7 and Lemma 4.1), we can construct a unitary operator U exp that is a ( 1, 0 ) -blockencoding of P 2 β ( diag ( u -max { ˜ u , ˜ u ∗ } ) /4 β ) , using O ( β log ( ϵ -1 P )) queries to U -, where

for every x ∈ [ -1, 0 ] and ϵ P ∈ ( 0, 1/2 ) is to be determined. Suppose U exp has an a -qubit ancilla system, and let | ˜ u post ⟩ = ⟨ 0 | ⊗ a U exp | u guess ⟩| 0 ⟩ ⊗ a . Note that

It can be shown (Proposition D.12) that ∥| ˜ u post ⟩∥ 2 ≥ Θ ( k / n ) ; thus by quantum amplitude amplification (Theorem D.8), we can obtain

using O ( √ n / k ) queries to U exp . By measuring | ˜ u Gibbs ⟩ on the computational basis, we return the outcome as a sample from the distribution ˜ u Gibbs ; it can be shown (Proposition D.13) that the total variation distance between ˜ u Gibbs and O Gibbs u is bounded by

By taking δ = 1/2 β and ϵ P = k ϵ 2 G /300 n , we can produce one sample from ˜ u Gibbs , using ˜ O ( β √ n / k ) queries to Uu and U max ˜ u , with ˜ O ( β √ nk ) -time precomputation.

Finally, by applying k times the above procedure (with the precomputation processed only once), we can produce k independent and identically distributed samples from ˜ u Gibbs that is ϵ Gibbs -close to the Gibbs distribution O Gibbs u , with total time complexity

D.3 Technical Lemmas

Proposition D.10. Let V defined by Equation (11) , we have

Proof.

Proposition D.11. In the proof of Theorem D.9, Uu is a ( 1, 0 ) -block-encoding of diag ( u ) / β .

Proof. To see this, for every i , j ∈ [ n ] ,

Proposition D.12. In the proof of Theorem D.9, if δ = 1/2 β , E = ∑ j ∈ [ n ] exp ( uj ) , and ϵ P = k ϵ 2 G /300 n , then

Proof. We first give an upper bound for W in terms of ui and ˜ u ∗ . Notice that ˜ ui ≤ ui + 2 βδ for all i ∈ S , we have:

Note that then we have

With this, noting that ( a -b ) 2 ≥ a 2 -2 ab for any real a and b , we have

On the other hand, a similar argument using the inequality ( a + b ) 2 ≤ a 2 + 3 ab for positive real a ≥ b gives

These yield the proof.

Proposition D.13. In the proof of Theorem D.9, the total variation distance between the two distributions ˜ u Gibbs and O Gibbs u is bounded by

Proof. Define E = ∑ j ∈ [ n ] exp ( uj ) . Let

be the intended quantum state with amplitudes the same as the Gibbs distribution O Gibbs u . The inner product between | ˜ u post ⟩ and | u Gibbs ⟩ can be bounded by:

The last step is by Cauchy's inequality. By Proposition D.12 and Equation (12), we have

Finally, we have

which is bounded by ϵ G by the choice of ϵ P .

Peer reviews:
comments: SUMMARY:
The paper presents an improved quantum online learning algorithm for approximating the Nash equilibrium of a zero-sum game. Notably, this is the first algorithm to achieve $\tilde{O}(1)$ regret with quantum speedup. The presented algorithm can then be applied to linear programming problems using the primal-dual approach. Additional contribution of the paper is a sampling method for Gibbs distribution for obtaining multiple samples with quantum speedups on both the number of samples and the number of possible outcomes.

STRENGTHS:
- Multiple novel contributions on quantum algorithms: speedups for multi-Gibbs sampling and logarithmic regret Nash equilibrium approximation
- The paper is well-written and easy to follow

WEAKNESSES:
While the presented theoretical results are interesting, I consider the necessity of QRAM as a minor weakness since (to my knowledge) it is still a highly theoretical model whose physical realizability is unclear.

QUESTIONS:
The presented algorithm requires QRAM to work. How does this compare to the quantum algorithms of regret $\tilde{O}(\sqrt{T})$, i.e., do they too require QRAM?

Typos:
- Line 285 should probably say "Line 7"

LIMITATIONS:
I do not consider the work to have any serious limitations other than the ones that follow from the limited practicality of quantum computing (for now).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-14

comments: SUMMARY:
The paper studies the online learning quantum algorithms for zero-sum games. It proposes the online quantum algorithm for zero-sum games with near-optimal regret, which makes progress to online learning algorithms in the quantum setting. The algorithm quantizes classical algorithms based on the MMW method and incorporates a fast quantum multi-sampling procedure for the Gibbs sampling problem. By developing a sample-based stochastic version of the optimistic multiplicative weight update method, this method achieves a quadratic improvement over classical algorithms and presents a fast quantum linear programming solver as an application.

STRENGTHS:
1. The paper addresses a timely and relevant topic in quantum learning theory, proposing the first online quantum algorithm for zero-sum games with near-optimal regret.
2. The algorithm achieves a quadratic improvement over classical algorithms for computing the approximate Nash equilibrium.
3. The paper presents a fast quantum linear programming solver as an application.

WEAKNESSES:
1. The main algorithm is derived from classical methods, and the explanation of the features/contributions of quantum computing in achieving the speedup or advantage could be further elaborated.
2. As a theory paper, it could benefit from a clear and well-motivated example demonstrating the application of the proposed algorithm. Including such an example would help readers better understand the potential practical implications and usefulness of the work.
3. Could the authors discuss whether this online quantum algorithm could be dequantized?
4. In the abstract, the authors claim that "a fast quantum linear programming solver" but haven't identified under which setting.

QUESTIONS:
Refer to the part about weaknesses.

LIMITATIONS:
NA.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of developing quantum, low-regret, algorithms for solving zero-sum games. This paper provides a quantum algorithm which matches the state-of-the-art for solving zero-sum games while improving upon the regret of the associated algorithm. The paper achieves this result by providing a stochastic variant of optimistic mirror descent which they then implement by a quantum algorithm for developing multiple samples from a suitable Gibbs distribution. Additionally, the paper discusses implications and lower bounds for linear programming more broadly.\n\nSTRENGTHS:\nZero sum games are an important optimization problem in many disciplines. The mere fact that the paper shows how to depart from previous quantum algorithms for solving zero sum games (in using optimistic mirror descent versus mirror descent) while retaining state-of-the-art bounds is interesting and potentially useful for further study of the problem. Additionally, by following this alternative approach the paper obtains improved regret and a type of parallelism in solving the problem. This is an interesting advancement for the study of the problem. Finally, this alternative optimization method gives rise to a different Gibb’s distribution sampling problem than the one considered in prior work. Furthermore, the paper gives an interesting algorithm for solving it which could motivate further study. Finally, the additional discussion of linear programming provided in the paper (though seemingly straightforward) is a nice addition as well.\n\nWEAKNESSES:\nIt is not clear how substantial the departure from this paper from prior work is, how novel the advancements are, and how complete the comparison of this paper to prior work is. While I think the advancements of the paper are interesting and potentially exciting (as discussed in the strength section), I fear that the paper does not well-articulate them or situate them. For example:\n\n* A comparison of the techniques in the Gibb’s sampling algorithm of this procedure and those in prior quantum-zero sum games algorithm does not seem to be provided. This seems particularly important as the techniques were key to recent advances in quantum algorithms for solving zero-sum-games.\n* It is not clear what exactly the benefits of low regret are over prior work and why previous results couldn’t be used to obtain low-regret (e.g. by only considering a subsequence of the iterates taken). I think the low-regret aspect is interesting and this algorithm is potentially more parallel than prior algorithm’s with state-of-the-art complexity and the sampling problems considered may be simpler than in prior work; however I am not sure that the paper argues this clearly.\n* It is not clear what obstacles were present to obtaining the result. It would help if the paper clarified whether moving from mirror-descent to optimistic mirror-descent was the main insight and the rest was straightforward (though technical and important) or if further insights were needed. \n\nSome additional comments along this line are given “Questions.”\n\nQUESTIONS:\nMy main questions / comments are the items in the previous “Weaknesses” section. Beyond what was written there: How does the Gibb’s sampling compare to prior work? Is it simpler in any way? The same? Is a completely different techniques used? Is there a major insight of the paper beyond the use of stochastic optimistic mirror descent? \n\nBeyond this, below are some more detailed suggestions, questions, and comments:\n\n* Line 1: “for zero-sum games” --> “for solving zero-sum games”\n* Line 4: “yielding a quadratic improvement” as written, I am concerned this is suggesting that it is the first such improvement.\n* Line 27: “For, this approximation task, online learning becomes significant” – I am not exactly sure what was meant by this.\n* Line 33: It might be beneficial to define regret or explain the model more clearly earlier so the improvement over prior work is clearer. (Regret is well-known in general, but when stated for a static optimization problem, the definition is less clear.)\n* Line 39: I believe zero-sum games were shown to be solvable in $\\tilde{O}(1/\\epsilon)$ iterations earlier, e.g. “Prox-method with rate of convergence O (1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems” by Nemirovski in 2004. I’m less sure about whether this works in regret context but in any event I believe should be mentioned when discussing the history of solving zero-sum games.\n* Line 81: it might be good to clarify whether this is a new feature of this algorithm or not\n* Line 130 – 135: it might be good to clarify why prior work could or could not be used for this problem\nMy main questions / comments are the items in the previous “Weaknesses” section. Beyond what was written there: How does the Gibb’s sampling compare to prior work? Is it simpler in any way? The same? Is a completely different techniques used? Is there a major insight of the paper beyond the use of stochastic optimistic mirror descent? \n\nBeyond this, below are some more detailed suggestions, questions, and comments:\n\n* Line 1: “for zero-sum games” --> “for solving zero-sum games”\n* Line 4: “yielding a quadratic improvement” as written, I am concerned this is suggesting that it is the first such improvement.\n* Line 27: “For, this approximation task, online learning becomes significant” – I am not exactly sure what was meant by this.\n* Line 33: It might be beneficial to define regret or explain the model more clearly earlier so the improvement over prior work is clearer. (Regret is well-known in general, but when stated for a static optimization problem, the definition is less clear.)\n* Line 39: I believe zero-sum games were shown to be solvable in $\\tilde{O}(1/\\epsilon)$ iterations earlier, e.g. “Prox-method with rate of convergence O (1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems” by Nemirovski in 2004. I’m less sure about whether this works in regret context but in any event I believe should be mentioned when discussing the history of solving zero-sum games.\n* Line 81: it might be good to clarify whether this is a new feature of this algorithm or not\n* Line 130 – 135: it might be good to clarify why prior work could or could not be used for this problem\n\nLIMITATIONS:\nThis is primarily a theory paper, and it is unclear how applicable this is. However, the weaknesses and questions discussed reflect limitations for which further discussion might be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors consider the task of computing the eps-approximate Nash equilibrium of zero-sum games. In particular, for a matrix A in R^{m x n}, the goal is to find the minimum distributions x,y s.t., max_y x^t A y - min_x x^t A y is at most epsilon, given quantum query access to A.  In this paper the authors are further concerned with the online learning/regret model, i.e., in each round the players need to choose probability distributions (u,v) and their goal is to pick these distributions (u,v) such that it is as close as possible to the optimal choice of eps-Nash equilibrium. \n\nPrior known quantum algorithms for computing the Nash equilibrium ran in time sqrt{m+n}/eps^2.5 but achieved a regret of sqrt{T}  for a T round algorithm. In this paper, the authors show how to obtain a regret of O(1) with the same complexity as sqrt{m+n}/eps^2.5.\n\nSTRENGTHS:\nThe main strength of this paper is in a way it solves an important question of computing Nash equilibrium with a O(1)-based regret algorithm.  Using this, they algorithm obtain a state of the art quantum algorithm for linear programming solvers. I think both of these contributions are interesting on their own right!\n\nWEAKNESSES:\nAs for weakness, i feel there are a couple of weaknesses:\n\n(i) The main techniques used in this paper, aren't very novel. Indeed, they do improve upon prior works and need new tools, but the novelty of the main contributions towards obtaining their speedups doesn't make it a strong contribution. For example, obtaining k copies of a Gibbs state in time sqrt{nk} (compared to the trivial k sqrt{n} algorithm) isn't too surprising (indeed new, but not surprising given Hamoudi's algorithm and also complexity of Grover's search with k marked items). \n\n(ii) The speedup to obtain the regret bound is by quantizing the optimistic  online algorithm. I think there are some subtelties when quantizing this algorithm, but the main ideas involved aren't novel.\n\nHaving said the above, i think overall for the ML community, i think the main results and these interesting quantum subroutines above are nice.\n\nQUESTIONS:\nI think the definitions of online learning and regret can be defined much better. In particular, the online learning/regret model for Nash equilibrium is very messily written and I'd have appreciated a better explanation.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}