Title: Payoff-based Learning with Matrix Multiplicative Weights in Quantum Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: quantum games

Matrix Multiplicative Weights

zero-sum games

Nash equilibrium
Abstract: In this paper, we study the problem of learning in quantum games - and other classes of semidefinite games - with scalar, payoff-based feedback.
For concreteness, we focus on the widely used matrix multiplicative weights (MMW) algorithm and, instead of requiring players to have full knowledge of the game (and/or each other's chosen states), we introduce a suite of minimal-information matrix multiplicative weights (3MW) methods tailored to different information frameworks.
The main difficulty to attaining convergence in this setting is that, in contrast to classical finite games, quantum games have an infinite continuum of pure states (the quantum equivalent of pure strategies), so standard importance-weighting techniques for estimating payoff vectors cannot be employed.
Instead, we borrow ideas from bandit convex optimization and we design a zeroth-order gradient sampler adapted to the semidefinite geometry of the problem at hand.
As a first result, we show that the 3MW method with deterministic payoff feedback retains the $\mathcal{O}(1/\sqrt{T})$ convergence rate of the vanilla, full information MMW algorithm in quantum min-max games, even though the players only observe a single scalar.
Subsequently, we relax the algorithm's information requirements even further and we provide a 3MW method that only requires players to observe a random realization of their payoff observable, and converges to equilibrium at an $\mathcal{O}(T^{-1/4})$ rate.
Finally, going beyond zero-sum games, we show that a regularized variant of the proposed 3MW method guarantees local convergence with high probability to all equilibria that satisfy a certain first-order stability condition.

Payoff-based Learning with

Abstract

In this paper, we study the problem of learning in quantum games - and other classes of semidefinite games - with scalar, payoff-based feedback. For concreteness, we focus on the widely used matrix multiplicative weights (MMW) algorithm and, instead of requiring players to have full knowledge of the game (and/or each other's chosen states), we introduce a suite of minimal-information matrix multiplicative weights (3MW) methods tailored to different information frameworks. The main difficulty to attaining convergence in this setting is that, in contrast to classical finite games, quantum games have an infinite continuum of pure states (the quantum equivalent of pure strategies), so standard importance-weighting techniques for estimating payoff vectors cannot be employed. Instead, we borrow ideas from bandit convex optimization and we design a zeroth-order gradient sampler adapted to the semidefinite geometry of the problem at hand. As a first result, we show that the 3MW method with deterministic payoff feedback retains the O ( 1 / √ 𝑇 ) convergence rate of the vanilla, full information MMW algorithm in quantum min-max games, even though the players only observe a single scalar. Subsequently, we relax the algorithm's information requirements even further and we provide a 3MW method that only requires players to observe a random realization of their payoff observable, and converges to equilibrium at an O ( 𝑇 -1 / 4 ) rate. Finally, going beyond zero-sum games, we show that a regularized variant of the proposed 3MW method guarantees local convergence with high probability to all equilibria that satisfy a certain first-order stability condition.

1 Introduction

The integration of quantum information theory into computer science and machine learning [4, 51, 64] has the potential ofy providing faster and more efficient computing resources, new encryption and security protocols, and improved machine learning algorithms, enabling advancements in areas such as quantum cryptography, shadow tomography, quantum GANs, and adversarial learning [1, 15, 18, 37]. As a well-known example, Google's 'Sycamore' 54 -qubit processor recently showcased this 'quantum advantage' by training an autonomous vehicle model in less than 200 seconds [4], a fact made possible by the ability of quantum computers to prepare superpositions of qubits that exceed the operational capabilities of standard Boolean gates.

Deploying such models within a multi-agent context, such as the utilization of QGANs or autonomous vehicles, leads to a significant transformation compared to classical non-cooperative environments.

Indeed, unlike classical games (where a mixed strategy is a probabilistic mixture of the underlying pure strategies), quantum games utilize mixed states, which represent probabilistic mixtures of quantum projectors. As a consequence, a mixed quantum state can yield payoffs that cannot be expressed as a convex combination of classical pure strategies.

In light of this, quantum learning has drawn significant attention in recent years [2, 27, 28, 33, 34, 60]. In a multi-agent context, the most widely used framework is the so-called matrix multiplicative weights (MMW) algorithm [2, 16, 27, 28, 35]: First introduced by Tsuda et al. [59] in the context of matrix and dictionary learning, MMW can be viewed as a semidefinite analogue of the standard Hedge / EXP3 methods for multi-armed bandits [6, 36, 61], and is a special case of the mirror descent family of algorithms [47]. Specifically, in the contrete setting of two-player, zero-sum quantum games, Jain & Watrous [27] showed that players using the MMW algorithm can learn an 𝜀 -equilibrium in O ( 1 / 𝜀 2 ) iterations - or, in terms of speed of convergence after 𝑇 iterations, they converge to equilibrium at a O ( 1 / √ 𝑇 ) rate.

To the best of our knowledge, this result remains the tightest known bound for equilibrium learning in quantum games - and the more general class of semidefinite games [26]. At this point, we highlight that we focus on classical computing algorithms for solving quantum games, unlike recent results [10, 22] that employ quantum algorithms to solve classical games and semidefinite programs. Building on [27], Jain et al. [28] studied its continuous-time analogue - the quantum replicator dynamics (QRD) - in quantum min-max games, focusing on the recurrence and volume conservation properties of the players' actual trajectory of play. Going beyond the min-max case, [38] examined the convergence of the dynamics of 'follow the quantum leader' (FTQL), a class of continuous-time dynamics that includes the QRD as a special case. The main result of [38] was that the only states that are asymptotically stable under the (continuous-time) dynamics of FTQL are those that satisfy a certain first-order stationarity condition known as variational stability [44, 46]. In a similar line of work, Lin et al. [35] studied the continuous-time QRD, and discrete-time MMW in quantum potential games, utilizing a Riemannian metric to obtain a gradient flow in the spirit of [41, 42].

Our contributions in the context of previous work. All works mentioned above, in both continuous and discrete time, assume full information, i.e., players have access to their individual payoff gradients - which, among others, might imply that they have full knowledge of the game. However, this condition is rarely met in online learning environments where players only observe their in-game payoffs; this is precisely the starting point of our paper which aims to derive a convergent payoff-based, gradient-free variant of MMW algorithm for learning in quantum games.

Amajor roadblock in this is that standard approaches from learning in finite games fail in the quantum setup for two reasons: First and foremost, there is a continuum of pure states available to every player, unlike classical finite games where there is only a finite set of pure actions. Second, even after the realization of the pure states of the players, there is an inherent uncertainty and randomness due to the payoff-generating quantum process (an aspect that has no classical counterpart). To overcome this hurdle, we employ a continuous-action reformulation of quantum games, and we leverage techniques from bandit convex optimization for estimating the players' payoff gradients.

Our first contribution is a variant of MMW that only requires mixed payoff observations and achieves an O ( 1 / √ 𝑇 ) equilibrium convergence rate in two-player zero-sum quantum games, matching the rate of the full information MMW in [27]. Then, to account for information-starved environments where players are only able to observe their in-game, realized payoff observable, we also develop a bandit variant of MMW which utilizes a single-point gradient estimation technique in the spirit of [55] and achieves an O ( 𝑇 -1 / 4 ) equilibrium convergence rate. Finally, we also examine the behavior of the MMW algorithm with bandit information in general 𝑁 -player games, where we show that variationally stable equilibria are locally attracting with high probability.

Importantly, the above results transfer to more general games with a semidefinite structure - such as multi-agent covariance matrix optimization in signal processing, energy efficiency maximization in multi-antenna systems, etc. [43, 45, 62]. While we do not provide a complete theory, we discuss a number of non-quantum applications that showcase how our results can be generalized further.

Notation. Given a (complex) Hilbert space H , we will use Dirac's bra-ket notation and write | 𝜓 ⟩ for an element of H and ⟨ 𝜓 | for its adjoint; otherwise, when a specific basis is implied by the context, we will use the dagger notation ' † ' to denote the Hermitian transpose 𝜓 † of 𝜓 . We will also write

ℍ 𝑑 for the space of 𝑑 × 𝑑 Hermitian matrices, and ℍ 𝑑 + for the cone of positive-semidefinite matrices in ℍ 𝑑 . Finally, we denote by ∥ A ∥ 𝐹 = √︁ tr [ A † A ] the Frobenius norm of A in ℍ 𝑑 .

2 Problem setup and preliminaries

We begin by reviewing some basic notions from the theory of quantum games, mainly intended to set notation and terminology; for a comprehensive introduction, see [23]. To streamline our presentation, we introduce the primitives of quantum games in a 2 -player setting before treating the general case.

Quantum games. Following [20, 23], a 2 -player quantum game consists of the following:

Each player 𝑖 ∈ N : = { 1 , 2 } has access to a complex Hilbert space H 𝑖 GLYPH<27> ℂ 𝑑 𝑖 describing the set of (pure) quantum states available to the player (typically a discrete register of qubits). A quantum state is an element 𝜓 𝑖 of H 𝑖 with unit norm, so the set of pure states is the unit sphere Ψ 𝑖 : = { 𝜓 𝑖 ∈ H 𝑖 : ∥ 𝜓 𝑖 ∥ 𝐹 = 1 } of H 𝑖 . We will write Ψ : = Ψ 1 × Ψ 2 for the space of all ensembles 𝜓 = ( 𝜓 1 , 𝜓 2 ) of pure states 𝜓 𝑖 ∈ Ψ 𝑖 that are independently prepared by each player.

The rewards that players receive are based on their individual payoff functions 𝑢 𝑖 : Ψ → ℝ , and they are derived through a positive operator-valued measure (POVM) quantum measurement process. Following [17], this unfolds as follows: Given a finite set of measurement outcomes Ω that a referee can observe from the players' quantum states (e.g., measure a player-prepared qubit to be 'up' or 'down'), each outcome 𝜔 ∈ Ω is associated to a positive semi-definite operator P 𝜔 : H → H defined on the tensor product H : = H 1 ⊗ H 2 of the players' individual state spaces. We further assume that P 𝜔 ∈ Ω P 𝜔 = I so the probability of observing 𝜔 ∈ Ω at state 𝜓 ∈ Ψ is 𝑃 𝜔 ( 𝜓 ) = ⟨ 𝜓 1 ⊗ 𝜓 2 | P 𝜔 | 𝜓 1 ⊗ 𝜓 2 ⟩ .

The payoff of each player is then generated by this measurement process via a payoff observable 𝑈 𝑖 : Ω → ℝ : specifically, the measurement 𝜔 is drawn from Ω based on the players' state profile 𝜓 = ( 𝜓 1 , 𝜓 2 ) , and each player 𝑖 ∈ N receives as reward the quantity 𝑈 𝑖 ( 𝜔 ) . Accordingly, the player's expected payoff at state 𝜓 ∈ Ψ is 𝑢 𝑖 ( 𝜓 ) : = ⟨ 𝑈 𝑖 ⟩ ≡ P 𝜔 𝑃 𝜔 ( 𝜓 ) 𝑈 𝑖 ( 𝜔 ) .

A quantum game is then defined as a tuple Q ≡ Q ( N , Ψ , 𝑢 ) with players, states, and payoff as above.

Mixed states. Apart from pure states, each player 𝑖 ∈ N may prepare probabilistic mixtures thereof, known as mixed states . These mixed states differ from mixed strategies used in classical, finite games as they do not correspond to convex combinations of their pure counterparts; instead, given a family of pure quantum states 𝜓 𝑖 𝛼 𝑖 ∈ Ψ 𝑖 indexed by 𝛼 𝑖 ∈ A 𝑖 , a mixed state is described by a density matrix of the form

where the mixing weights 𝑥 𝑖 𝛼 𝑖 ≥ 0 of each 𝜓 𝑖 𝛼 𝑖 are normalized so that tr X 𝑖 = 1 . By Born's rule, this means that the probability of observing 𝜔 ∈ Ω under X = ( X 1 , X 2 ) is

where 𝜓 𝛼 = 𝜓 1 , 𝛼 1 ⊗ 𝜓 2 , 𝛼 2 . Therefore, in a slight abuse of notation, the expected payoff of player 𝑖 ∈ N under X will be 𝑢 𝑖 ( X ) = P 𝛼 ∈ A 𝑥 𝛼 𝑢 𝑖 ( 𝜓 𝛼 ) . which, equivalently, can be written as:

This gives a succint representation of the payoff structure of Q - see also Eq. (5) below.

Continuous game reformulation. In view of the above, treating a quantum game as a 'tensorial' extension of a finite game can be misleading. For our purposes, it would be more suitable to treat a quantum game as a continuous game where each player 𝑖 ∈ N controls a matrix variable X 𝑖 drawn from the 'spectraplex' defined as X 𝑖 = { X 𝑖 ∈ ℍ 𝑑 𝑖 + : tr X 𝑖 = 1 } . In this interpretation, the players' payoff functions 𝑢 𝑖 : X ≡ X 1 × X 2 → ℝ are linear in each player's density matrix X 𝑖 ∈ X 𝑖 , 𝑖 ∈ N . Since 𝑢 1 , 𝑢 2 are linear in X 1 and X 2 , the individual payoff gradients of each player will be given by

so we can further write each player's payoff function as

Since X is compact and each 𝑢 𝑖 is multilinear in X , the players' payoff functions are automatically bounded, Lipschitz continuous and Lipschitz smooth, i.e., there exist constants 𝐵 𝑖 , 𝐺 𝑖 and 𝐿 𝑖 , 𝑖 ∈ N , such that, for all X , X ′ ∈ X , we have:

Boundedness: | 𝑢 𝑖 ( X )| ≤ 𝐵 𝑖

Lipschitz continuity: | 𝑢 𝑖 ( X ) - 𝑢 𝑖 ( X ′ )| ≤ 𝐺 𝑖 ∥ X - X ′ ∥ 𝐹

Lipschitz smoothness: ∥ V 𝑖 ( X ) - V 𝑖 ( X ′ )∥ 𝐹 ≤ 𝐿 𝑖 ∥ X - X ′ ∥ 𝐹

Nash equilibrium. The most widely used solution concept in game theory is that of a Nash equilibrium (NE). In our context, it is mixed profile X ∗ ∈ X from which no player has incentive to deviate, i.e., 𝑢 1 ( X ∗ ) ≥ 𝑢 1 ( X 1; X ∗ 2 ) and 𝑢 2 ( X ∗ ) ≥ 𝑢 2 ( X ∗ 1 ; X 2 ) for all X 1 ∈ X 1 , X 2 ∈ X 2 . Since X 𝑖 is convex and 𝑢 𝑖 linear in X 𝑖 , the existence of Nash equilibria follows from the Debreu's theorem [19].

Zero-sum quantum games. In the case where 𝑢 1 = -𝑢 2 , and setting L : X 1 × X 2 → ℝ , the Nash equilibria of Q are the saddle points of L , i.e., the solutions of the minimax problem

By Sion's minimax theorem [54], the set of Nash equilibria is nonempty. Then, given a Nash equilibrium X ∗ , we define the duality gap of X = ( X 1 , X 2 ) as

so Gap L ( X ) ≥ 0 with equality if and only if X is itself a Nash equilibrium. In particular, X is an 𝜀 -Nash equilibrium of Q if and only if Gap L ( X ) ≤ 𝜀 .

Other semidefinite games. In addition to quantum games, our framework can also be used for learning in other classes of games with a semidefinite structure as per [26, 45]. As an example, consider the problem of covariance matrix optimization in vector Gaussian multiple-access channels [9, 43, 57, 62]. In this case, there is a finite set of players indexed by 𝑖 ∈ N = { 1 , . . . , 𝑁 } ; each player 𝑖 ∈ N picks a unit-trace semidefinite matrix X 𝑖 ∈ X 𝑖 and their payoff is given by the Shannon-Telatar capacity formula [57], viz.

where each H 𝑖 is a player-specific gain matrix [58]. Even though 𝑢 𝑖 is no longer multilinear in X , the algorithms we derive later in the paper can be applied to this setting essentially verbatim.

3 The matrix multiplicative weights algorithm

Throughout the sequel, we will focus on equilibrium learning in quantum - and semidefinite - games. In the context of two-player, zero-sum quantum games, the state-of-the-art method is based on the so-called matrix multiplicative weights (MMW) algorithm [7, 27, 29, 59] which updates as

In the above, ( a ) X 𝑡 = ( X 1 ,𝑡 , X 2 ,𝑡 ) denotes the players' density matrix profile at each stage 𝑡 = 1 , 2 , . . . of the process; ( b ) V 𝑖 ( X 𝑡 ) is the payoff gradient of player 𝑖 ∈ N under X 𝑡 ; ( c ) Y 𝑡 is an auxiliary state matrix that aggregates gradient steps over time; and ( d ) 𝛾 𝑡 > 0 , 𝑡 = 1 , 2 , . . . , is a learning rate (or step-size) parameter that can be freely tuned by the players.

Importantly, as stated, (MMW) requires full information at the player end: specifically, at each stage 𝑡 = 1 , 2 , . . . of the process, each player 𝑖 ∈ N must receive their individual payoff gradient V 𝑖 ( X 𝑡 ) in order to perform the gradient update step in (MMW). Under this assumption, Jain & Watrous [27] showed that the induced empirical frequency of play

√

converges to equilibrium at a rate of O ( 1 / 𝑇 ) as per the formal result below:

Theorem 1 (Jain & Watrous [27]) . Suppose that each player of a 2 -player zero-sum game Q follows (MMW) for 𝑇 epochs with learning rate 𝛾 = 𝐺 -1 √︁ 2 𝐻 / 𝑇 where 𝐻 = log ( 𝑑 1 𝑑 2 ) . Then the players' empirical frequency of play enjoys the bound

In particular, if (MMW) is run for 𝑇 = O ( 1 / 𝜀 2 ) iterations, ¯ X 𝑇 will be an 𝜀 -Nash equilibrium of Q .

To the best of our knowledge, this guarantee of Jain & Watrous [27] remains the tightest known bound for Nash equilibrium learning in 2 -player zero-sum quantum games. At the same time, Theorem 1 hinges on the players having perfect access to their individual gradients - which, among others, might entail full knowledge of the game, observing the other player's density matrix, etc. Our goal in the sequel will be to relax precisely this assumption and develop a payoff-based variant of (MMW) that can be employed without stringent information and observability requirements as above.

4 Matrix learning without matrix feedback

In an online learning framework, it is more realistic to assume that players observe only the outcome of their actions - i.e., their individual payoffs. In this information-starved, payoff-based setting, our main goal will be to employ a minimal-information matrix multiplicative weights (3MW) algorithm that updates as

where ˆ V 𝑖,𝑡 is some payoff-based estimate of the payoff gradient V 𝑖 ( X 𝑡 ) of player 𝑖 at X 𝑡 , and all other quantities are defined as per (MMW). In this regard, the main challenge that arises is how to reconstruct each player's payoff gradient matrices when they are not accessible via an oracle.

4.1. The classical approach: Importance weighted estimators. In the context of classical, finite games and multi-armed bandits, a standard approach for reconstructing ˆ V 𝑖,𝑡 is via the so-called importance weighted estimator (IWE) [12, 14, 32]. To state it in the context of finite games, assume that each player has at their disposal a finite set of pure strategies 𝛼 𝑖 ∈ A 𝑖 , and if each player plays ˆ 𝛼 𝑖 ∈ A 𝑖 , then, in obvious notation, their individual payoff will be ˆ 𝑢 𝑖 = 𝑢 𝑖 ( ˆ 𝛼 𝑖 ; ˆ 𝛼 -𝑖 ) . Then, if each player is using a mixed strategy 𝑥 𝑖 ∈ Δ ( A 𝑖 ) to draw their chosen action ˆ 𝛼 𝑖 , the importance weighted estimator (IWE) for the payoff of the (possibly unplayed) action 𝛼 𝑖 ∈ A 𝑖 of player 𝑖 is defined as

with the assumption that 𝑥 𝑖 has full support, i.e., each action 𝛼 𝑖 ∈ A 𝑖 has strictly positive probability 𝑥 𝑖 𝛼 𝑖 of being chosen by the 𝑖 -th player. 1

This approach has proven extremely fruitful in the context of multi-armed bandits and finite games where (IWE) is an essential ingredient of the optimal algorithms for each context [5, 12, 32, 65]. However, in our case, there are two insurmountable difficulties in extending (IWE) to a quantum context: First and foremost, the quantum regime is characterized by a continuum of pure states with highly correlated payoffs (in the sense that quantum states that are close in the Bloch sphere will have highly correlated POVM payoff observables); this comes in stark contrast to the classical regime of finite normal-form games, where players only have to contend with a finite number of actions (with no prior payoff correlations between them). Secondly, even after the realization of the pure states of the players, there is an inherent uncertainty and randomness due to the quantum measurement process that is invovled in the payoff-generating process; as such, the players' payoffs are also affected by an exogenous source of randomness which is altogether absent from (IWE).

Our approach to tackle these issues will be to exploit the reformulation of a quantum game as a continuous game with multilinear payoffs over the spectraplex (or, rather, a product thereof), and use ideas from bandit convex optimization - in the spirit of [21, 31] - to estimate the players' payoff gradients with minimal, scalar information requirements.

4.2. Gradient estimation via finite-difference quotients on the spectraplex. To provide some intuition for the analysis to come, consider first a single-variable smooth function 𝑓 : ℝ → ℝ and a point 𝑥 ∈ ℝ . Then, for error tolerance 𝛿 > 0 , a two-point estimate of the derivative of 𝑓 at 𝑥 is given by the expression

Going to higher dimensions, letting 𝑓 : ℝ 𝑑 → ℝ be a smooth function, { 𝑒 1 , . . . , 𝑒 𝑑 } be the standard basis of ℝ 𝑑 and 𝑠 drawn from { 𝑒 1 , . . . , 𝑒 𝑑 } uniformly at random, the estimator

is a O ( 𝛿 ) -approximation of the gradient, i.e., ∥ 𝔼 𝑠 [ ˆ 𝑓 𝑥 ] - ∇ 𝑓 ( 𝑥 )∥ 𝐹 = O ( 𝛿 ) . This idea is the basis of the Kiefer-Wolfowitzs stochastic approximation scheme [30] and will be the backbone of our work.

Now, to employ this type of estimator for a function over the set of density matrices X in ℍ 𝑑 , we need to ensure two things: (i) the feasibility of the sampling direction , and (ii) the feasibility of the evaluation point . The first caveat is due to the fact that the set of the density matrices forms a lower dimensional manifold in the set of Hermitian operators, and therefore, not all directions from a base of ℍ 𝑑 are feasible. The second one is due to the fact that X is bounded, thus, even if the sampling direction is feasible, the evaluation point can lie outside the set X . We proceed to ensure all this in a series of concrete steps below.

Sampling Directions. We begin with the issue of defining a proper sampling set for the estimator's finite-difference directions. To that end, we will first construct an orthonormal basis of the tangent hull Z = { Z ∈ ℍ 𝑑 : tr Z = 0 } of X , i.e., the subspace of traceless matrices of ℍ 𝑑 . Note that if Z ∈ Z then for any X ∈ ℍ 𝑑 it holds ( a ) X + Z ∈ ℍ 𝑑 , and ( b ) tr [ X + Z ] = tr [ X ] .

Denoting by 𝚫 𝑘ℓ ∈ ℍ 𝑑 the matrix with 1 in the ( 𝑘, ℓ ) -position and 0 's everywhere else, it is easy to see that the set GLYPH<8> { 𝚫 𝑗 𝑗 } 𝑑 𝑗 = 1 { e 𝑘ℓ } 𝑘<ℓ , { ˜ e 𝑘ℓ } 𝑘<ℓ GLYPH<9> is an orthonormal basis of ℍ 𝑑 , where

for 1 ≤ 𝑘 < ℓ ≤ 𝑑 , where 𝑖 is the imaginary unit with 𝑖 2 = -1 . The next proposition provides a basis for the subspace Z , whose proof lies in the appendix.

. n o .

In the sequel, we will use this basis as an orthnormal sampler from which to pick the finite-difference directions for the estimation of V .

Feasibility Adjustment. After establishing an orthonormal basis for Z as per Proposition 1, we readily get that for any X ∈ X , any Z ∈ E ± : = GLYPH<8> {± E 𝑗 } 𝑑 -1 𝑗 = 1 , {± e 𝑘ℓ } 𝑘<ℓ , {± ˜ e 𝑘ℓ } 𝑘<ℓ GLYPH<9> and 𝛿 > 0 , the point X + 𝛿 Z belongs to Z . However, depending on the value of the exploration parameter 𝛿 and the distance of X from the boundary of X , the point X + 𝛿 Z ∈ ℍ 𝑑 may fail to lie in X due to violation of the positive-semidefinite condition. On that account, we now treat the latter restriction, i.e., the feasibility of the evaluation point .

To tackle this, the idea is to transfer the point X toward the interior of X and move along the sampled direction from there. For this, we need to find a reference point R ∈ ri ( X ) and a 'safety net' 𝑟 > 0 such that R + 𝑟 Z ∈ X for any Z ∈ E ± . Then, for 𝛿 ∈ ( 0 , 𝑟 ) , the point

lies in ri ( X ) , and moving along Z ∈ E ± , the point X ( 𝛿 ) + 𝛿 Z = ( 1 -𝛿 𝑟 ) X + 𝛿 𝑟 ( R + 𝑟 Z ) remains in X as a convex combination of two elements in X . The following proposition provides an exact expression for R and 𝑟 , which we will use next to guarantee the feasibility of the sampled iterates.

Proposition 2. Let R = 1 𝑑 P 𝑑 𝑗 = 1 𝚫 𝑗 𝑗 . Then, for 𝑟 = min GLYPH<26> 1 √ 𝑑 ( 𝑑 -1 ) , √ 2 𝑑 GLYPH<27> , it holds that R + 𝑟 Z ∈ X for any direction Z ∈ E ± .

5 Bandit learning in zero-sum quantum games

With all these in hand, we are now ready to proceed to the presentation of the MMW with limited feedback information. To streamline our presentation, before delving into the more difficult 'bandit feedback' case - where each player 𝑖 ∈ N only observes the realized payoff observable 𝑈 𝑖 ( 𝜔 ) - we begin with the simpler case where players observe their mixed payoffs 𝑢 𝑖 at a given profile X ∈ X .

5.1. Learning with mixed payoff observations. Our main idea to exploit the observation of mixed payoffs and the finite-difference sampling to the fullest will be to introduce a 'coordination phase' where players take a sampling step before updating their state variables and continue playing. In more detail, we will take an approach similar to Bervoets et al. [8] and assume that players alternate between an 'exploration' and an 'exploitation' update that allows them to sample the landscape of L efficiently at each iteration. Concretely, writing X 𝑡 and 𝛿 𝑡 for the players' state profile and sampling radius 𝛿 𝑡 at stage 𝑡 = 1 , 2 , . . . , the sequence of events that we envision proceeds as follows:

Step 1. Draw a sampling direction Z 𝑖,𝑡 ∈ E 𝑖 and 𝑠 𝑖,𝑡 ∈ {± 1 } uniformly at random.

Step 3. Approximate V 𝑖 ( X 𝑡 ) via the two-point estimator (2PE):

where 𝐷 𝑖 = 𝑑 2 𝑖 -1 is the dimension of ℍ 𝑑 𝑖 , and 𝐷 : = max 𝑖 ∈ N 𝐷 𝑖 .

The main guarantee of the resulting (3MW) + (2PE) algorithm may then be stated as follows:

Theorem 2. Suppose that each player of a 2 -player zero-sum game Q follows (3MW) for 𝑇 epochs with learning rate 𝛾 , sampling radius 𝛿 , and gradient estimates provided by (2PE) . Then the players' empirical frequency of play enjoys the duality gap guarantee

where 𝐻 = log ( 𝑑 1 𝑑 2 ) . In particular, for 𝛾 = ( 𝐷𝐺 ) -1 √︁ 𝐻 /( 8 𝑇 ) and 𝛿 = ( 𝐺 / 𝐿 ) √︁ 𝐻 /( 8 𝑇 ) , the players enjoy the equilibrium convergence guarantee

Compared to Theorem 1, the convergence rate (16) of Theorem 2 is quite significant because it only differs by a factor which is linear in the dimension of the ambient space and otherwise maintains the same O ( √ 𝑇 ) dependence on the algorithm's runtime. In this regard, Theorem 2 shows that the 'explore-exploit' sampler underlying (2PE) is essentially as powerful as the full information framework of Jain & Watrous [27] - and this, despite the fact that players no longer require access to the gradient matrix V of L . This echoes a range of previous findings in stochastic convex optimization for the efficiency of two-point samplers [3, 53], a similarity we find particularly surprising given the stark differences between the two settings - non-commutativity, min-max versus min-min landscape. The key ingredients for the equilibrium convergence rate of Theorem 2 are the two technical results below. The first is a feedback-agnostic 'energy inequality' which is tied to the update structure of (MMW) and is stated in terms of the quantum relative entropy function

for P , X ∈ X with X ≻ 0 . Concretely, we have the following estimate.

Lemma 1. Fix some P ∈ X , and let X 𝑡 , X 𝑡 + 1 be two successive iterates of (3MW) , without any assumptions for the input sequence ˆ V 𝑡 . We then have

The proof of Lemma 1 follows established techniques in the theory of (MMW), so we defer a detailed discussion to the appendix. The second result that we will need is tailored to the estimator (2PE) and provides a tight estimate of its moments conditioned on the history F 𝑡 = F ( X 1 , . . . , X 𝑡 ) of X 𝑡 .

Algorithm 1: MMWwith bandit feedback

1: Input: Y 1 ← 0 ; safety parameter 𝑟 𝑖 and anchor point R 𝑖 , 𝑖 ∈ N ; step-size 𝛾 𝑡 ; sampling radius 𝛿 𝑡

2: for 𝑡 = 1 , 2 , . . . do simultaneously for all 𝑖 ∈ N

3: Set X 𝑖,𝑡 = exp ( Y 𝑖,𝑡 )/ tr [ exp ( Y 𝑖,𝑡 )] .

4: Sample Z 𝑖,𝑡 uniformly from E ± 𝑖 .

5: Play X ( 𝛿 ) 𝑖,𝑡 + 𝛿 𝑡 Z 𝑖,𝑡 .

6: Observe 𝑈 𝑖 ( 𝜔 𝑡 ) .

7: Set ˆ V 𝑖,𝑡 : = 𝐷 𝑖 / 𝛿 𝑡 · 𝑈 𝑖 ( 𝜔 𝑡 ) Z 𝑖,𝑡 .

8: Update Y 𝑖,𝑡 + 1 ← Y 𝑖,𝑡 + 𝛾 𝑡 ˆ V 𝑖,𝑡 .

9: end for

Proposition 3. The estimator (2PE) enjoys the conditional bounds

The defining element in Proposition 3 is that even though the estimator (2PE) is biased, its second moment is bounded as O ( 1 ) . This is ultimately due to the multilinearity of the players' payoff functions and plays a pivotal role in showing that the duality gap of ¯ X 𝑡 under (3MW) is of the same order as under (MMW), because the bias can be controlled with affecting the variance of the estimator. We provide a detailed proof of Lemma 1, Proposition 3, and Theorem 2 in the appendix.

5.2. Learning with bandit feedback. Despite its strong convergence guarantees, a major limiting factor in the applicability of Theorem 2 is that, in many cases, the game's players may only be able to observe their realized payoff observables 𝑈 𝑖 ( 𝜔 ) , and their mixed payoffs 𝑢 𝑖 ( X ) could be completely inaccessible. In particular, as we described in Section 2, each outcome 𝜔 ∈ Ω of the POVM occurs with probability 𝑃 𝜔 ( X 𝑡 ) under the strategy profile X 𝑡 . Accordingly, if this is the only information available to the players, they will need to estimate their individual payoff gradients through the single observation of the (random) scalar 𝑈 𝑖 ( 𝜔 𝑡 ) ∈ ℝ . In view of this, and inspired by previous works on payoff-based learning and zeroth-order optimization [8, 9, 11, 25, 49, 50], we will consider the single-point stochastic approximation approach of [21, 55] which unfolds as follows:

Step 1. Each player draws a sampling direction Z 𝑖,𝑡 ∈ E ± 𝑖 uniformly at random.

Step 2. Each player plays X ( 𝛿 ) 𝑖,𝑡 + 𝛿 𝑡 Z 𝑖,𝑡 .

Step 3. Each player receives 𝑈 𝑖 ( 𝜔 𝑡 ) .

Step 4. Each player approximates V 𝑖 ( X 𝑡 ) via the the one-point estimator (1PE):

In this case, the players' gradient estimates may be bounded as follows:

Proposition 4. The estimator (1PE) enjoys the conditional bounds

The crucial difference between Propositions 3 and 4 is that the former leads to a gradient estimator with O ( 1 ) variance and magnitude, whereas the magnitude of the latter is inversely proportional to 𝛿 𝑡 ; however, since 𝛿 𝑡 in turn controls the bias of the gradient estimator, we must now resolve a bias-variance dilemma, which was absent in the case of (2PE). This leads to the following variant of Theorem 2 with bandit, realization-based feedback:

Theorem 3. Suppose that each player of a 2 -player zero-sum game Q follows (3MW) for 𝑇 epochs with learning rate 𝛾 , sampling radius 𝛿 , and gradient estimates provided by (1PE) . Then the players' empirical frequency of play enjoys the duality gap guarantee

where 𝐻 = log ( 𝑑 1 𝑑 2 ) . In particular, for 𝛾 = GLYPH<0> 𝐻 2 𝑇 GLYPH<1> 3 / 4 1 2 𝐷 √ 𝐵𝐿 and 𝛿 = GLYPH<0> 𝐻 2 𝑇 GLYPH<1> 1 / 4 √︃ 𝐵 4 𝐿 , the players enjoy the equilibrium convergence guarantee:

An important observation here is that the players' equilibrium convergence rate under (3MW) + (1PE) no longer matches the convergence rate of the vanilla MMW algorithm (Theorem 1). The reason for this is the bias-variance trade-off in the estimator (1PE), and is reminiscent of the drop in the rate of regret minimization from O ( 𝑇 1 / 2 ) to O ( 𝑇 2 / 3 ) under (IWE) with bandit feedback and explicit exploration in finite games. A kernel-based approach in the spirit of Bubeck et al. [13] could possibly be used to fill the O ( 𝑇 1 / 4 ) gap between Theorems 1 and 3, but this would come at the cost of a possibly catastrophic dependence on the dimension (which is already quadratic in our setting). This consideration is beyond the scope of our work, but it would constitute an important future direction.

6 Bandit learning in 𝑁 -player quantum games

We conclude our paper with an examination of the behavior of the MMW algorithm in general, 𝑁 -player quantum games. Here, a major difficulty that arises is that, in stark contrast to the min-max case, the set of the game's equilibria can be disconnected, so any convergence result will have to be, by necessity, local. In addition, because general 𝑁 -games do not have the amenable profile of a bilinear min-max problem - they are multilinear, multi-objective problems - it will not be possible to obtain any convergence guarantees for the game's empirical frequency of play (since there is no convex structure to exploit). Instead, we will have to focus squarely on the induced trajectory of play, which carries with it a fair share of complications.

Inspired by the very recent work of [38], we will not constrain our focus to a specific class of games , but to a specific class of equilibria . In particular, we will consider the behavior of MMW-based learning with respect to Nash equilibria X ∗ ∈ X that satisfy the variational stability condition

This condition can be traced back to [44], and can be seen as a game-theoretic analogue of first-order stationarity in the context of continuous optimization, or as an equilibrium refinement in the spirit of the seminal concept of evolutionary stability in population games [39, 40]. 2 Importantly, as was shown in [38], variationally stable equilibria are the only equilibria that are asymptotically stable under the continuous-time dynamics of the 'follow the regularized leader' (FTRL) class of learning policies, so it stands to reason to ask whether they enjoy a similar convergence landscape in the context of bona fide, discrete-time learning with minimal, payoff-based feedback.

Our final result provides an unambiguously positive answer to this question: 3

Theorem 4. Fix some tolerance level 𝜂 ∈ ( 0 , 1 ) and suppose that the players of an 𝑁 -player quantum game follow (3MW) with bandit, realization-based feedback, and surrogate gradients provided by the estimator (1PE) with step-size and sampling radius parameters such that

If X ∗ is variationally stable, there exists a neighborhoold U of X ∗ such that

It is worth noting that the last-iterate convergence guarantee of Theorem 4 is considerably stronger than the time-averaged variants of Theorems 1-3, and we are not aware of any comparable convergence guarantee for general quantum games. [Trivially, last-iterate convergence implies time-averaged convergence, but the converse, of course, may fail to hold] As such, especially in cases that require to track the trajectory of the system or the players' day-to-day rewards, Theorem 4 provides an important guarantee for the realized sequence of events.

On the other hand, in contrast to Theorem 4, it should be noted that the guarantees of Theorems 1-3 are global. Given that general quantum games may in general possess a large number of disjoint Nash equilibria, this transition from global to local convergence guarantees seems unavoidable. It is, however, an open question whether (VS) could be exploited further in order to deduce the rate of convergence to such equilibria; we leave this as a direction for future research.

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 -4

10 -3

10 -2

10 -1

10 0

10 1

MMW

2PE

1PE

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 0

10 1

10 2

10 3

10 4

MMW

2PE

1PE

7 Numerical Experiments

In this last section, we provide numerical simulations to validate and explore the performance of (MMW) with payoff-based feedback. Additional experiments can be found in Appendix E.

Game setup. Our testbed is a two-player zero-sum quantum game, which is the quantum analogue of a 2 × 2 min-max game with actions { 𝛼 1 , 𝛼 2 } and { 𝛽 1 , 𝛽 2 } , and payoff matrix

In the quantum regime, the payoff information of the quantum game is encoded in the Hermitian matrices W 1 = diag ( 4 , 2 , -4 , -2 ) , and W 2 = -W 1 as per Eq. (3) in Section 2. By elementary considerations, the action profile ( 𝛼 1 , 𝛽 2 ) is a strict Nash equilibrium of the classical zero-sum game, which corresponds to the pure quantum state with density matrix profile X ∗ = ( X ∗ 1 , X ∗ 2 ) where X ∗ 1 = 𝑒 1 ⊗ 𝑒 1 and X ∗ 2 = 𝑒 2 ⊗ 𝑒 2 in the standard basis in which W 1 and W 2 are diagonal.

Convergence speed analysis. In Fig. 1, we evaluate the convergence properties of (3MW) using the estimators (2PE) and (1PE), and compare it with the full information variant (MMW). For each method, we perform 10 different runs, with 𝑇 = 10 5 steps each, and compute the mean value of the duality gap as a function of the iteration 𝑡 = 1 , 2 , . . . , 𝑇 . The solid lines correspond to the mean values of the duality gap of each method, and the shaded regions enclose the area of ± 1 (sample) standard deviation among the 10 different runs. Note that the red line, which corresponds to the full information (MMW), does not have a shaded region, since there is no randomness in the algorithm. All the runs for the three different methods were initialized for Y = 0 and we used 𝛾 = 10 -2 for all methods. In particular, for (3MW) with gradient estimates given by (2PE) estimator, we used a sampling radius 𝛿 = 10 -2 , and for (3MW) with (1PE) estimator, we used 𝛿 = 10 -1 (in tune with our theoretical results which suggest the use of a tighter sampling radius when mixed payoff information is available to the players).

Figure 1 has several important take-aways. First and foremost, as is to be expected, the payoff-based methods lag behind the full-information variant of (MMW); however, what is particularly surprising is that the drop in performance is singularly mild. As we see in the second plot in Fig. 1, the various algorithms achieved a rate of convergence closer to O ( 1 / 𝑇 ) , which is significantly faster than O ( 1 / √ 𝑇 ) and/or O ( 1 / 𝑇 1 / 4 ) . This suggests that, in practice, the bandit variants of (MMW) may yield excellent performance benefits, despite the high degree of uncertainty incurred by the complete lack of information on the game being played.

Acknowledgments and Disclosure of Funding

This work has been partially supported by the Air Force Office of Scientific Research under award number FA9550-20-1-0397. Additional support is gratefully acknowledged from NSF 1915967, 2118199, 2229012, 2312204. KL is grateful for financial support by the Onassis Foundation (F ZR 033-1/2021-2022). PM is also with the Archimedes Research Unit - Athena RC - University of Athens, and gratefully acknowledges financial support by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program, and by the French National Research Agency (ANR) in the framework of the 'Investissements d'avenir' program (ANR-15-IDEX-02), the LabEx PERSYVAL (ANR-11-LABX-0025-01), MIAI@Grenoble Alpes (ANR-19-P3IA-0003). NB was supported by the Koret Foundation via the Digital Living 2030 project.

References

Aaronson, S. Shadow tomography of quantum states. SIAM Journal on Computing , 49(5), January 2020.

Aaronson, S., Chen, X., Hazan, E., Kale, S., and Nayak, A. Online learning of quantum states. In NeurIPS '18: Proceedings of the 32nd International Conference of Neural Information Processing Systems , 2018.

Agarwal, A., Dekel, O., and Xiao, L. Optimal algorithms for online convex optimization with multi-point bandit feedback. In COLT '10: Proceedings of the 23rd Annual Conference on Learning Theory , 2010.

Arute, F., Arya, K., Babbush, R., Bacon, D., Bardin, J. C., Barends, R., Biswas, R., Boixo, S., Brandao, F. G., Buell, D. A., Burkett, B., Chen, Y., Chen, Z., Chiaro, B., Collins, R., Courtney, W., Dunsworth, A., Farhi, E., Foxen, B., Fowler, A., Gidney, C., Giustina, M., Graff, R., Guerin, K., Habegger, S., Harrigan, M. P., Hartmann, M. J., Ho, A., Hoffmann, M. R., Huang, T., Humble, T. S., Isakov, S. V., Jeffrey, E., Jiang, Z., Kafri, D., Kechedzhi, K., Kelly, J., Klimov, P. V ., Knysh, S., Korotkov, A. N., Kostritsa, F., Landhuis, D., Lindmark, M., Lucero, E., Lyakh, D., Mandra, S., McClean, J. R., McEwen, M., Megrant, A., Mi, X., Michielsen, K., Mohseni, M., Mutus, J., Naaman, O., Neeley, M., Neill, C., Niu, M. Y., Ostby, E., Petukhov, A., Platt, J. C., Quintana, C., Rieffel, E. G., Roushan, P., Rubin, N. C., Sank, D., Satzinger, K. J., Smelyanskiy, V., Sung, K. J., Trevithick, M. D., Vainsencher, A., Villalonga, B., White, T., Yao, Z. J., Yeh, P., Zalcman, A., Neven, H., and Martinis, J. M. Quantum supremacy using a programmable superconducting processor. Nature , 2019.

Audibert, J.-Y. and Bubeck, S. Regret bounds and minimax policies under partial monitoring. Journal of Machine Learning Research , 11:2635-2686, 2010.

Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Proceedings of the 36th Annual Symposium on Foundations of Computer Science , 1995.

Beck, A. and Teboulle, M. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31(3):167-175, 2003.

Bervoets, S., Bravo, M., and Faure, M. Learning with minimal information in continuous games. Theoretical Economics , 15:1471-1508, 2020.

Bilenne, O., Mertikopoulos, P., and Belmega, E. V. Fast optimization with zeroth-order feedback in distributed multi-user MIMO systems. IEEE Trans. Signal Process. , 68:6085-6100, October 2020.

Brandao, F. G. S. L. and Svore, K. Quantum speed-ups for semidefinite programming, 2017.

Bravo, M., Leslie, D. S., and Mertikopoulos, P. Bandit learning in concave 𝑁 -person games. In NeurIPS '18: Proceedings of the 32nd International Conference of Neural Information Processing Systems , 2018.

Bubeck, S. and Cesa-Bianchi, N. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning , 5(1):1-122, 2012.

Bubeck, S., Lee, Y. T., and Eldan, R. Kernel-based methods for bandit convex optimization. In STOC '17: Proceedings of the 49th annual ACM SIGACT symposium on the Theory of Computing , 2017.

Cesa-Bianchi, N. and Lugosi, G. Prediction, Learning, and Games . Cambridge University Press, 2006.

Chakrabarti, S., Huang, Y., Li, T., Feizi, S., and Wu, X. Quantum Wasserstein generative adversarial networks. In NeurIPS '19: Proceedings of the 33rd International Conference on Neural Information Processing Systems , 2019.

Chen, X., Hazan, E., Li, T., Lu, Z., Wang, X., and Yang, R. Adaptive online learning of quantum states, 2022.

Chuang, I. and Nielsen, M. Quantum Computation and Quantum Information . Cambridge University Press, 2 edition, 2010.

Dallaire-Demers, P.-L. and Killoran, N. Quantum generative adversarial networks. Physical Review A , 98 (1):012324, 2018.

Debreu, G. A social equilibrium existence theorem. Proceedings of the National Academy of Sciences of the USA , October 1952.

Eisert, J., Wilkens, M., and Lewenstein, M. Quantum games and quantum strategies. Physical Review Letters , 83, October 1999.

Flaxman, A. D., Kalai, A. T., and McMahan, H. B. Online convex optimization in the bandit setting: gradient descent without a gradient. In SODA '05: Proceedings of the 16th annual ACM-SIAM Symposium on Discrete Algorithms , pp. 385-394, 2005.

Gao, M., Ji, Z., Li, T., and Wang, Q. Logarithmic-regret quantum learning algorithms for zero-sum games, 2023.

Gutoski, G. and Watrous, J. Toward a general theory of quantum games. In STOC '07: Proceedings of the 39th annual ACM symposium on the Theory of Computing , 2007.

Hall, P. and Heyde, C. C. Martingale Limit Theory and Its Application . Probability and Mathematical Statistics. Academic Press, New York, 1980.

Héliou, A., Mertikopoulos, P., and Zhou, Z. Gradient-free online learning in continuous games with delayed rewards. In ICML '20: Proceedings of the 37th International Conference on Machine Learning , 2020.

Ickstadt, C., Theobald, T., and Tsigaridas, E. Semidefinite games. https://arxiv.org/pdf/2202. 12035 , 2022.

Jain, R. and Watrous, J. Parallel approximation of non-interactive zero-sum quantum games. In CCC '09: Proceedings of the 2009 IEEE International Conference on Computational Complexity , 2009.

Jain, R., Piliouras, G., and Sim, R. Matrix multiplicative weights updates in quantum games: Conservation law & recurrence. In NeurIPS '22: Proceedings of the 36th International Conference on Neural Information Processing Systems , 2022.

Kakade, S. M., Shalev-Shwartz, S., and Tewari, A. Regularization techniques for learning with matrices. The Journal of Machine Learning Research , 13:1865-1890, 2012.

Kiefer, J. and Wolfowitz, J. Stochastic estimation of the maximum of a regression function. The Annals of Mathematical Statistics , 23(3):462-466, 1952.

Kleinberg, R. D. Nearly tight bounds for the continuum-armed bandit problem. In NIPS' 04: Proceedings of the 18th Annual Conference on Neural Information Processing Systems , 2004.

Lattimore, T. and Szepesvári, C. Bandit Algorithms . Cambridge University Press, Cambridge, UK, 2020.

Li, T., Chakrabarti, S., and Wu, X. Sublinear quantum algorithms for training linear and kernel-based classifiers. In International Conference on Machine Learning , 2019.

Li, T., Wang, C., Chakrabarti, S., and Wu, X. Sublinear classical and quantum algorithms for general matrix games. Proceedings of the AAAI Conference on Artificial Intelligence , 35(10), May 2021. doi: 10.1609/aaai.v35i10.17028.

Lin, W., Piliouras, G., Sim, R., and Varvitsiotis, A. Quantum potential games, replicator dynamics, and the separability problem, 2023.

Littlestone, N. and Warmuth, M. K. The weighted majority algorithm. Information and Computation , 108 (2):212-261, 1994.

Lloyd, S. and Weedbrook, C. Quantum generative adversarial learning. Physical Review Letters , 121(4), July 2018.

Lotidis, K., Mertikopoulos, P., and Bambos, N. Learning in quantum games, 2023.

Maynard Smith, J. Evolution and the Theory of Games . Cambridge University Press, Cambridge, 1982.

Maynard Smith, J. and Price, G. R. The logic of animal conflict. Nature , 246:15-18, November 1973.

Mertikopoulos, P. Strange bedfellows: Riemann, Gibbs and vector Gaussian multiple access channels. In NetGCoop '12: Proceedings of the 2012 International Conference on Network Games, Control and Optimization , 2012.

Mertikopoulos, P. and Moustakas, A. L. Entropy-driven optimization dynamics for Gaussian vector multiple access channels. In ICC '13: Proceedings of the 2013 IEEE International Conference on Communications , 2013.

Mertikopoulos, P. and Moustakas, A. L. Learning in an uncertain world: MIMO covariance matrix optimization with imperfect feedback. IEEE Trans. Signal Process. , 64(1), January 2016.

Mertikopoulos, P. and Zhou, Z. Learning in games with continuous action sets and unknown payoff functions. Mathematical Programming , 173(1-2):465-507, January 2019.

Mertikopoulos, P., Belmega, E. V., Negrel, R., and Sanguinetti, L. Distributed stochastic optimization via matrix exponential learning. IEEE Trans. Signal Process. , 65(9):2277-2290, May 2017.

Mertikopoulos, P., Hsieh, Y.-P., and Cevher, V. A unified stochastic approximation framework for learning in games. Mathematical Programming , forthcoming, 2023.

Nemirovski, A. S. and Yudin, D. B. Problem Complexity and Method Efficiency in Optimization . Wiley, New York, NY, 1983.

Nielsen, M. A. and Chuang, I. L. Quantum Computation and Quantum Information: 10th Anniversary Edition . Cambridge University Press, 2010. doi: 10.1017/CBO9780511976667.

Perkins, S. and Leslie, D. S. Stochastic fictitious play with continuous action sets. Journal of Economic Theory , 152:179-213, July 2014.

Perkins, S., Mertikopoulos, P., and Leslie, D. S. Mixed-strategy learning with continuous action sets. IEEE Trans. Autom. Control , 62(1):379-384, January 2017.

Preskill, J. Quantum computing in the NISQ era and beyond. Quantum , 2:79, August 2018.

Robbins, H. and Siegmund, D. A convergence theorem for nonnegative almost supermartingales and some applications. In Rustagi, J. S. (ed.), Optimizing Methods in Statistics . Academic Press, New York, NY, 1971.

Shamir, O. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research , 18(52):1-11, 2017.

Sion, M. On general minimax theorems. Pacific Journal of Mathematics , 8(1):171 - 176, 1958.

Spall, J. C. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Trans. Autom. Control , 37(3):332-341, March 1992.

Taylor, P. D. Evolutionarily stable strategies with two types of player. Journal of Applied Probability , 16 (1):76-83, March 1979.

Telatar, I. E. Capacity of multi-antenna Gaussian channels. European Transactions on Telecommunications and Related Technologies , 10(6):585-596, 1999.

Tse, D. and Viswanath, P. Fundamentals of Wireless Communication . Cambridge University Press, Cambridge, UK, 2005.

Tsuda, K., Rätsch, G., and Warmuth, M. K. Matrix exponentiated gradient updates for on-line Bregman projection. Journal of Machine Learning Research , 6:995-1018, 2005.

van Apeldoorn, J. and Gily'en, A. Quantum algorithms for zero-sum games. arXiv: Quantum Physics , 2019.

Vovk, V. G. Aggregating strategies. In COLT '90: Proceedings of the 3rd Workshop on Computational Learning Theory , pp. 371-383, 1990.

Yu, W., Rhee, W., Boyd, S. P., and Cioffi, J. M. Iterative water-filling for Gaussian vector multiple-access channels. IEEE Trans. Inf. Theory , 50(1):145-152, 2004.

Yu, Y.-L. The strong convexity of von Neumann's entropy. 2015.

Zhong, H.-S., Wang, H., Deng, Y.-H., Chen, M.-C., Peng, L.-C., Luo, Y.-H., Qin, J., Wu, D., Ding, X., Hu, Y., Hu, P., Yang, X.-Y., Zhang, W.-J., Li, H., Li, Y., Jiang, X., Gan, L., Yang, G., You, L., Wang, Z., Li, L., Liu, N.-L., Lu, C.-Y., and Pan, J.-W. Quantum computational advantage using photons. Science , 370 (6523):1460-1463, December 2020.

Zimmert, J. and Seldin, Y. Tsallis-INF: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research , 22(28):1-49, 2021.

Appendix

In the series of technical appendices that follow, we provide the missing proofs from the main part of our paper, and we provide some numerical illustrations of the performance of the proposed algorithms. As a roadmap, we begin in Appendix A with some auxiliary results that are required throughout our analysis. Subsequently, in Appendices B-D, we provide the proofs of the results presented in Sections 4-6 respectively. Finally, in Appendix E, we provide a suite of numerical experiments to assess the practical performance of (3MW) using the estimators (2PE) and (1PE), and we compare it with the full information setting underlying (MMW).

A Auxiliary Results

We now introduce some notation for quantum games in a 𝑁 -player setting, and explain how the extension from the 2-player setting is straightforward.

𝑁 -player quantum games. First of all, a quantum game Q consists of a finite set of players 𝑖 ∈ N = { 1 , . . . , 𝑁 } , where each player 𝑖 ∈ N has access to a complex Hilbert space H 𝑖 GLYPH<27> ℂ 𝑑 𝑖 . The set of pure states is the unit sphere Ψ 𝑖 : = { 𝜓 𝑖 ∈ H 𝑖 : ∥ 𝜓 𝑖 ∥ 𝐹 = 1 } of H 𝑖 . We will write Ψ : = Q 𝑖 Ψ 𝑖 for the space of all ensembles 𝜓 = ( 𝜓 1 , . . . , 𝜓 𝑁 ) of pure states 𝜓 𝑖 ∈ Ψ 𝑖 that are independently prepared by each 𝑖 ∈ N .

In analogy with the 2-player case, each outcome 𝜔 ∈ Ω is associated to a positive semi-definite operator P 𝜔 : H → H defined on the tensor product H : = N 𝑖 H 𝑖 of the players' individual state spaces; we further assume that P 𝜔 ∈ Ω P 𝜔 = I , thus, the probability of observing 𝜔 ∈ Ω at state 𝜓 ∈ Ψ is

and, the player's expected payoff at state 𝜓 ∈ Ψ will be

Similarly to the 2-player setting, if each player 𝑖 ∈ N prepares a density matrix X 𝑖 as per (1), the expected payoff of player 𝑖 ∈ N under X = ( X 1 , . . . , X 𝑁 ) will be

where W 𝑖 = P 𝜔 ∈ Ω 𝑈 𝑖 ( 𝜔 ) P 𝜔 ∈ H for 𝑖 ∈ N . Finally, we denote by V 𝑖 ( X ) the individual payoff gradient of player 𝑖 under X as

All other notions are extended, accordingly. ♦

As noted in Section 2, we define the norm ∥ 𝐴 ∥ 𝐹 = √︁ tr [ 𝐴 † 𝐴 ] for any 𝐴 ∈ ℍ 𝑑 𝑖 , i.e., ( ℍ 𝑑 𝑖 , ∥·∥ 𝐹 ) is an inner-product space. With a slight abuse of notation, we define for X = ( X 1 , . . . , X 𝑁 ) ∈ X its norm as:

Lemma A.1. For any X 𝑖 ∈ X 𝑖 , it holds ∥ X 𝑖 ∥ 𝐹 ≤ 1 , and diam ( X ) = 2 √ 𝑁 .

Proof. For the first part, since X 𝑖 ∈ X 𝑖 , it admits an orthonormal decomposition 𝑄 Λ 𝑄 † such that 𝑄𝑄 † = 𝑄 † 𝑄 = I and Λ = diag ( 𝜆 1 , . . . , 𝜆 𝑑 𝑖 ) with P 𝑑 𝑖 𝑗 = 1 𝜆 𝑗 = 1 , and 𝜆 𝑗 ≥ 0 for all 𝑗 . Hence

where the last inequality holds, since 0 ≤ 𝜆 𝑗 ≤ 1 , and the result follows.

For the second part, letting X = ( X 1 , . . . , X 𝑁 ) and X ′ = ( X ′ 1 , . . . , X ′ 𝑁 ) be two points in X , we have

and since the equality is attained, we get the result.

Our next result concerns the quantum relative entropy

The lemma we will require is a semidefinite version of Pinsker's inequality which reads as follows: Lemma A.2. For all P ∈ X and X ∈ ri ( X ) we have

Proof. Focusing on player 𝑖 ∈ N , we will show first that

To this end, we define the function ℎ 𝑖 : ℍ 𝑑 𝑖 + → ℝ as ℎ 𝑖 ( X 𝑖 ) = tr [ X 𝑖 log X 𝑖 ] , which is 1-strongly convex with respect to the nuclear norm ∥·∥ 1 [63], and since ∥ X 𝑖 ∥ 1 ≥ ∥ X 𝑖 ∥ 𝐹 for all X 𝑖 ∈ X 𝑖 , we readily get that ℎ 𝑖 is 1-strongly convex with respect to the Frobenius norm, as well.

Letting ∇ ℎ 𝑖 ( X 𝑖 ) = log X 𝑖 + I , by 1-strong convexity, we have for P = ( P 1 , . . . , P 𝑁 ) ∈ X and X = ( X 1 , . . . , X 𝑁 ) ∈ ri ( X ) :

where we used that tr [ P 𝑖 -X 𝑖 ] = 0 . Hence, by reordering, we automatically get that

and the proof is completed.

B Omitted proofs from Section 4

In this appendix, we develop the basic scaffolding required for the estimators (2PE) and (1PE). We begin with the construction of the estimators' sampling basis, as encoded in Proposition 1, which we restate below for convenience:

Then, the set E = n { E 𝑗 } 𝑑 -1 𝑗 = 1 , { e 𝑘ℓ } 𝑘<ℓ , { ˜ e 𝑘ℓ } 𝑘<ℓ o is an orthonormal basis of Z .

Proof. First of all, note that

■

■

Therefore, we have:

Unit norm. To begin with, we will show that all elements in E have unit norm. Indeed, we have:

For 𝑗 = 1 , . . . , 𝑑 -1 , we have:

For 𝑘 < ℓ , we have:

For 𝑘 < ℓ , we also have:

Orthogonality. Now, we will show that any two elements of E are orthogonal to each other.

For 𝑚 < 𝑛 , we have:

For 𝑘 < ℓ , we have:

For ( 𝑘, ℓ ) ≠ ( 𝑚, 𝑛 ) with 𝑘 < ℓ and 𝑚 < 𝑛 , we have:

since all the nonzero terms in e † 𝑘ℓ e 𝑚𝑛 , e † 𝑘ℓ ˜ e 𝑚𝑛 and ˜ e † 𝑘ℓ ˜ e 𝑚𝑛 are of the form 𝑐 · 𝚫 𝛼𝛽 for some 𝑐 ∈ ℂ , and 𝛼, 𝛽 ∈ { 𝑘, ℓ, 𝑚, 𝑛 } with 𝛼 ≠ 𝛽 . Thus, tr GLYPH<2> 𝑐 · 𝚫 𝛼𝛽 GLYPH<3> = 0 , since all the diagonal elements are equal to 0 . Note that it is not possible to have 𝛼 = 𝛽 because this would imply that ( 𝑘, ℓ ) = ( 𝑚, 𝑛 ) .

For 𝑘 < ℓ and 𝑗 = 1 , . . . , 𝑑 -1 , we have:

since the non-zero terms of both e † 𝑘ℓ E 𝑗 and ˜ e † 𝑘ℓ E 𝑗 are of the form 𝚫 𝑘𝑛 , 𝚫 ℓ𝑚 for 𝑘 ≠ 𝑛 and ℓ ≠ 𝑚 .

We thus conclude that any two elements of E are orthogonal.

Finally, it is clear E ⊆ aff ( X 0 ) , since E ⊆ ℍ 𝑑 and tr [ e 𝑘ℓ ] = tr [ ˜ e 𝑘ℓ ] = tr GLYPH<2> E 𝑗 GLYPH<3> = 0 , for 𝑘 < ℓ and 𝑗 = 1 , . . . , 𝑑 -1 . Therefore, the elements in E form an orthonormal basis of aff ( X 0 ) and dim ( aff ( X 0 )) = 𝑑 2 -1 . ■

We now proceed with the construction of the precise 'safety net' that guarantees that the sampling perturbation of the gradient estimator remains within the problem's feasible region. Again, for convenience, we restate the relevant result below:

Proposition 2. Let R = 1 𝑑 P 𝑑 𝑗 = 1 𝚫 𝑗 𝑗 . Then, for 𝑟 = min GLYPH<26> 1 √ 𝑑 ( 𝑑 -1 ) , √ 2 𝑑 GLYPH<27> , it holds that R + 𝑟 Z ∈ X for any direction Z ∈ E ± .

Proof. To begin with, it is clear that R ∈ ℍ 𝑑 and tr [ R ] = P 𝑑 𝑗 = 1 1 / 𝑑 = 1 . Moreover, for any 𝑢 ∈ ℂ 𝑑 \ { 0 } , we have:

where | 𝑢 𝑗 | is the modulus of the complex number 𝑢 𝑗 ∈ ℂ . Therefore, R is positive definite, i.e., lies in ri ( X ) .

Now, we need to find 𝑟 > 0 such that for any Z ∈ E ± .

It is clear that for any Z ∈ E ± , we have tr [ R + 𝑟 Z ] = tr [ R ] = 1 , since tr [ Z ] = 0 . Hence, it remains to consider the positive semi-definite constraint. For this, we will use the following identities, for 𝑘 < ℓ :

and

For Z = 1 √ 2 ( 𝚫 𝑘ℓ + 𝚫 ℓ𝑘 ) and 𝑢 ∈ ℂ 𝑑 \ { 0 } , and using (B.11), we have:

If Re ( ¯ 𝑢 𝑘 𝑢 ℓ ) > 0 , we get that 𝑢 † ( R + 𝑟 Z ) 𝑢 > 0 , while if Re ( ¯ 𝑢 𝑘 𝑢 ℓ ) ≤ 0 and 𝑟 ≤ √ 2 / 𝑑 :

Hence, for 𝑟 ≤ √ 2 / 𝑑 , and Z = 1 √ 2 ( 𝚫 𝑘ℓ + 𝚫 ℓ𝑘 ) , we have that 𝑢 † ( R + 𝑟 Z ) 𝑢 ≥ 0 for all 𝑢 ∈ ℂ 𝑑 .

For Z = -1 √ 2 ( 𝚫 𝑘ℓ + 𝚫 ℓ𝑘 ) , we have

If Re ( ¯ 𝑢 𝑘 𝑢 ℓ ) < 0 , we get that 𝑢 † ( R + 𝑟 Z ) 𝑢 > 0 , while if Re ( ¯ 𝑢 𝑘 𝑢 ℓ ) ≥ 0 and 𝑟 ≤ √ 2 / 𝑑 :

Hence, for 𝑟 ≤ √ 2 / 𝑑 , and Z = -1 √ 2 ( 𝚫 𝑘ℓ + 𝚫 ℓ𝑘 ) , we have that 𝑢 † ( R + 𝑟 Z ) 𝑢 ≥ 0 for all 𝑢 ∈ ℂ 𝑑 .

For Z = 𝑖 √ 2 ( 𝚫 𝑘ℓ -𝚫 ℓ𝑘 ) and 𝑢 ∈ ℂ 𝑑 \ { 0 } , and using (B.11), we have:

If Im ( ¯ 𝑢 𝑘 𝑢 ℓ ) < 0 , we get that 𝑢 † ( R + 𝑟 Z ) 𝑢 > 0 , while if Im ( ¯ 𝑢 𝑘 𝑢 ℓ ) ≥ 0 and 𝑟 ≤ √ 2 / 𝑑 :

Hence, for 𝑟 ≤ √ 2 / 2 𝑑 , and Z = 𝑖 √ 2 ( 𝚫 𝑘ℓ -𝚫 ℓ𝑘 ) , we have that 𝑢 † ( R + 𝑟 Z ) 𝑢 ≥ 0 for all 𝑢 ∈ ℂ 𝑑 .

For Z = -𝑖 √ 2 ( 𝚫 𝑘ℓ -𝚫 ℓ𝑘 ) and 𝑢 ∈ ℂ 𝑑 \ { 0 } , and using (B.11), we have:

If Im ( ¯ 𝑢 𝑘 𝑢 ℓ ) > 0 , we get that 𝑢 † ( R + 𝑟 Z ) 𝑢 > 0 , while if Im ( ¯ 𝑢 𝑘 𝑢 ℓ ) ≤ 0 and 𝑟 ≤ √ 2 / 𝑑 :

Hence, for 𝑟 ≤ √ 2 / 2 𝑑 , and Z = -𝑖 √ 2 ( 𝚫 𝑘ℓ -𝚫 ℓ𝑘 ) , we have that 𝑢 † ( R + 𝑟 Z ) 𝑢 ≥ 0 for all 𝑢 ∈ ℂ 𝑑 .

For Z = 1 √ 𝑗 ( 𝑗 + 1 ) GLYPH<0> 𝚫 11 + · · · + 𝚫 𝑗 𝑗 -𝑗 𝚫 ( 𝑗 + 1 ) ( 𝑗 + 1 ) GLYPH<1> , we have:

Thus, we need to ensure that

for all 𝑗 = 1 , . . . , 𝑑 -1 . Because the function 𝑥 ↦→ √︁ 𝑥 ( 𝑥 + 1 )/ 𝑥 is decreasing, it obtains the smallest value from 𝑥 = 𝑑 -1 . Therefore, for 𝑟 ≤ 1 / √︁ 𝑑 ( 𝑑 -1 ) , we readily obtain that 𝑢 † ( R + 𝑟 Z ) 𝑢 ≥ 0 for all 𝑢 ∈ ℂ 𝑑 .

For Z = -1 √ 𝑗 ( 𝑗 + 1 ) GLYPH<0> 𝚫 11 + · · · + 𝚫 𝑗 𝑗 -𝑗 𝚫 ( 𝑗 + 1 ) ( 𝑗 + 1 ) GLYPH<1> , we have:

Thus, we need to ensure that

for all 𝑗 = 1 , . . . , 𝑑 -1 . Because it holds that

we obtain the inequality for free by the previous case, i.e., for 𝑟 ≤ 1 / √︁ 𝑑 ( 𝑑 -1 ) .

Therefore, for

we readily obtain that 𝑢 † ( R + 𝑟 Z ) 𝑢 ≥ 0 for all 𝑢 ∈ ℂ 𝑑 , and our proof is complete.

C Omitted proofs from Section 5

Our aim in this appendix will be to prove the basic guarantees of (3MW) with payoff-based feedback. The structure of this appendix shadows that of Section 5 and is broken into two parts, depending on the specific type of input available to the players. The only point of departure is the energy inequality of Lemma 1, which is common to both algorithms, and which we restate and prove below:

Lemma 1. Fix some P ∈ X , and let X 𝑡 , X 𝑡 + 1 be two successive iterates of (3MW) , without any assumptions for the input sequence ˆ V 𝑡 . We then have

Proof. By the definition of 𝐷 , it is easy to see that for P ∈ X and X , X ′ ∈ ri ( X ) , we have

Since ∇ ℎ ( X ) = log X + I , the above equality can be written as:

Setting X as X 𝑡 + 1 , and X ′ as X 𝑡 , and invoking the easily verifiable fact that ∇ ℎ ( X 𝑡 + 1 )-∇ ℎ ( X 𝑡 ) = 𝛾 𝑡 ˆ V 𝑡 , we get

and hence:

where the first inequality holds due to Lemma A.2, and in the last step we used that ∥·∥ 𝐹 is an inner-product norm on X , so

This concludes our proof.

■

With this template inequality in hand, we proceed with the guarantees of (3MW) in the next sections.

■

C.1. Learning with mixed payoff observations. We begin with the statistics of the 2 -point sampler (2PE), which we restate below:

Proposition 3. The estimator (2PE) enjoys the conditional bounds

Proof. We prove each part separately.

and

Taking a first-order Taylor expansion of 𝑢 𝑖 , we obtain:

and

where 𝑅 2 (·) is the 2nd order Taylor remainder. Now, for 𝑗 ≠ 𝑖 ∈ N , since 𝑠 𝑖,𝑡 is zero-mean and independent of any other process:

and using that Ξ (+) 𝑖,𝑡 -Ξ (-) 𝑖,𝑡 = 2 𝑠 𝑖,𝑡 𝛿 𝑡 Z 𝑖,𝑡 , we have:

where in the last step, with a slight abuse of notation, we identify proj E 𝑖 ( V 𝑖 ( X 𝑡 )) with V 𝑖 ( X 𝑡 ) . The reason for this is that we apply the differential operator V 𝑖 ( X 𝑡 ) only on elements of X 𝑖 , and thus, we can ignore the component of V 𝑖 ( X 𝑡 ) that is orthogonal to span ( E 𝑖 ) .

Moreover, we have that and, hence

and similarly, we get the same bound for | 𝑅 2 ( Ξ (-) 𝑡 )| . Therefore, in light of the above, we obtain the bound:

(ii) By the definition of ˆ V 𝑖,𝑡 , we have:

and therefore, we readily obtain that:

■

so and our proof is complete.

With all these technical elements in place, we are finally in a position to prove our convergence result for (3MW) run with 2 -point gradient estimators. As before, we restate our result below for convenience:

Theorem 2. Suppose that each player of a 2 -player zero-sum game Q follows (3MW) for 𝑇 epochs with learning rate 𝛾 , sampling radius 𝛿 , and gradient estimates provided by (2PE) . Then the players' empirical frequency of play enjoys the duality gap guarantee

where 𝐻 = log ( 𝑑 1 𝑑 2 ) . In particular, for 𝛾 = ( 𝐷𝐺 ) -1 √︁ 𝐻 /( 8 𝑇 ) and 𝛿 = ( 𝐺 / 𝐿 ) √︁ 𝐻 /( 8 𝑇 ) , the players enjoy the equilibrium convergence guarantee

Proof. Let X ∗ ∈ X be a NE point. By Lemma 1 for P = X ∗ , and setting 𝐹 𝑡 : = 𝐷 ( X ∗ , X 𝑡 ) for all 𝑡 = 1 , 2 . . . , we have

Summing over the whole sequence 𝑡 = 1 , . . . , 𝑇 , we get:

which can be rewritten by setting 𝛾 0 = ∞ , as:

Decomposing ˆ V 𝑡 as with

or, equivalently

equation (D.18) becomes:

The left-hand side (LHS) of (D.20) gives:

Hence, dividing by 𝑇 , we get:

or, equivalently,

Now, we focus on the right-hand side (RHS) of (D.20). Specifically, we have:

since X 𝑡 is F 𝑡 -measurable and 𝔼 [ 𝑈 𝑡 | F 𝑡 ] = 0 .

Moreover, by Proposition 3, we have:

Hence, taking expectation in (D.20), we obtain:

and

Setting 𝛾 𝑡 = 𝛾 and 𝛿 𝑡 = 𝛿 , we obtain:

for 𝐻 = log ( 𝑑 1 𝑑 2 ) . Hence, after tuning 𝛾 to optimize this last expression, our result follows by setting 𝛾 = √︃ 𝐻 8 𝑇𝐷 2 𝐺 2 and 𝛿 = √︃ 𝐺 2 𝐻 8 𝐿 2 𝑇 . ■

C.2. Learning with bandit feedback. We now proceed with the more arduous task of proving the bona fide, bandit guarantees of (3MW) with 1 -point, stochastic, payoff-based feedback. The key difference with our previous analysis lies in the different statistical properties of the 1 -point estimator (1PE). The relevant result that we will need is restated below:

Proposition 4. The estimator (1PE) enjoys the conditional bounds

Proof. We prove each part separately.

(i) Let Ξ 𝑖,𝑡 be defined for all players 𝑖 ∈ N :

Taking a first-order Taylor expansion of 𝑢 𝑖 , we obtain:

Since 𝔼 [ 𝑈 𝑖 ( 𝜔 𝑡 ) | F 𝑡 , Z 𝑡 ] = 𝑢 ( X ( 𝛿 ) 𝑡 + 𝛿 𝑡 Z 𝑡 ) , combining it with (D.3), we readily get:

Now, because 𝔼 GLYPH<2> Z 𝑖,𝑡 GLYPH<12> GLYPH<12> F 𝑡 GLYPH<3> = 0 and Z 𝑖,𝑡 is sampled independent of any other process, we have:

and finally, noting that we get:

and for 𝑗 ≠ 𝑖 ∈ N :

Therefore, we obtain:

where in the last step, we identify proj E 𝑖 ( V 𝑖 ( X 𝑡 )) with V 𝑖 ( X 𝑡 ) , as explained in the proof of Proposition 3. Moreover, we have that

In view of the above, we have:

(ii) By the definition of ˆ V 𝑖,𝑡 , we have:

and therefore, we readily obtain that:

■

We thus obtain and our proof is complete.

The only step missing is the proof of the actual guarantee of (3MW) with bandit feedback. We restate and prove the relevant result below:

Theorem 3. Suppose that each player of a 2 -player zero-sum game Q follows (3MW) for 𝑇 epochs with learning rate 𝛾 , sampling radius 𝛿 , and gradient estimates provided by (1PE) . Then the players' empirical frequency of play enjoys the duality gap guarantee

where 𝐻 = log ( 𝑑 1 𝑑 2 ) . In particular, for 𝛾 = GLYPH<0> 𝐻 2 𝑇 GLYPH<1> 3 / 4 1 2 𝐷 √ 𝐵𝐿 and 𝛿 = GLYPH<0> 𝐻 2 𝑇 GLYPH<1> 1 / 4 √︃ 𝐵 4 𝐿 , the players enjoy the equilibrium convergence guarantee:

Proof. Following the same procedure as in the proof of Theorem 2, we readily obtain:

Now, we have:

since X 𝑡 is F 𝑡 -measurable and 𝔼 [ 𝑈 𝑡 | F 𝑡 ] = 0 .

and, therefore,

Moreover, by Proposition 4, we have:

Hence, taking expectation in (D.34), we obtain:

Setting 𝛾 𝑡 = 𝛾 and 𝛿 𝑡 = 𝛿 , we obtain:

where 𝐻 = log ( 𝑑 1 𝑑 2 ) . Hence, after tuning 𝛾 and 𝛿 to optimize this last expression, our result follows by setting 𝛾 = GLYPH<0> 𝐻 2 𝑇 GLYPH<1> 3 / 4 1 2 𝐷 √ 𝐵𝐿 and 𝛿 = GLYPH<0> 𝐻 2 𝑇 GLYPH<1> 1 / 4 √︃ 𝐵 4 𝐿 . ■

D Omitted proofs from Section 6

We provide first the bounds of the estimator (1PE) in a 𝑁 -player quantum game. Formally, we have: Lemma D.1. The estimator (1PE) in a 𝑁 -player quantum game Q enjoys the conditional bounds

Proof. We prove each part separately.

(i) Let Ξ 𝑖,𝑡 be defined for all players 𝑖 ∈ N :

Taking a 1st-order Taylor expansion of 𝑢 𝑖 , we obtain:

Since 𝔼 [ 𝑈 𝑖 ( 𝜔 𝑡 ) | F 𝑡 , Z 𝑡 ] = 𝑢 ( X ( 𝛿 ) 𝑡 + 𝛿 𝑡 Z 𝑡 ) , combining it with (D.3), we readily get:

and and finally, noting that

we get:

Now, because 𝔼 GLYPH<2> Z 𝑖,𝑡 GLYPH<12> GLYPH<12> F 𝑡 GLYPH<3> = 0 and Z 𝑖,𝑡 is sampled independent of any other process, we have:

and for 𝑗 ≠ 𝑖 ∈ N :

Therefore, we obtain:

where in the last step, we identify proj E 𝑖 ( V 𝑖 ( X 𝑡 )) with V 𝑖 ( X 𝑡 ) , as explained in the proof of Proposition 3. Moreover, we have that

In view of the above, we have:

(ii) By the definition of ˆ V 𝑖,𝑡 , we have:

and therefore, we readily obtain that:

Hence, ultimately, we get the bound

■

and our proof is complete.

With all this in hand, we are finally in a position to proceed with the proof of Theorem 4, which we restate below for convenience:

Theorem 4. Fix some tolerance level 𝜂 ∈ ( 0 , 1 ) and suppose that the players of an 𝑁 -player quantum game follow (3MW) with bandit, realization-based feedback, and surrogate gradients provided by the estimator (1PE) with step-size and sampling radius parameters such that

If X ∗ is variationally stable, there exists a neighborhoold U of X ∗ such that

and, therefore,

Proof. Since X ∗ is variationally stable, there exists a neighborhood U vs of it such that

we readily obtain by the continuity of X ↦→ 𝐷 ( X ∗ , X ) at X ∗ that there exists a neighborhood U 𝜀 of X ∗ such that U 𝜀 ⊆ U vs. Note that if 𝜀 1 < 𝜀 2 , we automatically get that U 𝜀 1 ⊆ U 𝜀 2 .

In view of this, we let X 1 ∈ U 𝜀 / 4 ⊆ U 𝜀 ⊆ U vs. We divide the rest of the proof in steps.

Step 1. Deriving the general energy inequality

By Lemma 1 we have that:

Decomposing ˆ V 𝑡 into

For any 𝜀 ′ > 0 , defining

as per (C.20) and applying (D.16) inequality iteratively, we get that

Defining the processes Ψ 𝑡 , 𝑀 𝑡 and 𝑍 𝑡 for 𝑡 = 1 , 2 , . . . as

equation (D.18) can be rewritten as

Step 2. Bounding the noise terms

Let 𝜀 > 0 as defined in the beginning of the proof.

Regarding the term 𝑍 𝑡 , it is clear that the process { 𝑍 𝑡 : 𝑡 ≥ 1 } is a sub-martingale. Hence, by Doob's maximal inequality for sub-martingales [24], we get that:

By ensuring that

and taking 𝑡 go to ∞ , (D.21) becomes:

Similarly, it is clear that the process { Ψ 𝑡 : 𝑡 ≥ 1 } is a sub-martingale. Following the same procedure, by Doob's maximal inequality for sub-martingales [24], we get that:

By ensuring that

and taking 𝑡 →∞ , (D.24) becomes:

Finally, regarding the term 𝑀 𝑡 , the process { 𝑀 𝑡 : 𝑡 ≥ 1 } is a martingale. Following the same procedure, by Doob's maximal inequality for martingales [24], we get that:

where we used the fact that

itself following from the total expectation

Now, by ensuring that

and taking 𝑡 go to ∞ , (D.27) becomes:

Therefore, combining (D.23), (D.26) and (D.31) and applying a union bound, we get:

Thus, defining the event 𝐸 : = GLYPH<8> sup 𝑡 ≥ 1 𝑍 𝑡 + Ψ 𝑡 + 𝑀 𝑡 < 3 4 𝜀 GLYPH<9> , Eq. (D.32) readily implies that:

Step 3. X 𝑡 ∈ U vs with high probability

Since X 1 ∈ U 𝜀 / 4 ⊆ U vs, by induction on 𝑡 we have that under the event 𝐸

where in the last step we used the inductive hypothesis that X 𝑠 ∈ U vs for all 𝑠 = 1 , . . . , 𝑡 , which implies tr [ V ( X 𝑠 )( X 𝑠 -X ∗ )] < 0 . This implies that X 𝑡 + 1 ∈ U 𝜀 ⊆ U vs .

Therefore, we obtain that X 𝑡 + 1 ∈ U 𝜀 ⊆ U vs for all 𝑡 ≥ 1 . For the rest of the proof we will work under the event 𝐸 .

Step 4. Subsequential convergence

Now we will show that there exists a subsequence { X 𝑡 𝑘 : 𝑘 ≥ 1 } suct that lim 𝑘 →∞ X 𝑡 𝑘 = X ∗ . Suppose it does not. Then, this would mean that the quantity tr [ V ( X 𝑡 )( X 𝑡 -X ∗ )] is bounded away from zero. Combining it with the fact that X 𝑡 ∈ U vs for all 𝑡 ≥ 0 , we readily get that there exists 𝑐 > 0 such that:

Hence, taking 𝑡 →∞ , and using that P 𝑡 ≥ 1 𝛾 𝑡 = ∞ , we would get that 𝐷 ( X ∗ , X 𝑡 ) →-∞ , which is a contradiction, since 𝐷 ( X ∗ , X 𝑡 ) ≥ 0 .

Hence, there exists a subsequence { X 𝑡 𝑘 : 𝑘 ≥ 1 } suct that lim 𝑘 →∞ X 𝑡 𝑘 = X ∗ , i.e.,

Then, (D.20) would give:

Step 5. Existence of lim 𝑡 →∞ 𝐷 ( X ∗ , X 𝑡 )

We define the sequence of events { 𝐸 𝑡 : 𝑡 ≥ 1 } as

and

Then we have that 𝐸 𝑡 ∈ F 𝑡 and 𝐸 𝑡 ⊆ { X 𝑠 ∈ U vs : 𝑠 = 1 , . . . , 𝑡 } .

Defining the random process GLYPH<8> ˜ 𝐷 𝑡 : 𝑡 ≥ 1 GLYPH<9> as

Then, by (D.16) we have

Multiplying the above relation with 1 𝐸 𝑡 , and noting that 1 𝐸 𝑡 + 1 ≤ 1 𝐸 𝑡 , since 𝐸 𝑡 + 1 ⊆ 𝐸 𝑡 , we have

where in the last step we used that tr [ V ( X 𝑡 )( X 𝑡 -X ∗ )] 1 𝐸 𝑡 ≤ 0 . Therefore, we obtain that:

where we used that

Therefore, GLYPH<8> ˜ 𝐷 𝑡 : 𝑡 ≥ 1 GLYPH<9> is an almost super-martingale [52] and, thus, there exists ˜ 𝐷 ∞ with ˜ 𝐷 ∞ finite (a.s.) and ˜ 𝐷 𝑡 → ˜ 𝐷 ∞ (a.s.).

Since 𝐸 = ∩ 𝑡 ≥ 1 𝐸 𝑡 , we have:

Hence, lim 𝑡 →∞ ˜ 𝐷 𝑡 exists on 𝐸 and by Step 3 we readily get that lim 𝑡 →∞ ˜ 𝐷 𝑡 = 0 on 𝐸 . Thus, by Lemma A.2, we get

and setting U = U 𝜀 / 4 , we obtain

This concludes our discussion and our proof.

E Numerical experiments

In this last appendix, we provide a series of additional numerical simulations to validate and explore the performance of (MMW) with payoff-based feedback.

■

(a)

Full information (MMW).

(b)

(3MW) with the (2PE) estimator

(c)

(3MW) with the (1PE) estimator

0

0

(0)

(0)

(0)

+

Q

1

Trajectory analysis. First, we proceed to a trajectory analysis of the game setup presented in Section 7. Specifically, in Fig. 2, we provide a visualization of the actual trajectories of play generated by the three methods with the same parameters as before, for different initial conditions. The trajectories are presented in Bloch spheres [48], where the points | 0 ⟩ and | 1 ⟩ in the figure correspond to the density matrices

respectively. In all figures, the points in red indicate the trajectory of Player 1, while the points in blue are for Player 2 . The initial points of the red trajectories are marked with · , while the initial points of the blue ones are marked with ■ . [Each column of Bloch spheres in Fig. 2 has the same initial conditions.]

An important remark here is that, as suggested by Theorem 4, the trajectories of all methods converge - and quite rapidly at that - to the game's (strict) Nash equilibrium. In fact, given that the trajectories converge to a pure state, this goes to explain the faster convergence rates observed in Fig. 1: instead

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 -5

10 -4

10 -3

10 -2

10 -1

10 0

10 1

10 2

MMW

2PE

1PE

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 -1

10 0

10 1

10 2

MMW

2PE

1PE

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 1

10 2

10 3

10 4

MMW

2PE

1PE

(a)

Performance evaluation and comparison on

Q

2

.

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 -4

10 -3

10 -2

10 -1

10 0

10 1

MMW

2PE

1PE

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 -1

10 0

10 1

10 2

10 3

MMW

2PE

1PE

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 0

10 1

10 2

10 3

10 4

10 5

10 6

MMW

2PE

1PE

(b)

Performance evaluation and comparison on

Q

3

.

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 -5

10 -4

10 -3

10 -2

10 -1

10 0

10 1

10 2

MMW

2PE

1PE

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 -1

10 0

10 1

10 2

10 3

MMW

2PE

1PE

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 1

10 2

10 3

10 4

10 5

MMW

2PE

1PE

(c)

Performance evaluation and comparison on

Q

.

4

of oscillating around a solution, the MMW orbits actually converge to equilibrium in this case, so the trailing average converges at a much faster rate. This holds in all zero-sum games with a pure equilibrium, thus indicating a very important class of zero-sum games where the worst-case guarantees of MMW algorithms can be significantly improved.

Convergence speed analysis. In addition to the game setup described in Section 7, we consider the following quantum games:

Q 2 : quantum analogue of the 2 × 2 min-max game with payoff matrix

Q 3 : quantum analogue of the 3 × 3 min-max game with payoff

Q 4 : quantum analogue of the 3 × 3 min-max game with payoff

In Fig. 3, we evaluate the convergence properties of (3MW) using the estimators (2PE) and (1PE), and compare it with the full information variant (MMW), following the same setup as described in Section 7. Specifically, for each method, we perform 10 different runs, with 𝑇 = 10 5 steps each, and compute the mean value of the duality gap as a function of the iteration 𝑡 = 1 , 2 , . . . , 𝑇 . The solid lines correspond to the mean values of the duality gap of each method, and the shaded regions enclose the area of ± 1 (sample) standard deviation among the 10 different runs. Note that the red line, which corresponds to the full information (MMW), does not have a shaded region, since there is no randomness in the algorithm. All the runs for the three different methods were initialized for Y = 0 and we used 𝛾 = 10 -2 for all methods. In particular, for (3MW) with gradient estimates given by (2PE) estimator, we used a sampling radius 𝛿 = 10 -2 , and for (3MW) with (1PE) estimator, we used 𝛿 = 10 -1 (in tune with our theoretical results which suggest the use of a tighter sampling radius when mixed payoff information is available to the players). As highlighted in the main text, we observe that the decrease in performance is mild, and the different algorithms achieved better rates than their theoretical guarantees.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the setting of learning in quantum games, in the case where agents only have access to a scalar (for instance, a payoff value) instead of their full individual gradient, and have to update their strategies in order to converge to some equilibrium. The primary contribution is a variant of the well-studied MMW algorithm called 3MW, which matches the equilibrium convergence rate of (full information) MMW in 2-player quantum zero-sum games. Moreover, with a relaxation of 3MW’s information requirements (i.e. that players observe a random realization of their payoff observable), the algorithm converges to equilibrium at O(T^{-1/4}) rate. Finally, in general N-player quantum games, a regularized version of 3MW using estimator 1PE is able to locally converge to Nash equilibria that are variationally stable.\n\nSTRENGTHS:\nThe paper answers an important question in the realm of learning in quantum games - in this framework it is quite unreasonable to assume full information of gradients, especially without properly defining how measurements are made, how players access said measurements etc. Hence, in my opinion the bandit framework is very fitting for the quantum setting, and indeed players should be able to update their strategies in a truly ‘online’ setting using only their measured/realized payoff observable. The techniques used in the paper, while not entirely novel, show interesting similarities and differences between classical and quantum settings. Finally, I think the results on variational stability are quite significant, as not only does this give (to my knowledge) the broadest last-iterate convergence result in N-player quantum games yet, it also points to a potential class of FTRL variants using bandit feedback that could potentially also be convergent in the last-iterate sense.\n\nWEAKNESSES:\nI think overall this is a strong paper, the only issue to me is that most of the techniques and ideas used in the paper are not particularly different from the classical bandit setting. If there is significant technical novelty, it is not made clear to the reader in the main text. I would be very interested to see what the differences are in the analysis between the quantum and classical setting, if there are any.\n\nQUESTIONS:\nIn the experiments, 3MW is compared with MMW and is shown to exhibit decent performance even with the lack of information. Does the practical performance of 3MW deteriorate significantly for games of larger dimension? To what extent can we rely on 3MW in practical settings (i.e. large games, games with larger number of players)?\n\nLIMITATIONS:\nLimitations are very briefly discussed, and perhaps a longer discussion would improve the paper (if space allows)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online learning of quantum algorithms based on the matrix multiplicative weight (MMW) updates. A main technical contribution is to introduce a weaker version called the minimal-information matrix multiplicative weights (3MW), which only requires zeroth-order gradient sampler and can achieve an overall regret of O(T^{-1/4}). In addition, for N-player quantum games, the paper proved a convergence result that 3MW guarantees local convergence with high probability to all equilibria that satisfy a certain first-order stability condition.\n\nSTRENGTHS:\nFrom my perspective, the most notable strength of this paper is the nice observation that less information is required to make MMW works. Considering that MMW has wide applications in quantum algorithms for machine learning and optimization, I believe that the result may be of general interest. Technically, this is achieved by exploiting ideas from machine learning, in particular bandit optimization, which provides interesting perspective for designing future quantum algorithms.\n\nWEAKNESSES:\nThe most notable weakness is the claim in intro that the result by Jain and Watrous [21] with O(\\sqrt{T}) regret for quantum online learning algorithms remains the tightest known bound for Nash equilibrium learning in quantum games. This is not true, as a recent paper by Gao et al. https://arxiv.org/abs/2304.14197 gave a quantum algorithm with poly-loga regret for zero-sum games. Compared to this paper, the results obtained by this paper is sub-optimal.\n\nAnother notable weakness is that the dependence in dimension is at least D, and does not present a quantum speedup here. In fact, a main merit of applying MMW into quantum algorithms is that MMW provides fast quantum algorithms for solving relevant problems. An important paper was Brandao and Svore https://arxiv.org/pdf/1609.05537.pdf in FOCS 2017, which can solve semidefinite programs with complexity square-root in D. If the current framework 3MW has at least linear dependence in D, it will prohibit its further applications in quantum computing.\n\nQUESTIONS:\n1. Can the authors further comment on whether the convergence rate of the proposed 3MW algorithm, or in other words the online regret, can be further improved? Given the existence of works such as Gao et al. mentioned above, it would be helpful if the authors can provide a more complete picture here. \n\n2. I’m confused by the sentence “in contrast to classical finite games, quantum games have an infinite continuum of pure states” in the abstract. On classical computers, even if we play games with finite cardinality, we can still choose mixed strategies and the set of probability distributions supported on a finite set can be uncountable.\n\n3. It would in general be helpful if the authors can introduce more on the applications of 3MW. In particular, in MMW the payoff gradient of player i is defined as V_i(X_t). However, in most existing quantum papers, V_i is very simple: the identity function. For instance, in the above Brandao-Svore paper as well as several important follow-up works that solves semidefinite programs with quantum speedup, V_i is the identity function and the update is simply a learning rate \\eta * X_t. \n\nIn all, it’s nice to know that 3MW can be applied with bandit information and for general V_i, but this should probably be further motivated about why we need to consider more complicated V_i’s.\n\n4. In numerical experiments, around Page 29, the authors claimed that the their algorithms achieved a rate of convergence closer O(1/T) instead of the weaker O(1/\\sqrt{T}) or O(1/T^{1/4}) bounds. However, I didn’t see detailed evidence for this; it would be helpful if the authors can present more detailed data and plots for demonstrating this phenomenon.\n\nNote that this is related to my question 1: if the convergence rate is ~O(1/T), the overall online regret will be poly-log. So it is probably worth closer study about this, both theoretically and experimentally.\n\nLIMITATIONS:\nThis work is mainly theoretical so I don't think there are notable limitations, but it would work better if the authors can articulate this more clearly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper studies Quantum Games (mainly in the two-player zero-sum setting) and develops payoff-based learning algorithms with provable convergence-rate guarantees. The learning algorithm combines matrix multiplicative weights (MMW) and zeroth order method for gradient estimation. The overall convergence rate is either $O(1/\sqrt{T})$ or $1/T^{1/4}$, depending the information structure. The authors also provide numerical simulations in the appendix to verify the performance of the proposed algorithm.

STRENGTHS:
This paper is very well-written. The appendix is also of high quality.

WEAKNESSES:
(1) The learning algorithm combines MMW with zeroth order method for gradient estimation. Since the convergence rate of MMW was established in the literature, and the zeroth order method is well studied in optimization, the technical novelty is not entirely clear. Are there any significant challenges in the analysis? 

(2) From Theorem 2, it seems that there are no consequences for choosing $\delta$ to be arbitrarily small. Am I missing anything?

QUESTIONS:
Do the authors expect the bound to be tight? Are there any known lower bounds for the convergence rate in the literature?

LIMITATIONS:
The authors did not explicitly point out the limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-16

comments: SUMMARY:
The authors consider the quantum games in the bandit setting, which players observe only the payoff value of their actions. In order to facilitate this, the authors extend the online bandit estimation methodology into the quantum-state space. As a result, they are able to realize a duality gap of $\mathcal O(T^{-\frac{1}{4}})$ and $\mathcal O(T^{-\frac{1}{2}})$ for bilinear quantum games.

STRENGTHS:
- Originality & Significance: Considering the concept of games within quantum state spaces in the bandit setting is innovative, and the result is nice and matches those found in classical matrix games.

- Quality & Clarity: The paper is effectively structured, allowing readers to follow the presented arguments and conclusions with ease.

WEAKNESSES:
- The applications of blinear quantum games are questionable.

QUESTIONS:
- I would like to inquire about the necessity of the $L$-smooth assumption in the quantum game setting. Specifically, it seems that in classical games or online learning problems, the bandit estimator can be constructed without requiring an $L$-smooth condition. Could the authors elaborate on whether this condition is necessary in the context of quantum games and, if so, could you explain the rationale behind this requirement?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-22