Title: Alternation makes the adversary weaker in two-player games
Venue: NeurIPS 2023
Accepted: yes
Keywords: Online Learning

Regret Minimization

Game Theory
Abstract: Motivated by alternating game-play in two-player games, we study an altenating variant of the \textit{Online Linear Optimization} (OLO). In alternating OLO,  a \textit{learner} at each round $t \in [n]$ selects a vector $x^t$ and then an \textit{adversary} selects a cost-vector $c^t \in [-1,1]^n$. The learner then experiences cost $(c^t + c^{t-1})^\top x^t$ instead of $(c^t)^\top x^t$ as in standard OLO. We establish that under this small twist, the $\Omega(\sqrt{T})$ lower bound on the regret is no longer valid. More precisely, we present two online learning algorithms for alternating OLO that respectively admit $\mathcal{O}((\log n)^{4/3} T^{1/3})$ regret for the $n$-dimensional simplex and $\mathcal{O}(\rho \log T)$ regret for the ball of radius $\rho>0$. Our results imply that in alternating game-play, an agent can always guarantee $\mathcal{\tilde{O}}((\log n)^{4/3} T^{1/3})$ regardless the strategies of the other agent while the regret bound improves to $\mathcal{O}(\log T)$ in case the agent admits only two actions.

Alternation makes the adversary weaker in two-player games

Abstract

Motivated by alternating game-play in two-player games, we study an altenating variant of the Online Linear Optimization (OLO). In alternating OLO, a learner at each round t ∈ [ n ] selects a vector x t and then an adversary selects a cost-vector c t ∈ [ -1 , 1] n . The learner then experiences cost ( c t + c t -1 ) ⊤ x t instead of ( c t ) ⊤ x t as in standard OLO. We establish that under this small twist, the Ω( √ T ) lower bound on the regret is no longer valid. More precisely, we present two online learning algorithms for alternating OLO that respectively admit O ((log n ) 4 / 3 T 1 / 3 ) regret for the n -dimensional simplex and O ( ρ log T ) regret for the ball of radius ρ > 0 . Our results imply that in alternating game-play, an agent can always guarantee ˜ O ((log n ) 4 / 3 T 1 / 3 ) regardless the strategies of the other agent while the regret bound improves to O (log T ) in case the agent admits only two actions.

1 Introduction

Game-dynamics study settings at which a set of selfish agents engaged in a repeated game update their strategies over time in their attempt to minimize their overall individual cost. In simultaneous play all agents simultaneously update their strategies, while in alternating play only one agent updates its strategy at each round while all the other agents stand still. Intuitively, each agent only updates its strategy in response to an observed change in another agent.

Alternating game-play captures interactions arising in various context such as animal behavior, social behavior, traffic networks etc. (see [33] for various interesting examples) and thus has received considerable attention from a game-theoretic point of view [11, 3, 33, 41, 40]. At the same time, alternation has been proven a valuable tool in tackling min-max problems arising in modern machine learning applications (e.g. training GANs, adversarial examples etc.) and thus has also been studied from an offline optimization perspective [37, 35, 21, 42, 9, 8, 10].

In the context of two-players, alternating game-play admits the following form: Alice (odd player) and Bob (even player) respectively update their strategies on odd and even rounds. Alice (resp. Bob) should select her strategy at an odd round so as to exploit Bob's strategy of the previous (even) round while at the same time protecting herself from Bob's response in the next (even) round. As a result, the following question arises:

Q1: How should Alice (resp. Bob) update her actions in the odd rounds so that, regardless of Bob's strategies, her overall cost (over the T rounds of play) is minimized?

1.1 Standard and Alternating Online Linear Minimization

Motivated by the above question and building on the recent line of research studying online learning settings with restricted adversaries [15, 25, 4, 5, 6, 34], we study an online linear optimization setting [43], called alternating online linear optimization . We use the term ' alternating ' to highlight the connection with alternating game-play that we subsequently present in Section 1.2.

In Algorithm 1 we jointly present both standard and alternating OLO so as to better illustrate the differences of the two settings.

Algorithm 1 Standard and Alternating Online Linear Minimization

1: Input : A feasibility set D ⊆ R n and c 0 ← (0 , . . . , 0) .

2: for each round t = 1 , . . . , T do

3: The learner selects a vector x t ∈ D based on c 1 , . . . , c t -1 ∈ [ -1 , 1] n

4: The adversary learns x t ∈ D and selects a cost vector c t ∈ [ -1 , 1] n (based on x 1 , . . . , x t ).

5: The learner learns c t ∈ [ -1 , 1] n and receives cost,

6: end for

In both standard and alternating OLO, the adversary selects c t after the the learner's selection of x t . The only difference between standard and alternating OLM is that in the first case the learner admits cost ( c t ) ⊤ x t while in the second its cost is ( c t + c t -1 ) ⊤ x t . An online learning algorithm 1 selects x t ∈ D solely based on the previous cost-vector sequence c 1 , . . . , c t -1 ∈ [ -1 , 1] n with the goal minimizing the overall cost that is slightly different in standard and alternating OLO.

The quality of an online learning algorithm A in standard OLO is captured through the notion of regret [23], comparing A 's overall cost with the overall cost of the best fixed action ,

When R A ( T ) = o ( T ) , the algorithm A is called no-regret since it ensured that regardless of the cost-vector sequence c 1 , . . . , c T , the time-averaged overall cost of A approaches the time-averaged overall cost of the best fixed action with rate o ( T ) /T → 0 . Correspondingly, the quality of an online learning algorithm A in alternating OLO is captured through the notion of alternating regret ,

Over the years various no-regret algorithms have been proposed for different OLO settings 2 achieving R A ( T ) = ˜ O ( √ T ) regret [28, 20, 43]. The latter regret bounds are optimal since there is is a simple √

probabilistic construction establishing that any online learning algorithm A admits R A ( T ) = Ω( T ) even when D is the 2 -dimensional simplex. This negative results comes from the fact that the adversary has access to the action x t of the algorithm and can appropriately select c t to maximize A 's regret.

At a first sight, it may seem that the adversary can still enforce Ω ( √ T ) alternating regret to any online learning algorithm A by appropriately selecting c t based on x t and possibly on c t -1 . Interestingly enough the construction establishing Ω( √ T ) regret, fails in the case of alternating regret (see Section 2). As a result, the following question naturally arises,

Q2 : Are there online learning algorthm with o ( √ T ) alternating regret?

Apart from its interest in the context of online learning, answering Q2 implies a very sound answer to Q1 . In Section 1.2 we present the connection between Alternating OLO and Alternating Game-Play.

1.2 Alternating OLO and Alternating Game-Play

Alternating game-play in the context of two-player games can be described formally as follows: Let ( A,B ) be a game played between Alice and Bob. The matrix A ∈ [ -1 , 1] n × m represents Alice's costs, A ij is the cost of Alice if she selects action i ∈ [ n ] and Bob selects action j ∈ [ m ] (respectively B ∈ [ -1 , 1] m × n for Bob). Initially Alice selects a mixed strategy x 1 ∈ ∆ n . Then,

At the even rounds t = 2 , 4 , 6 , . . . , 2 k : Bob plays a new mixed strategy y t ∈ ∆ m and Alice plays x t -1 ∈ ∆ n . Alice and Bob incur costs ( x t -1 ) ⊤ Ay t and ( y t ) ⊤ Bx t -1 respectively.

At the odd rounds t = 3 , 5 , . . . , 2 k -1 : Alice plays a new mixed strategy x t ∈ ∆ n and Bob plays y t -1 ∈ ∆ m . Alice and Bob incur costs ( x t ) ⊤ Ay t -1 and ( y t -1 ) ⊤ Bx t respectively.

From the perspective of Alice (resp. Bob), the question is how to select her mixed strategies x 1 , x 3 , . . . , x 2 k -1 ∈ ∆ n so as to minimize her overall cost

In Corollary 1.1 we establish that if Alice uses an online learning algorithm A then her overall regret (over the course of T rounds of play) is at most R alt A ( T/ 2) . As a result, in case Q2 admits a positive answer, then Alice can guarantee at most o ( √ T ) regret and improve over the ˜ O ( √ T ) regret bound provided by standard no-regret algorithms [28, 20, 43, 23].

Corollary 1.1. In case Alice (resp. Bob) uses an online learning algorithm A to update her strategies in the odd rounds, x 2 k +1 := A ( Ay 2 , Ay 4 , . . . , Ay 2 k ) for k = 1 , . . . , T / 2 -1 . Then no matter Bob's selected sequence y 2 , y 4 , . . . , y T ∈ ∆ m ,

Remark 1.2. We remark that Corollary 1.1 refers to the standard notion of regret [23] and R alt A ( T/ 2) appears only as an upper bound. We additionally remark that if both Alice and Bob respectively use algorithms A and B in the context of alternating play, then the time-average strategy vector converges with rate O (max( R A ( T ) , R B ( T )) /T ) to Nash Equilibrium in case of zero-sum games ( A = -B ⊤ ) and to Coarse Correlated Equilibrium for general two-player games [32]. Our objective is more general: we focus on optimizing the performance of a single player regardless of the actions of the other player.

1.3 Our Contribution and Techniques

In this work we answer Q2 on the affirmative. More precisely we establish that,

There exists an online learning algorithm (Algorithm 3) with alternating regret ˜ O ( (log n ) 4 / 3 T 1 / 3 ) for D = ∆ n ( n -dimensional simplex).

There exists an online learning algorithm (Algorithm 4) with alternating regret O ( ρ log T ) for D = B ( c, ρ ) (ball of radius ρ ).

There exists an online learning algorithm with alternating regret O (log T ) for D = ∆ 2 ( 2 -dimensional simplex), through a straight-forward reduction from D = B ( c, ρ ) .

Due to Corollary 1.1 our results provide a non-trivial answer to Q1 and establish that Alice can substantially improve over the O ( √ T ) regret guarantees of standard no-regret algorithms.

Corollary 1.3. In the context of alternating game play, Alice can always guarantee at most ˜ O ( (log n ) 4 / 3 T 1 / 3 ) regret regardless the actions of Bob. Moreover in case Alice admits only 2 actions ( n = 2 ), the regret bound improves to O (log T ) .

Bailey et al. [3] studied alternating game-play in unconstrained two-player games (the strategy space is R n instead of ∆ n ). They established that if the x -player (resp. the y -player) uses Online Gradient Descent (OGD) with constant step-size γ > 0 ( x 2 k := x 2 k -2 -γAy 2 k -1 ) then it experiences at most O (1 /γ ) regret regardless the actions of the y -player. In the context of alternating OLM this result implies that OGD admits O (1 /γ ) alternating regret as long as it always stays in the interior of D . However the latter cannot be guaranteed for bounded domains (simplex, ball). In fact there is a simple example for D = ∆ 2 at which OGD with γ step-size admits Ω(1 /γ + γT ) alternating regret. More recently, [40] studied alternating game-play in zero-sum games ( B = -A ⊤ ). They established that if both player adopt Online Mirror Descent (OMD) the individual regret of each player is at most O ( T 1 / 3 ) and thus the time-averaged strategies converge to Nash Equilibrium with O ( T -2 / 3 ) rate. The setting considered in this works differs because where the y -player can behave adversarially.

In order to achieve ˜ O ( (log n ) 4 / 3 T 1 / 3 ) alternating regret in case D = ∆ n , we first propose an ˜ O ( T 1 / 3 ) algorithm for the special case of D = ∆ 2 . For this special case our proposed algorithm is an optimistic-type of Follow the Regularized Leader (FTRL) with log-barrier regularization . Using the latter as an algorithmic primitive, we derive the ˜ O ( (log n ) 4 / 3 T 1 / 3 ) alternating regret algorithm for D = ∆ n , by upper bounding the overall alternating regret by the sum of local alternating regret of 2 -actions decision points on a binary tree at which the leafs corresponds to the actual n actions.

In order to achieve O ( ρ log T ) alternating regret for D = B ( c, ρ ) we follow a relatively different path. The major primitive of our algorithm is FTRL with adaptive step-size [16, 5]. The cornerstone of our approach is to establish that in case Adaptive FTRL admits more than O ( ρ log T ) alternating regret, then unormalized best-response ( -c t -1 ) can compensate for the additional cost. By using a recent result on Online Gradient Descent with Shrinking Domains [5], we provide an algorithm interpolating between Adaptive FTRL and -c t -1 that achieves O ( ρ log T ) alternating regret.

1.4 Further Related Work

The question of going beyond O ( T ) regret in the context of simultaneous game-play has received a lot of attention. A recent line of work establishes that if both agents simultaneously use the same no-regret algorithm (in most cases Optimistic Hedge) to update their strategies, then the individual regret of each agent is ˜ O (1) [1, 14, 13, 2, 36, 26, 17].

√

Our work also relates with the more recent works in establishing improved regret bounds parametrized by the cost-vector sequence c 1 , . . . , c T , sometimes also called 'adaptive' regret bounds [16, 29, 38, 30, 12]. However these parametrized upper bounds focus on finding 'easy' instances while still maintaining O ( √ T ) in the worst case. Alternating OLO can be considered as providing a slight "hint" to the learner that fundamentally changes the worst-case behavior, since its cost is ( c t + c t -1 ) ⊤ x t with the learner being aware of c t -1 prior to selecting x t . Improved regret bounds under different notions of hints have been established in [4, 5, 15, 34, 24, 39].

Finally our work also relates with the research line of no-regret learning in the context of Extensive Form Games [44, 18, 19] and Stackelberg Games [27, 22].

2 Preliminaries

We denote with ∆ n ⊆ R n the n -dimensional simplex, ∆ n := { x ∈ R n : x i ≥ 0 and ∑ n i =1 x i = 1 } . B ( c, ρ ) denotes the ball of radius ρ > 0 centered at c ∈ R n , B ( c, ρ ) := { x ∈ R n : ∥ x -c ∥ 2 ≤ ρ } . We also denote with [ x ] D := arg min z ∈D ∥ z -x ∥ 2 the projection operator to set D .

2.1 Standard and Alternating Online Linear Minimization

As depicted in Algorithm 1 the only difference between standard and Alternating OLM is the cost of the learner, ( c t ) ⊤ x t (OLM) and ( c t + c t -1 ) ⊤ x t (Alternating OLM). Thus, the notion of an online learning algorithm is exactly the same in both settings.

Definition 2.1. An online learning algorithm A , for an Online Linear Optimization setting with D ⊆ R n , is a sequence of functions A := ( A 1 , . . . , A t , . . . ) where A t : R d × . . . × R d ︸ ︷︷ ︸ t -1 ↦→D .

As Definition 2.1 reveals, the notion of an online learning algorithm depends only on the feasibility set D . As a result, an online learning algorithm A simultaneously admits both standard R A ( T ) and alternating regret R alt A ( T ) (see Equations 1 and 2 for the respective definitions). In Theorem 2.2, we present the well-known lower bound establishing that any online learning algorithm A admits R A ( T ) = Ω( √ T ) and explain why it fails in the case of alternating regret R alt A ( T ) .

Proof. Let c t be independently selected between ( -1 , 1) and (1 , -1) with probability 1 / 2 . Since c t is independent of ( c 1 , . . . , c t -1 ) then ∑ T t =1 E [ ( c t ) ⊤ x t ] = 0 where x t := A t ( c 1 , . . . , c t -1 ) . At the same time, E [ -min x ∈ ∆ 2 ∑ T t =1 ( c t ) ⊤ x ] ≤ O ( √ T ) . As a result, R A ( T ) ≥ Ω( √ T ) .

We now explain why the above randomized construction does not apply for alternating regret R alt A ( T ) . Let A be the best-response algorithm , A t ( c 1 , . . . , c t -1 ) := argmin x ∈ ∆ 2 ( c t -1 ) ⊤ x . Since c t = (1 , -1) or c t = ( -1 , 1) we get that min x ∈ ∆ 2 ( c t -1 ) ⊤ x = -1 while E [ ( c t ) ⊤ x t ] = 0 since x t := argmin x ∈ ∆ 2 ( c t -1 ) ⊤ x and c t is independent of c t -1 . As a result,

The latter implies that there exists at least one online learning algorithm ( Best-Response ) that admits Θ( -T ) alternating regret in the above randomized construction. However the latter is not very informative since there is a simple construction at which Best-Response admits linear alternating regret.

We conclude this section with the formal statement of our results. First, for the case that D is the simplex, we show ˜ O ( T 1 / 3 ) alternating regret (Section 3):

Theorem 2.3. Let D be the n -dimensional simplex, D = ∆ n . There exists an online learning algorithm A (Algorithm 3) such that for any cost-vector sequence c 1 , . . . , c T ∈ [ -1 , 1] n , T T

Next, when D is a ball of radius ρ , we can improve to ˜ O (1) alternating regret (Section 4):

Theorem 2.4. Let D be a ball of radius ρ , D = B ( c, ρ ) . There exists an online learning algorithm A (Algorithm 4) such that for any cost-vector sequence c 1 , . . . , c T where ∥ c t ∥ 2 ≤ 1 ,

Remark 2.5. Using Algorithm 4 we directly get an online learning algorithm with O (log T ) alternating regret for D = ∆ 2 .

2.2 Alternating Game-Play

A two-player normal form game ( A,B ) is defined by the payoff matrix A ∈ [ -1 , 1] n × m denoting the payoff of Alice and the matrix B ∈ [ -1 , 1] m × n denoting the payoff of Bob. Once the Alice selects a mixed strategy x ∈ ∆ n (prob. distr. over [ n ] ) and Bob selects a mixed strategy y ∈ ∆ m (prob. distr. over [ n ] ). Then Alice suffers (expected) cost x ⊤ Ay and Bob y ⊤ Bx .

In alternating game-play, Alice updates her mixed strategy in the even rounds while Bob updates in the odd rounds. As a result, a sequence of alternating play for T = 2 K rounds (resp. for T = 2 K +1 ) admits the form ( x 1 , y 2 ) , ( x 3 , y 2 ) , . . . , ( x 2 k +1 , y 2 k ) , ( x 2 k +1 , y 2 k +2 ) , . . . , ( x 2 K -1 , y 2 K ) . Thus, the regret of Alice in the above sequence of play equals the difference between her overall cost and the cost of the best-fixed action ,

If Alice selects x 2 k +1 := A k ( Ay 2 , Ay 4 , . . . , Ay 2 k -2 , Ay 2 k ) for k ∈ [ K -1] and x 1 = A 1 ( · ) then by the definition of alternating regret in Equation 2, we get that

which establishes Corollary 1.1. The proof for T = 2 K +1 is the same by considering Ay 2 K +2 = 0 .

3 The Simplex case

Before presenting our algorithm for the n -dimensional simplex, we present Algorithm 2 that admits O (log 2 / 3 T · T 1 / 3 ) alternating regret for the 2 -simplex and is the basis of our algorithm for ∆ n .

Definition 3.1 (Log-Barrier Regularization) . Let the function R : ∆ 2 ↦→ R ≥ 0 where R ( x ) := -log x 1 -log x 2 .

Algorithm 2 Online Learning Algorithm for 2D-Simplex

1: Input: c 0 ← (0 , 0)

2: for rounds t = 1 , . . . , T do

3: The learner selects x t := min x ∈ ∆ 2 [2( c t -1 ) ⊤ x + ∑ t -1 τ =1 ( c τ + c τ -1 ) ⊤ x + R ( x ) /γ ] .

4: The adversary selects cost vector c t ∈ [0 , 1] n

5: The learner suffers cost ( c t + c t -1 ) ⊤ x t

6: end for

In order to analyze Algorithm 2 we will compare its performance with the performance of the Be the Regularized Leader algorithm with log-barrier regularization that is ensured to achieve O (log T/γ ) alternating regret [23]. The latter is formally stated and established in Lemma 3.2.

In Lemma 3.3 we provide a closed formula capturing the difference between the output x t ∈ ∆ 2 of Algorithm 2 and the output y t ∈ ∆ 2 of Be the Regularized Leader algorithm defined in Lemma 3.2.

Lemma 3.3. Let x t = ( x t 1 , x t 2 ) ∈ ∆ 2 as in Algorithm 2 and y t = ( y t 1 , y t 2 ) ∈ ∆ 2 as in Lemma 3.2. Then,

Up next we use Lemma 3.2 and Lemma 3.3 to establish that Algorithm 2 admits O (log 2 / 3 T · T 1 / 3 ) alternating regret.

Theorem 3.4. Let x 1 , . . . , x T ∈ ∆ 2 the sequence produced by Algorithm 2 for the cost sequence c 1 , . . . , c T ∈ [ -1 , 1] 2 with γ = O ( log 1 / 3 T · T -1 / 3 ) then R alt ( T ) = O ( log 2 / 3 T · T 1 / 3 ) .

Proof. By Lemma 3.2 then ∑ t ∈ [ T ] ( c t + c t -1 ) ⊤ x t -min i ∈ [ n ] ∑ t ∈ [ T ] ( c t i + c t -1 i ) ≤ O (log T/γ ) + ∑ t ∈ [ T ] ( c t + c t -1 ) ⊤ ( x t -y t ) where y t ∈ ∆ 2 as in Lemma 3.2. Using Lemma 3.3 we get that

Hence R alt ( T ) ≤ O ( log T/γ + γ 2 T ) ) ≤ O ( log 2 / 3 T · T 1 / 3 ) for γ := O ( log 1 / 3 T/T 1 / 3 ) .

3.1 The n -Dimensional Simplex

In this section we extend Algorithm 2 to the case of the n -dimensional simplex. Our extension is motivated and builds upon the CFR algorithm develloped in the context of EFGs [44].

Without loss of generality we assume that n = 2 H . We consider a complete binary tree T ( V, E ) of height H = log n where the leaves L ⊆ V corresponds to the n actions , | L | = n . Each node s ∈ V/L admits exactly two children with ℓ ( s ) , r ( s ) respectively denoting the left and right child. Moreover, Level(h) ⊆ V denotes the nodes lying at depth h from the root ( Level(1) = { root } and Level(log n ) = L ). Up next we present the notion of policy on the nodes of T ( V, E ) .

Definition 3.5. · A policy over the nodes π : V/L ↦→ ∆ 2 encodes the probability of selecting the left/right child at node s ∈ V . Specifically π ( s ) = ( π ( ℓ ( s ) | s ) , π ( r ( s ) | s )) where π ( ℓ ( s ) | s ) + π ( r ( s ) | s ) = 1 and π ( ℓ ( s ) | s ) is the probability of selecting ℓ ( s ) (resp. for r ( s ) ).

Pr( s, i, π ) denotes the probability of reaching leaf i ∈ L starting from node s ∈ V/L and following π ( · ) at each step.

x π ∈ ∆ n denotes the probability distribution over the leaves/actions induced by π ( · ) . Formally, we have x π i := Pr(root , i, π ) for each leaf i ∈ L .

Definition 3.6. Given a cost vector c ∈ [ -1 , 1] n for the leaves/actions , the virtual cost of a node s ∈ V under policy π ( · ) , denoted as Q ( s, π, c ) , equals

The virtual cost vector of s ∈ V under π ( · ) is defined as q ( s, π, c ) := ( Q ( ℓ ( s ) , π, c ) , Q ( r ( s ) , π, c )) .

We remark that Q ( s, π, c ) is the expected cost of the random walk starting from s ∈ V and following policy π ( · ) until a leaf i ∈ L is reached in which case cost c i is occurred.

Our online learning algorithm for the n -dimensional simplex is illustrated in Algorithm 3.

Algorithm 3 An Online Learning Algorithm for the n -Dimensional Simplex

We remark that at each round t , the learner computes a policy π t ( · ) as an intermediate step (Step 6 ) that then uses to select the probability distribution x t := x π t ∈ ∆ n (Step 9 ). Notice that the computation of policy π t ( · ) is performed in Steps ( 4 )-( 8 ). Since nodes are processed in decreasing order (with respect to their level), during Step 6 π t ( · ) has already been determined for nodes ℓ ( s ) , r ( s ) and thus Q ( ℓ ( s ) , π t , c t -1 ) , Q ( r ( s ) , π t , c t -1 ) are well-defined.

Up next we present the main steps for establishing Theorem 2.3. A key notion in the analysis of Algorithm 3 is that of local alternating regret of a node s ∈ V presented in Definition 3.7. As established in Lemma 3.8 the overall alternating regret of Algorithm 3 can be upper bounded by the sum of the local alternating regrets of the nodes lying in the path of the best fixed leaf/action .

Definition 3.7. For any sequence c 1 , . . . , c T ∈ [ -1 , 1] n the alternating local regret of a node s ∈ V , denoted as R T loc ( s ) , is defined as

Lemma 3.8. Let a leaf/action i ∈ L and consider the path p = (root = s 1 , . . . , s H = i ) from the root to the leaf i ∈ L . Then, ∑ T t =1 ( c t + c t -1 ) ⊤ x π t -2 ∑ T t =1 c t i ≤ ∑ H ℓ =1 R loc ( s ℓ ) .

̸

Up to this point, it is evident that in order to bound the overall alternating regret of Algorithm 3, we just need to bound the local alternating regret of any node s ∈ V . Using Theorem 3.4 we can bound the local regret of leaves i ∈ L for which q ( i, π t , c t -1 ) = q ( i, π t -1 , c t -1 ) . However this approach does apply for nodes s ∈ V/L since the local regret does not have the alternating structure , q ( s, π t , c t -1 ) = q ( s, π t -1 , c t -1 ) . To overcome the latter in Lemma 3.9 we establish that q ( s, π t , c t -1 ) , q ( s, π t -1 , c t -1 ) are in distance O ( γ ) which permits us to bound R T loc ( s ) for s ∈ V/L by tweaking the proof of Theorem 3.4.

Using Lemma 3.9 we can establish an upper bound on the local regret of any actions s ∈ V . The proof of Lemma 3.10 lies in Appendix B and follows a similar structure with the proof of Theorem 3.4.

Theorem 2.3 directly follows by combining Lemma 3.10, Lemma 3.8 and H = log n .

4 The Ball case

In Algorithm 4 we present an online learning algorithm with O (log T ) for D = B (0 , 1) and ∥ c t ∥ 2 ≤ 1 . Then through the transformation ˆ x t := c + ρx t with x t ∈ B (0 , 1) , Algorithm 4 can be transformed to a O ( ρ log T ) -alternating regret algorithm for D = B ( c, ρ ) .

Algorithm 4 may seem complicated at the first sight however it is composed by two basic algorithmic primitives. At Step 4 Algorithm 4 computes the output w t ∈ B (0 , 1) of the Follow the Regularized Leader (FTRL) with Euclidean regularization and adaptive step-size r 0: t -1 (Step 3 of Algorithm 4).At Step 5 , it mixes the output w t ∈ B (0 , 1) of FTRL with the unnormalized best-response -c t -1 ∈ B (0 , 1) . The selection of the mixing coefficient p t is adaptively updated at Step 7 .

4.1 Proof of Theorem 2.4

In this section we present the main steps of the proof of Theorem 2.4. In Lemma 4.1 we provide a first upper bound on the alternating regret of Adaptive FTRL .

Lemma 4.1. Let w 1 , . . . , w T ∈ B (0 , 1) the sequence produced by Adaptive FTRL (Step 4 of Algorithm 4) given as input the cost-vector sequence c 1 , . . . , c T ∈ B (0 , 1) . Let t 1 denote the maximum time-index such that ∑ t s =1 ( c s + c s -1 ) ⊤ w t ≥ -∑ t s =1 ∥ c s + c s -1 ∥ 2 2 / 4 . Then,

Lemma 4.1 guarantees that Adaptive FTRL admits only o ( √ T ) alternating regret in case t 1 = o ( T ) . Using Lemma 4.1, we establish Lemma 4.2 which is the cornerstone of our algorithm and guarantees that once Adaptive FTRL is appropriately mixed with unormalized best-response ( -c t -1 ), then the resulting algorithm always admits O (log T ) regret.

Lemma 4.2. Let w 1 , . . . , w T ∈ B (0 , 1) be produced by Adaptive FTRL given as input c 1 , . . . , c T ∈ B (0 , 1) and t 1 be the maximum round such that ∑ t s =1 ( c s + c s -1 ) ⊤ w s ≥ -∑ t s =1 ∥ c s + c s -1 ∥ 2 2 / 4 .

Algorithm 4 Online Learning Algorithm for Unit Ball

1: p 1 ← 0 , D 1 ← [0 , 1] and c 0 ← (0 , . . . , 0) .

2: for each round t = 1 , · · · , T do

3: The learner computes the coefficient r 0: t -1 ← √ 1 + ∑ t -1 s =1 ∥ c s + c s -1 ∥ 2 2

4: The learner computes the output of FTRL,

5: The learner selects the action x t ← (1 -p t ) w t + p t ( -c t -1 ) # Mixing Adaptive FTRL with Unormalized Best-Response

6: The adversary selects cost c t with ∥ c t ∥ 2 ≤ 1 and the learner suffers cost ( c t -1 + c t ) ⊤ x t .

7: The learner updates the interval D t ⊆ [0 , 1] as follows,

and then updates the coefficient p t ∈ [0 , 1] as follows,

8: end for

Lemma 4.2 establishes that in case at Step 5 , Algorithm 4 mixed the output w t of Adaptive FTRL with the unormalized best-response ( -c t -1 ∈ B (0 , 1) ) as follows,

then it would admit O (log T ) alternating regret. Obviously, Algorithm 4 does not know a-priori neither t 1 nor ∑ t 1 t =1 ∥ c t + c t -1 ∥ 2 2 . However by using the recent result of [5] for Online Gradient Descent in Shrinking Domains , we can establish that the mixing coefficients p t ∈ [0 , 1] selected by Algorithm 4 at Step 7 , admit the exact same result as selecting q t ∈ [0 , 1] described in Equation 3. The latter is formalized in Lemma 4.3.

Lemma 4.3. Let the sequences w 1 , . . . , w T ∈ B (0 , 1) and p 1 , . . . , p T ∈ (0 , 1) produced by Algorithm 4 given as input c 1 , . . . , c T ∈ B (0 , 1) . Additionally let t 1 denote the maximum time such that ∑ t s =1 ( c s + c s -1 ) ⊤ w s ≥ -∑ t s =1 ∥ c s + c s -1 ∥ 2 2 / 4 and consider the sequence

5 Conclusion

In this paper we introduced a variant of the Online Linear Optimization that we call Alternating Online Linear Optimization for which we developed the first online learning algoithms with o ( √ T ) regret guarantees. Our work is motivated by the popular setting of alternating play in two-player games and raises some interesting open questions. The most natural ones is understanding whether

˜ O (1) regret guarantees can be established the n -dimensional simplex as well as establishing o ( √ T ) for general convex losses.

Limitations: The current work is limited to the linear losses setting. Notice that the classic reduction from convex to linear losses in Standard OLM no longer holds in Alternating OLM. Therefore the generalization to general convex losses seems to require new techniques. We defer this study for future work.

References

Ioannis Anagnostides, Constantinos Daskalakis, Gabriele Farina, Maxwell Fishelson, Noah Golowich, and Tuomas Sandholm. Near-optimal no-regret learning for correlated equilibria in multi-player general-sum games. In Stefano Leonardi and Anupam Gupta, editors, STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 736-749. ACM, 2022.

Ioannis Anagnostides, Gabriele Farina, Christian Kroer, Chung-Wei Lee, Haipeng Luo, and Tuomas Sandholm. Uncoupled learning dynamics with o(log T) swap regret in multiplayer games. CoRR , abs/2204.11417, 2022.

James P. Bailey, Gauthier Gidel, and Georgios Piliouras. Finite regret and cycles with fixed step-size via alternating gradient descent-ascent. In Jacob D. Abernethy and Shivani Agarwal, editors, Conference on Learning Theory, COLT 2020, 9-12 July 2020, Virtual Event [Graz, Austria] , volume 125 of Proceedings of Machine Learning Research , pages 391-407. PMLR, 2020.

Aditya Bhaskara, Ashok Cutkosky, Ravi Kumar, and Manish Purohit. Online learning with imperfect hints. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 822-831. PMLR, 2020.

Aditya Bhaskara, Ashok Cutkosky, Ravi Kumar, and Manish Purohit. Logarithmic regret from sublinear hints. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 28222-28232, 2021.

Aditya Bhaskara, Ashok Cutkosky, Ravi Kumar, and Manish Purohit. Power of hints for online learning with movement costs. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021 , volume 130 of Proceedings of Machine Learning Research , pages 2818-2826. PMLR, 2021.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Antonin Chambolle and Thomas Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. , 40(1):120-145, 2011.

Tatjana Chavdarova, Gauthier Gidel, François Fleuret, and Simon Lacoste-Julien. Reducing noise in GAN training with variance reduced extragradient. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett,

editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 391-401, 2019.

Tatjana Chavdarova, Matteo Pagliardini, Sebastian U. Stich, François Fleuret, and Martin Jaggi. Taming gans with lookahead-minmax. In 9th International Conference on Learning Representations, ICLR 2021 . OpenReview.net, 2021.

Steve Chien and Alistair Sinclair. Convergence to approximate nash equilibria in congestion games. In Nikhil Bansal, Kirk Pruhs, and Clifford Stein, editors, Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2007, New Orleans, Louisiana, USA, January 7-9, 2007 , pages 169-178. SIAM, 2007.

Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in banach spaces. In Conference On Learning Theory , pages 1493-1529. PMLR, 2018.

Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no-regret algorithms for zero-sum games. In Dana Randall, editor, Proceedings of the Twenty-Second Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2011 , pages 235-254. SIAM, 2011.

Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 27604-27616, 2021.

Ofer Dekel, Arthur Flajolet, Nika Haghtalab, and Patrick Jaillet. Online learning with a hint. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 , pages 5299-5308, 2017.

John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. In Adam Tauman Kalai and Mehryar Mohri, editors, COLT 2010 - The 23rd Conference on Learning Theory, Haifa, Israel, June 27-29, 2010 , pages 257-269. Omnipress, 2010.

Liad Erez, Tal Lancewicki, Uri Sherman, Tomer Koren, and Yishay Mansour. Regret minimization and convergence to equilibria in general-sum markov games. CoRR , abs/2207.14211, 2022.

Gabriele Farina and Tuomas Sandholm. Model-free online learning in unknown sequential decision making problems and games. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021 .

Gabriele Farina, Robin Schmucker, and Tuomas Sandholm. Bandit linear optimization for sequential decision making and extensive-form games. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021 .

Yoav Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. , 55(1):119-139, 1997.

Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. A variational inequality perspective on generative adversarial networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Denizalp Goktas, Jiayi Zhao, and Amy Greenwald. Robust no-regret learning in min-max stackelberg games. In 21st International Conference on Autonomous Agents and Multiagens Systems, AAMAS 2022 .

Elad Hazan. Introduction to online convex optimization. Found. Trends Optim. , 2(3-4):157-325, 2016.

Elad Hazan and Satyen Kale. Extracting certainty from uncertainty: regret bounded by variation in costs. Mach. Learn. , 80(2-3):165-188, 2010.

Elad Hazan and Nimrod Megiddo. Online learning with prior knowledge. In Nader H. Bshouty and Claudio Gentile, editors, Learning Theory, 20th Annual Conference on Learning Theory, COLT 2007 , volume 4539 of Lecture Notes in Computer Science , pages 499-513. Springer, 2007.

Yu-Guan Hsieh, Kimon Antonakopoulos, and Panayotis Mertikopoulos. Adaptive learning in continuous games: Optimal regret bounds and convergence to nash equilibrium. In Mikhail Belkin and Samory Kpotufe, editors, Conference on Learning Theory, COLT 2021 , volume 134 of Proceedings of Machine Learning Research , pages 2388-2422. PMLR, 2021.

Niklas T. Lauffer, Mahsa Ghasemi, Abolfazl Hashemi, Yagiz Savas, and Ufuk Topcu. No-regret learning in dynamic stackelberg games. CoRR , abs/2202.04786, 2022.

Nick Littlestone and Manfred K. Warmuth. The weighted majority algorithm. 108(2), 1994.

Haipeng Luo and Robert E Schapire. Achieving all with no parameters: Adanormalhedge. In Conference on Learning Theory , pages 1286-1304. PMLR, 2015.

H Brendan McMahan. A survey of algorithms and analysis for adaptive online learning. The Journal of Machine Learning Research , 18(1):3117-3166, 2017.

H. Brendan McMahan. A survey of algorithms and analysis for adaptive online learning. J. Mach. Learn. Res. , 18:90:1-90:50, 2017.

Noam Nisan, Tim Roughgarden, Eva Tardos, and Vijay V Vazirani. Algorithmic game theory . Cambridge university press, 2007.

Peter Park, Martin Nowak, and Christian Hilbe. Cooperation in alternating interactions with memory constraints. Nature Communications , 13:737, 02 2022.

Alexander Rakhlin and Karthik Sridharan. Online learning with predictable sequences. In Shai Shalev-Shwartz and Ingo Steinwart, editors, COLT 2013 - The 26th Annual Conference on Learning Theory , volume 30 of JMLR Workshop and Conference Proceedings , pages 993-1019. JMLR.org, 2013.

Mehmet Fatih Sahin, Armin Eftekhari, Ahmet Alacaoglu, Fabian Latorre Gómez, and Volkan Cevher. An inexact augmented lagrangian framework for nonconvex optimization with nonlinear constraints. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 , pages 13943-13955, 2019.

Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E. Schapire. Fast convergence of regularized learning in games. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015 , pages 2989-2997, 2015.

Oskari Tammelin, Neil Burch, Michael Johanson, and Michael Bowling. Solving heads-up limit texas hold'em. In Qiang Yang and Michael J. Wooldridge, editors, Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015 , pages 645-652. AAAI Press, 2015.

Tim Van Erven and Wouter M Koolen. Metagrad: Multiple learning rates in online learning. Advances in Neural Information Processing Systems , 29, 2016.

Chen-Yu Wei, Haipeng Luo, and Alekh Agarwal. Taking a hint: How to leverage loss predictors in contextual bandits? In Jacob D. Abernethy and Shivani Agarwal, editors, Conference on Learning Theory, COLT 2020, 9-12 July 2020, Virtual Event [Graz, Austria] , volume 125 of Proceedings of Machine Learning Research , pages 3583-3634. PMLR, 2020.

Andre Wibisono, Molei Tao, and Georgios Piliouras. Alternating mirror descent for constrained min-max games. In NeurIPS , 2022.

Guodong Zhang, Yuanhao Wang, Laurent Lessard, and Roger B. Grosse. Near-optimal local convergence of alternating gradient descent-ascent for minimax optimization. In Gustau CampsValls, Francisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artificial Intelligence and Statistics, AISTATS 2022 , volume 151 of Proceedings of Machine Learning Research , pages 7659-7679. PMLR, 2022.

Guojun Zhang and Yaoliang Yu. Convergence of gradient methods on bilinear zero-sum games. In 8th International Conference on Learning Representations, ICLR 2020 . OpenReview.net, 2020.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Tom Fawcett and Nina Mishra, editors, Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA , pages 928-936. AAAI Press, 2003.

Martin Zinkevich, Michael Johanson, Michael H. Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, NIPS 2007 , pages 1729-1736. Curran Associates, Inc., 2007.

A Omitted Proofs of Section 3

A.1 Auxilliary Lemmas

Lemma A.1. The log-barrier function R ( x ) = -log x -log(1 -x ) is 1 -strongly convex in [0 , 1] . More precisely, for all x, y ∈ [0 , 1]

Proof. Let f ( x ) := -log x then f ′ ( x ) = -1 x and f ′′ ( x ) = 1 x 2 . Since x ≤ 1 we get that f ′′ ( x ) ≥ 1 and thus

At the same time the function f ( x ) = -log(1 -x ) is convex in [0 , 1] . This concludes the proof.

Lemma A.2. Let x := argmin z ∈ [0 , 1] [ γc · z + R ( z )] and y := argmin z ∈ [0 , 1] [ γ ˆ c · z + R ( z )] where R ( · ) is an 1 -strongly convex function in R . Then,

Proof. By the strong convexity of the function γc T z + R ( z ) and first order optimality conditions for x , we get that

As a result, we get that

which implies that | x -y | ≤ 2 γ | c -ˆ c |

A.2 Proof of Lemma 3.2

Proof. We start by rewrite the regret minimization problem over ∆ 2 as an equivalent one over [0 , 1] , that is

where ˆ c t = c t 1 -c t 2 . Moreover notice that

By the "Follow the Leader/Be the Leader" Lemma [7, Lemma 3.1] , we have that

That implies

.

Now let x ⋆ = arg min p ∈ [0 , 1] ∑ T t =1 (ˆ c t + ˆ c t -1 ) p and let us subtract ∑ T t =1 (ˆ c t + ˆ c t -1 ) x ⋆ from both sides

In case x ⋆ = 0 , we upper bound the minimum on the right hand side with the same expression evaluated at p := 1 /T . As a result,

In case x ⋆ = 1 , we upper bound the minimum on the right hand side by the expression evaluated at p := 1 -1 /T . As a result,

Therefore putting together Equation (5) and Equation (6), we can conclude that ∑ T t =1 ( c t + c t -1 ) ⊤ x t -min i ∈ [2] ∑ T t =1 ( c t i + c t -1 i ) ≤ 2 log T/γ +2 .

A.3 Proof of Lemma 3.3

Before presenting the formal proof of Lemma 3.3 we present Lemma A.3 and Lemma A.4 that are necessary for its proof.

Lemma A.3. Let x t as in Algorithm 2 and y t 1 be the BTRL update as in Lemma 3.2 with R ( x ) = -log x -log(1 -x ) and x t 1 be the update as in Algorithm 3. Then the following hold.

| x t 1 -y t 1 | ≤ 8 γ

| x t 1 -x t +1 1 | ≤ 16 γ

Proof. Notice that for any x, y ∈ ∆ 2 and cost vector c = ( c 1 , c 2 ) ∈ R 2 , we have that

This means that we can reduce the bidimensional update in Algorithm 2 as

At this point, using strong convexity of the log barrier function (Lemma A.1), the form of the updates in Equation (4) and Equation (7), we can invoke Lemma A.2 using x = x t 1 and y = y t 1 , this gives

where we used that the cost sequence is in [ -1 , 1] . For the second fact, we invoke again Lemma A.2 but with x = x t 1 and y = x t +1 1 and we obtain

For the third fact, we use Lemma A.2 but with x = y t 1 and y = y t +1 1 and we obtain

Lemma A.4. Let ( x, y ) ∈ [0 , 1] 2 and ( x ′ , y ′ ) ∈ [0 , 1] 2 such that | x -y | ≤ B , | x -y ′ | ≤ B , | x ′ -y | ≤ B and | x ′ -y ′ | ≤ B with B ≤ 1 8 then

where

A

(

x, y

) = (

xy

)

-

1

-

(1

-

x

)

-

1

(1

-

y

)

-

1

.

Proof. To simplify notation we denote x t := tx +(1 -t ) x ′ and y t := ty +(1 -t ) y ′ . Then

Let us focus on bounding ∥∇ A -1 ( x t , y t ) ∥ ∞ . Notice that

Now, notice that | x t -y t | ≤ t | x -y | +(1 -t ) | x ′ -y ′ | ≤ B . Using the latter we can lower bound the denominator of Equation 9. More precisely,

So for B ≤ 1 8 we obtain

By symmetricity, we can bound with analogous steps the partial derivative wrt to y and hence we get

Plugging this bound back in Equation (8) concludes the proof.

Lemma 3.3. Let x t = ( x t 1 , x t 2 ) ∈ ∆ 2 as in Algorithm 2 and y t = ( y t 1 , y t 2 ) ∈ ∆ 2 as in Lemma 3.2. Then,

Proof. In order to prove this Lemma 3.3, we use the equivalent one-dimensional description provided in Equation 10.

Similarly the update of BTRL in Lemma 3.2 can be equivalently descirbed as,

Since lim p → ∂ [0 , 1] R ( p ) = ∞ both x t 1 , y t 1 ∈ [0 , 1] \ ∂ [0 , 1] . Therefore, the first order optimality for Equation (7) requires that

Using the same reasoning for the BTRL updates in Equation (4)

Now, subtracting Equation (12) to Equation (13), we obtain

that implies

Therefore, we can express the difference between the updates as a function of the costs according to the following formula

We conclude the proof by establishing that

By Lemma A.3 we are ensured that

In case γ ≤ 1 / (16 · 8) we are ensured that the conditions of Lemma A.4 are satisfied ( B ≤ 1 / 8 ) and thus

Combining the latter with the guarantees of Lemma A.4 we get that

B Omitted proofs for the n dimensional case.

B.1 Auxiliary Lemmas

Proof. For fact i) for any s ∈ Level( h ) , we have that

where the second last equality uses the fact that s ∈ Level( h ) = ⇒ ℓ ( s ) , r ( s ) ∈ Level( h +1) . Finally, fact ii) follows trivially from the definition of x π . Indeed, we have that

B.2 Proof of Lemma 3.8

Lemma 3.8. Let a leaf node i ∈ L and let p = (root = s 1 , . . . , s H = i ) denotes the path from the root to i . Then the following holds,

Proof. By Item 2 of Corollary B.1 and the fact that c 0 = 0 , we get

B.3 Proof of Lemma 3.9

Lemma 3.9. Let π 1 , . . . , π T the policies produced by Algorithm 3 then for any state s ∈ V , i ) ∥ π t ( s ) -π t -1 ( s ) ∥ 1 ≤ 48 γ and ii ) ∥ q ( s, π t , c t -1 ) -q ( s, π t -1 , c t -1 ) ∥ ∞ ≤ 48 γ log n .

Proof. We first establish that ∥ π t ( s ) -π t -1 ( s ) ∥ 1 ≤ 48 γ .

Let ¯ Q ( s, π, c ) := Q ( ℓ ( s ) , π, c ) -Q ( r ( s ) , π, c ) then policy update in Step 6 of Algorithm 3 admits the following one dimensional form,

Similarly for the policy π t -1 ,

Using Lemma A.2 we get that,

where the last inequality comes from the fact that -1 ≤ Q ( s, π, c ) ≤ 1 and thus ∣ ∣ ¯ Q ( s, π, c ) ∣ ∣ ≤ 2 . Up next we establish that

To simplify notation we let h 0 denote the depth of state s 0 ∈ V . Up next we prove that

In order to prove the latter we deploy a coupling argument by considering two correlated random walks ( s 0 , s 0 ) , ( s 1 , s ′ 1 ) , . . . , ( s H , s ′ H ) where both walks are initialized at ( s 0 , s 0 ) while at each level h ∈ { h 0 , . . . , H -1 } , the first walk marginally follows policy π ∈ ∆ 2 while the second walk marginally follows π ′ ∈ ∆ 2 .

More precisely, let ( s h , s ′ h ) the pair of nodes visited respectively by the first and the second walk at level h ∈ { h 0 , . . . , H -1 } . Then the next pair of nodes ( s h , s ′ h ) follow the following joint probability distribution.

̸

In case s ′ h = s h : The next pair of nodes ( s h +1 , s ′ h +1 ) are independent random variables respectively following π ( s h ) ∈ ∆ 2 and π ′ ( s ′ h ) ∈ ∆ 2 . More precisely,

In case s h = s ′ h = s and π ( ℓ ( s ) | s ) ≤ π ′ ( ℓ ( s ) | s ) : Then the next pair of nodes ( s j , s ′ h ) follows the joint probability distribution,

In case s h = s ′ h = s and π ( ℓ ( s ) | s ) ≥ π ′ ( ℓ ( s ) | s ) : Then the next pair of nodes ( s j , s ′ h ) follows the joint probability distribution,

The above joint random walk, guarantees that the first random walk (resp. the second) follows policy π (resp. π ′ for the second coordinate). More precisely,

As a result,

where ( i, i ′ ) ∈ L × L denotes the pair of leaves reached by the joint random walk initialized at s 0 ∈ V/L .

̸

̸

̸

̸

where in the second equality we used the fact that { s ′ h +1 = s h +1 , s ′ h = s h = s } s ∈ V,h ∈ [ H ] are disjoint events.

By setting π ′ = π t and π = π t -1 we get that

Finally,

B.4 Proof of Lemma 3.10

Lemma 3.10. Let γ := O ( log 1 / 3 T/ ( T 1 / 3 log 1 / 3 n ) ) in Algorithm 3 then R T loc ( s ) ≤ O ( log 2 / 3 T · log 1 / 3 n · T 1 / 3 ) for all s ∈ V .

Proof. Let the step-size γ > 0 of Algorithm 3 defined as γ := 1 32 · 8 ( log( T ) T log n ) 1 3 . Let us also introduce the BTRL update for state s that is

̸

We can bound two separate sources of regret, according to the decomposition

First, we recognize that Term I is the BTRL local regret, therefore applying Lemma 3.2, we have

Then, it remains to bound the term that quantifies the closeness between π t and ˜ π t , that is

Let ¯ Q ( s, π, c ) := Q ( ℓ ( s ) , π, c ) -Q ( r ( s ) , π, c ) then by using Corollary B.1 we get that

At the same time by Lemma 3.3 we get that

Hence combining Equation 17 with Equation 18 we obtain

At this point, we notice that unfortunately we can not rearrange the sum easily because of the term ¯ Q 2 ( s, π t , c t -1 ) that depends on both indices t and t -1 . To go around this issue, we add and subtract the term ¯ Q 2 ( s,π t -1 ,c t -1 ) A ( π t ( ℓ ( s ) | s ) , ˜ π t ( ℓ ( s ) | s )) ,

Now we bound the first term. Notice that the assumption of Lemma A.4 are satisfied with B = 32 γ and that γ ≤ (8 · 32) -1 ensures B ≤ 1 8 . , Therefore, rearranging the sum and invoking Lemma A.4

where in the last inequality we used ¯ Q 2 ( s, π t , c t ) ≤ 4 ∀ t and A ( π T ( ℓ ( s ) | s ) , ˜ π T ( ℓ ( s ) | s )) ≥ 1 8 . Then, for the second term in Equation (19), we use the second fact of Lemma 3.9. In more details, we have that

Therefore

Therefore, neglecting constants, and plugging in the bounds in Equation (16), we obtain

Therefore by our selection of γ := O ( log 1 / 3 T/ ( T 1 / 3 log 1 / 3 n ) ) we get

B.5 Proof of Therem 2.3

Theorem 2.3. Let D be the n -dimensional simplex, D = ∆ n . There exists an online learning algorithm A (Algorithm 3) such that for any cost-vector sequence c 1 , . . . , c T ∈ [ -1 , 1] n ,

Then, recalling that by construction H = log n and using the bound on R T loc ( s ) in Lemma 3.10 gives

C Omitted Proof of Section 4

In this section we present the omitted proofs of Section 4.

C.1 Proof of Lemma 4.1

To simplify notation we denote ˆ c t := c t + c t -1 for t ≥ 1 where c 0 = (0 , . . . , 0) . Moreover we denote with ∥·∥ the euclidean norm ∥·∥ 2 . Adaptive FTRL (Algorithm 5) admits the following equivalent form.

Algorithm 5 Adaptive FTRL

1: for round t = 1 , . . . , T do

2: The learner computes r 0: t -1 ← √ 1 + ∑ t -1 s =1 ∥ ˆ c s ∥

4: The adversary selects cost ˆ c t with ∥ ˆ c t ∥ 2 ≤ 2 and the learner receives cost ˆ c ⊤ t · x t .

5: end for

Lemma C.1 ([5]) . Let w 1 , . . . , w T ∈ B (0 , 1) the sequence of points produced by Adaptive FTRL given as input the cost-vector sequence ˆ c 1 , . . . , ˆ c T and x ∗ := argmin x ∈B (0 , 1) [ ∑ T t =1 ˆ c ⊤ t x ] . Then for any index S ∈ [ T ] ,

where r t = r 0: t -r 0: t -1 for t ≥ 1 .

Proof. Let f t ( x ) := ˆ c ⊤ t x + r t 2 ∥ x ∥ 2 where r 0 = 1 and ˆ c 0 = 0 . Let us also define f 0: t ( x ) := ∑ t t =0 f t ( x ) . Since ˆ c 0 = 0 we get that f 0: t ( x ) = ∑ t s =1 ˆ c ⊤ s x + r 0: t 2 ∥ x ∥ 2 and thus w t +1 := argmin x ∈ B (0 , 1) f 0: t ( x ) . Then,

As a result we get that,

By rearranging the terms and using the fact that ˆ c 0 = 0 and r 0 = 1 we get that,

Lemma C.2 ([6]) . Let w 1 , . . . , w T ∈ B (0 , 1) the sequence of points produced by Adaptive FTRL given as input the cost-vector sequence ˆ c 1 , . . . , ˆ c T and x ∗ := argmin x ∈B (0 , 1) [ ∑ T t =1 ˆ c ⊤ t x ] . Then,

Proof. Applying Lemma C.1 with S = 0 we get that,

Up next we bound the second term. Let f t ( x ) := ˆ c ⊤ t x + r t 2 . By Lemma 7 in [31] for f 1 := f 0: t -1 and f 2 := f 0: t . Since f 1 is 1 -strongly convex with respect to the norm r 0: t -1 ∥ x ∥ 2 and f 2 -f 1 is convex and 2 ∥ c t ∥ -Lipschitz. Then since w t := argmin x ∈B (0 , 1) f 1 ( x ) and w t +1 := argmin x ∈B (0 , 1) f 2 ( x ) , Lemma 7 in [31] implies that

As a result, we get that

Summing from t = 1 to T , we get that

Lemma C.3. Let w 1 , . . . , w T ∈ B (0 , 1) the sequence of points produced by Adaptive FTRL given as input the cost-vector sequence ˆ c 1 , . . . , ˆ c T . Let any round t ∗ ∈ [ T ] such that for all t ≥ t ∗ +1 ,

Then ∥ w t ∥ = 1 for all t ≥ t ∗ +1 and additionally,

Proof. To simplify notation we denote ˆ σ t := ∥ ˆ c t ∥ 2 Moreover we denote ˆ c 1: t = ∑ t s =1 ˆ c s and ˆ σ 1: t = ∑ t s =1 ˆ σ s . By the definition of t ∗ ∈ [ T ] we know that for all t ≥ t ∗ +1 ,

where the last inequality follows by the fact that σ 1: t ≥ 17 . Since w t ∈ B (0 , 1) the latter implies that ∥ w t ∥ = 1 for all t ≥ t ∗ +1 and thus,

where the last inequality follows by the triangle inequality, ∥ ˆ c 1: t ∥ ≤ ∥ ˆ c 1: t -1 ∥ + ∥ ˆ c t ∥ . As a result,

where the last inequality follows by the fact that t ≥ t ∗ +1 and thus ∥ ˆ c 1: t ∥ ≥ 1 4 ˆ σ 1: t . Finally we get that,

We conclude the section with the proof of Lemma 4.1. We restate the theorem so as to be consistent with the notation of the section.

Lemma 4.1. Let w 1 , . . . , w T ∈ B (0 , 1) the sequence of points produced by Adaptive FTRL given as input the cost-vector sequence ˆ c 1 , . . . , ˆ c T . Let t 1 denote the maximum index such that

Then the followig holds,

Proof. Let t 2 denotes the maximum index such that ∑ t s =1 ˆ c ⊤ s w s ≤ -∥ ˆ c 1: t ∥ and t 3 the maximum index such that ˆ σ 1: t ≤ 17 (as in the proof of Lemma C.3). We consider the following 3 mutually exclusive case,

· t 2 ≥ max( t 1 , t 3 ) :

Due to the fact that t 2 ≥ t 1 we have that for any t ≥ t 2 +1 ,

where the first inequality follows by the definition of t 2 while the second by the definition of t 1 , ∑ t s =1 ˆ c ⊤ s w s ≤ -1 4 ˆ σ 1: t for all t ≥ t 1 + 1 . Since t 2 ≥ t 3 we additionally get that ˆ σ 1: t ≥ 17 for all t ≥ t 2 +1 . As a result,

Meaning that the conditions of Lemma C.3 are satisfied for all t ≥ t 2 +1 and thus

Up next we analyze the regret of Adaptive FTRL ,

where Inequality (9) follows by the definition of t 2 i.e. ∑ t 2 t =1 ˆ c ⊤ t x t ≤ -∥ ˆ c 1: t 2 ∥ . Inequality (10) follows by the fact that ∑ t 2 t =1 ˆ c ⊤ t x t 2 +1 ≥ -ˆ c 1: t 2 . Inequality (11) follows by applying Lemma C.1 for S := t 2 . Equality (12) and Inequality (13) follow by Equation 23.

t 1 ≥ max( t 2 , t 3 ) : By using the exact same arguments as above we can establish that

Using the exact same arguments as above we conclude that

where the first inequality follows by applying Lemma C.2 for T = t 1 and the second by repeating Inequalities (11) -(15) .

t 2 ≥ max( t 1 , t 3 ) : By the exact same arguments as in the previous case,

where the last inequality follows by the fact that σ 1: t 3 ≤ 17 (definition of t 3 ).

As a result, we have established that in any case,

C.2 Proof of Lemma 4.3

To simplify notation we summarize the Step 7 of Algorithm 4 in Algorithm 6.

Algorithm 6 OGD with Shrinking Domain

2: for t = 1 . . . T do

3: The learner plays p t ∈ D t

4: The adversary selects z t and σ t ≤ 1 .

5: The learner updates the interval D t ⊆ [0 , 1] as follows,

and its actions p t +1 ∈ [0 , 1] as follows

6: end for

Remark C.4. We remark that Algorithm 6 corresponds to Step 7 of Algorithm 4 once

Definition C.5. Asequence q 1 , . . . , q T ∈ [0 , 1] is valid in hindsight if and only if there exists a round t ∗ ∈ [ T ] and a δ ∈ [0 , 1] such that the following hold,

q t = δ · I [ t ≤ t 1 ] ( q t = δ for all t ≤ t ∗ and q t = 0 for all t ≥ t ∗ +1 ).

At the switching point t ∗ ∈ [ T ] ,

In Theorem C.6 we present the payoff guarantees of Algorithm 6 with respect to any sequence q t that is valid in hindsight.

Theorem C.6 ([5]) . Let p 1 , . . . , p T ∈ [0 , 1] a sequence of points produced by Algorithm 6 given as input the sequence ( z 1 , σ 1 ) , . . . , ( z T , σ T ) . In case z 2 t ≤ 4 σ t for all rounds t ∈ [ T ] then for any valid in hindsight sequence q 1 , . . . , q T ∈ [0 , 1] (Definition C.5) the following holds,

We conclude the section with the proof of Lemma 4.3.

Lemma 4.3. Let the sequence of cost-vector c 1 , . . . , c T given to Algorithm 4 and the produced sequences x 1 , . . . , x t ∈ ∆ n and p 1 , . . . , p T ∈ (0 , 1) . Additionally let t 1 denote the maximum time such that

and consider the sequence q t := I [ t ≤ t 1 ] · ( 20 / √ 400 + ∑ t 1 t =1 ∥ c t + c t -1 ∥ 2 2 ) . Then the following holds,

Proof. The sequence q t is a valid sequence with switching point t ∗ := t 1 and

Now the sequence p t produced by Algorithm 4 in Steps 7 and Steps 8 can be viewed as the output of Algorithm 6 with of the input sequence z t := ( c t + c t -1 ) ⊤ · ( w t + c t -1 ) and σ t := ∥ c t + c t -1 ∥ 2 . Since

Lemma 4.3 follows by Theorem C.6.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors discuss a model of no-regret learning that simulates the use of *alternating* regret updates in a two-player game. In this model, the authors show that the classical lower bound of $\\Theta(\\sqrt{T})$ regret does not hold, and in fact, it is possible to achieve $\\tilde O(T^{1/3})$ regret.\n\nSTRENGTHS:\nI enjoyed reading this paper. It addresses a gap that has been observed empirically many times (the fact that alternating regret updates improves practical performance), and does so in a theoretically very nice way. The paper is also well written and easy to understand. I vote to accept.\n\nWEAKNESSES:\nI see no major issues, and I have only a few relatively minor questions; see the next section.\n\nQUESTIONS:\n1. Maybe some typos?\n    * Remark 2.5: should that refer to Alg 4 instead of Alg 2? The stated bound for Alg 2 on $\\Delta^2$ is $\\tilde O(T^{1/3})$. \n    * Algorithm 3, line 6 ($\\pi^t(s) = \\dots$): should there be a factor of $\\gamma$ on the first $2q(s, \\pi^t, c^{t-1})^\\top x$, to match it with Alg 2, Line 3? If not, ignore my Q2 below, as this is no longer an analogue of CFR, but in that case I would appreciate some intuition as to why $\\gamma$ is suddenly gone from that term.\n1. I think your algorithm for the n-dimensional simplex looks a lot like a version of CFR ([1]; see [2] for a more modern exposition) applied on the binary tree $T$ with Algorithm 2 as the local regret minimizer. This algorithm, then, generalizes immediately to extensive-form games. Does the analysis also generalize? If so, that'd be nice to state.\n1. It is quite surprising to me that there is a gap in the power of $T$ between the $\\ell_2$ ball and the simplex. Such a gap certainly doesn't exist in usual models of no-regret learning. Is there any intuition for this gap? Do you believe it to be fundamental/do you have a lower bound against achieving $\\tilde O(1)$ regret in the simplex case? For example, what happens if you use Alg 4 as your building-block regret minimizer on $\\Delta^2$ within Alg 3?\n\n\n[1] Martin Zinkevich, Michael Johanson, Michael Bowling, Carmelo Piccione. \"Regret Minimization in Games with Incomplete Information\", NeurIPS 2007.  \n[2] Gabriele Farina, Christian Kroer, Tuomas Sandholm. \"Online Convex Optimization for Sequential Decision Processes and Extensive-Form Games\", AAAI 2019\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper consider an online linear optimization (OLO) problem motivated by alternating game play in two-player games. Specifically, after the learner chooses $x^t$ and the adversary selects loss vector $c^t$, the learner suffers loss $(c^t + c^{t-1})x^t$ in the alternating OLO instead of $c^t x^t$ in the standard OLO model. The adversary in the alternating OLO model is weaker than that in the standard OLO model and the main results show that $o(\\sqrt{T})$ regret is possible for the following two cases: (1) $\\mathcal{O}( (\\log n)^{4/3} T^{1/3})$ regret for the $n$-dimensional simplex; (2) $O(\\log T)$ regret for a $\\ell_2$ ball constraints (and when the agent has only 2 actions).\n\nSTRENGTHS:\n1. The proposed alternating OLO model is interesting and has implication for game-play in two-player games. \n2. Although privous works show $o(\\sqrt{T})$ regret in some restricted cases such as unconstrained domain or when two players use the same algorithm, the main results in this paper hold for the n-dimensional simplex and against an adversary, which is stronger than privous works. The proofs of the main results are non-trivial. \n3. The paper is fairly well-written. The authors provided helpful intuition and high-level ideas in the main body, which makes the paper easy to follow.\n\nWEAKNESSES:\n1. The $o(\\sqrt{T})$ results for the alternating OLO model only hold for n-dimensional simplex and the ball constraints, but not general convex sets. \n2. As the authors pointed out, the current analysis is limited to the linear loss setting, and does not directly extend to general convex losses. \n3. There is no lower bound results. \n\nHowever, I would like to remark that the current results are still very interesting and are important first steps towars solving these open questions. \n\nSome minor comments:\n\n1. Line 108: \"...OGD with admits...\"\n2. ALG2-Line 3: check \"2\\gamma (c^{t-1})...\"\n3. Line 195: $x^t$ should be $y^t$?\n4. Check line 529-530\n5. Check the last inequality on page 22\n6. Missing squares in the inequality on line 609\n7. The references of inequalities on line 637-639, 643 are broken.\n\nQUESTIONS:\nThe results of the paper holds in the adversarial setting. In the game setting, the two players use the same algorithm and are not necessarily adversary to each other. I would like to ask whether better regret guarantees can be achieved in the game setting when the two players use the same algorithm e.g., Algorithm 2 in the paper. Such a result would lead to faster convergence to CCE in two-player games.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers online linear optimization where losses are given by $x_t^{\\top} (c_t + c_{t-1})$, where $c_t$ is the adversary’s choice at each round. The authors show that the standard $\\Omega(\\sqrt{T})$ lower bound no long holds, and give a $O(T^{1/3})$ regret algorithm over the simplex as well as a $O(\\rho \\log T)$ regret algorithm over a radius-$\\rho$ ball.\n\nSTRENGTHS:\nThe paper introduces a new setting for online learning with constrained adversaries, and gives two algorithms with new, non-trivial regret bounds. \nThe paper is somewhat easy to follow, albeit with some space allocation and presentation issues, and results appear largely correct (although the second algorithm is more difficult to evaluate due to reliance on prior work).\n\nWEAKNESSES:\nOn the whole, this strikes me as a paper whose core ideas have the potential to be quite compelling if sufficiently extended and contextualized, but which leaves many relevant questions underdiscussed and is not yet ready for publication. \n\nTo start, the connection between the costs $x_t^{\\top} (c_t + c_{t-1})$ and “alternating gameplay“ could be better motivated; in such turn-based games, why should we think of players as experiencing costs in each round regardless of whose turn it is? Why we should consider this as our cost model for such a game rather than those enabled by e.g. extensive-form games or Stackelberg games? Both of the latter are well-studied from a regret minimization perspective and should be discussed in related work. The formulation given seems more natural as perhaps an example of a more general framework (e.g. when the adversary’s movements are bounded each round) rather than a primary focus in its own right without further contextualization within the broader game theory literature. For example, Remark 1.2 regarding convergence to NE/CCE should probably be omitted unless there is further discussion as to why these solution concepts are appropriate targets for the “alternating” setup (vs. something like Stackelberg equilibria).\n\nTheorem 2.2 is well-known and does not need to be proved in the body; you could maybe state it as a proposition and discuss more informally, but in general, it is not particularly surprising that a specific construction for one setting fails in a more restricted setting, particularly when there are many other known settings where faster rates are obtainable against constrained adversaries (e.g. strongly convex losses).\n\nThe alternating setup considered by [WTP22] is a feature of the algorithmic setup and not directly comparable here. It might be more natural to discuss relations to the long line of work on fast convergence rates (e.g. https://arxiv.org/abs/1507.00407). Perhaps there is a deeper connection between fast rates and bounds on the per-round movement of the adversary, which appears in these works as well.\n\nFurther, the results are only given for special cases (simplex, ball), and no lower bounds are shown. It seems initially a bit strange that the rates for the simplex and the ball differ so dramatically. Why are the algorithm designs for each so different? The intuition behind the algorithms should be highlighted further, possibly in exchange for moving some proof details to the appendix. Is O(T^1/3) optimal for the simplex? Also, it is difficult to sanity-check the $O(\\log T)$ result over the ball for correctness; this result indeed seems surprising for such flexible adversaries, and the technical results should be explained more clearly in the body in order to highlight the primary difficulties and innovations.\n\nThroughout, there are also a number of typos and broken sentences (ex. the statement of “Q2”).\n\nQUESTIONS:\n- What are the barriers to proving fast-rate results for general convex bodies (e.g. in terms of ratio of inner/outer ball)?\n- Why is there such a gap between the ball and the simplex? \n- If log(T) regret is possible over the ball, it seems natural to expect it’d be possible over any convex body, which would then remove the need for a slower result on the simplex. \n- What is the intuition behind the tree design in the simplex algorithm?\n- Are there notable technical connections to fast-rates results for optimistic learning dynamics?\n\nLIMITATIONS:\nThe connection to concept of “alternation” in games isn’t particularly convincing, and seems like it should be generalizable to broader settings (e.g. movement-bounded adversaries). \n\nWhy only the simplex/ball and not convex sets? \n\nThroughout, there are a number of key omissions of important related work, as well as too much emphasis on restating well-known prior results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a variant of the usual online learning model with linear losses, whereby the environment is constrained to produce losses of a specific form. Such a form appears naturally when studying alternating no-regret dynamics in games. The authors establish that this restriction weakens the environment, in that the decision maker is now able (at least in certain domains) to attain unconditionally a regret bound of T^1/3, breaking the typical T^1/2 lower bound of regular regret minimization.\n\nSTRENGTHS:\nI believe this is a technically solid paper. While I have some reservations about the relevance of the model per se (that is, assuming adversarial losses of the specific form studied in the paper, which is justified by alternation), I think that taking the model as a given the result are interesting. The paper is also well organized and it is easy to follow the flow with a good understanding of the preliminaries on FTRL and online optimization.\n\nWEAKNESSES:\nMy only reservation about the paper is about the specific assumption on the losses in general adversarial settings. While the motivation of alternation is clear in two-player games using self-play, I would be curious if the authors could make shed light on what situations their model can model.\n\nThe reason why I think the two-player alternation connection is not super satisfying is that in two-player zero-sum games, optimistic (non-alternating) learning dynamics attain constant regret per player. Similarly, in multiplayer general sum games optimistic dynamics guarantee polylogarithmic regret per player when used in self-play. Given that optimism does not come at a computational cost, it is not obvious why, in practice, one would like to focus on alternation but not optimism.\n\nThat being said, I have a positive opinion of the technical ideas used in the paper.\n\nMinor comments:\nThe paper would benefit from a pass of proofreading, as it contains a few typos. A few examples:\n- L294: I assume there is a \"beyond\" missing in the sentence?\n- Display after L170: the dots should probably be removed for consistency\n- L189: the symbol \\mapsto should arguably be a \\to in this case, at least according to what I consider standard notation (see also https://math.stackexchange.com/questions/473247/difference-of-mapsto-and-right-arrow).\n- Algorithm 3 italicizes keywords such as \"select\", while Algorithms 1 and 2 bold them.\n- L10: I believe it should be \"regardless _of_ the strategies\", but please double check with a native speaker\n\nQUESTIONS:\nCould you please comment on my reservation above?\n\nAlso, I was curious to hear any thoughts you'd like to share about generalizing the algorithm to general convex and compact domains.\n\nLIMITATIONS:\nNo negative societal impact concerns", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}