Title: Online Learning under Adversarial Nonlinear Constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: online learning

online convex optimization

constrained optimization

adversarial nonlinear constraints

constraint violation oracle
Abstract: In many applications, learning systems are required to process continuous non-stationary data streams.
We study this problem in an online learning framework and propose an algorithm that can deal with adversarial time-varying and nonlinear constraints.
As we show in our work, the algorithm called Constraint Violation Velocity Projection (CVV-Pro) achieves $\sqrt{T}$ regret and converges to the feasible set at a rate of $1/\sqrt{T}$, despite the fact that the feasible set is slowly time-varying and a priori unknown to the learner. 
CVV-Pro only relies on local sparse linear approximations of the feasible set and therefore avoids optimizing over the entire set at each iteration, which is in sharp contrast to projected gradients or Frank-Wolfe methods. 
We also empirically evaluate our algorithm on two-player games, where the players are subjected to a shared constraint.

1

Abstract

In many applications, learning systems are required to process continuous nonstationary data streams. We study this problem in an online learning framework and propose an algorithm that can deal with adversarial time-varying and nonlinear constraints. As we show in our work, the algorithm called Constraint Violation Velocity Projection (CVV-Pro) achieves √ T regret and converges to the feasible set at a rate of 1 / √ T , despite the fact that the feasible set is slowly time-varying and a priori unknown to the learner. CVV-Pro only relies on local sparse linear approximations of the feasible set and therefore avoids optimizing over the entire set at each iteration, which is in sharp contrast to projected gradients or FrankWolfe methods. We also empirically evaluate our algorithm on two-player games, where the players are subjected to a shared constraint.

1 Introduction

Today's machine learning systems are able to combine computation, data, and algorithms at unprecedented scales, which opens up new and exciting avenues in many domains, such as computer vision, computer graphics, speech and text recognition, and robotics [Jordan and Mitchell, 2015]. One of the leading principles that has enabled this progress is the focus on relatively simple pattern recognition and empirical risk minimization approaches, which mostly rely on offline gradient-based optimization and stipulate that the training, validation, and test data are independent and identically distributed.

Somewhat overlooked in these developments is the role of non-stationarity and constraints [Jordan, 2019]. Indeed, emerging machine learning problems involve decision-making in the real world, which typically includes interactions with physical, social, or biological systems. These systems are not only time varying and affected by past interactions, but their behavior is often characterized via fundamental constraints. Examples include cyber-physical systems where constraints are imposed by the laws of physics, multi-agent systems that are subjected to a shared resource constraint, or a reinforcement learning agent that is subjected to safety and reliability constraints. In particular, in their seminal work Auer et al. [2002] gave a reduction for the multi-arm bandit setting to the full information online optimization setting, by employing the multiplicative weights framework [Littlestone and Warmuth, 1994]. This classical reduction was recently extended by Sun et al. [2017] to the contextual bandit setting with sequential (time-varying) risk constraints.

This motivates our work, which is in line with a recent trend in the machine learning community towards online learning, adaptive decision-making, and online optimization. More precisely, we study an online problem with slowly time-varying constraints, governed by the following interaction protocol (see Assumption 1.2). In each time step t , the learner commits a decision x t and then in addition to a loss value f t ( x t ) with its gradient ∇ f t ( x t ) receives partial information about the current feasible set C t := { x ∈ R n | g t ( x ) ≥ 0 } , where the constraint function g t ( x ) is defined as

[ g t, 1 ( x ) , . . . , g t,m ( x )] . The quality of the learner's decision making is measured, for every T ≥ 1 , by comparing to the best decision in hindsight x ⋆ T ∈ arg min x ∈C T ∑ T t =1 f t ( x ) , that is,

which will be shown to be bounded by O ( √ T ) for our algorithm. The functions f t and g t are restricted to f t ∈ F and g t ∈ G (as defined in Assumption 1.1) and c > 0 is an explicit constant.

It is important to note that our performance objective (1) is symmetric in the sense that the constraint x ∈ C T applies to both the learner's decision x T and the benchmark x ⋆ T . This contrasts prior work by Neely and Yu [2017], Yu et al. [2017], Sun et al. [2017], Chen et al. [2017], Cao and Liu [2019] and Liu et al. [2022], where a different notion of constraint violation ∑ T t =1 g t ( x t ) ≥ -c 0 √ T is used for the learner, while either a single benchmark x ⋆ 1: T satisfies g t ( x ⋆ 1: T ) ≥ 0 for all t ∈ { 1 , . . . , T } or multiple benchmarks { x ′ t } T t =1 satisfy x ′ t ∈ arg min x ∈C t f t ( x ) . Unlike (1), different requirements are imposed on the learner and the benchmark(s), which leads to asymmetric regret formulations: ∑ T t =1 f t ( x t ) -∑ T t =1 f t ( x ⋆ 1: T ) and ∑ T t =1 f t ( x t ) -∑ T t =1 f t ( x ′ t ) , respectively. Furthermore, as our bound g T ( x T ) ≥ -c/ √ T applies for all T ≥ 1 , it implies the cumulative constraint violation bound in Neely and Yu [2017] up to a constant factor ∑ T t =1 g t ( x t ) ≥ -c ∑ T t =1 1 / √ t ≥ -2 c √ T .

Even more intriguing is the fact that our algorithm is unaware of the feasible sets a-priori, and obtains, at each iteration, only a local sparse approximation of C t based on the first-order information of the violated constraints. The indices of all violated constraints at x t will be captured by the index set I ( x t ) := { i ∈ { 1 , . . . , m } | g t,i ( x t ) ≤ 0 } , while G ( x t ) := [ ∇ g t,i ( x t )] i ∈ I ( x t ) denotes the matrix whose columns store the corresponding gradients. In order to guarantee a regret of O ( √ T ) in (1) we require the following assumptions.

Assumption 1.1. There exist R,L F , L G > 0 : 1) F is a class of convex functions, where every f ∈ F satisfies ||∇ f ( x ) || ≤ L F , ∀ x ∈ B 4 R , with || · || the ℓ 2 norm and B R the hypersphere of radius R centered at the origin; 2) G is a class of concave β G -smooth functions, where every g satisfies ||∇ g ( x ) || ≤ L G , ∀ x ∈ B 4 R ; 3) The feasible set C t is non-empty and contained in B R for all t .

We note that these assumptions are standard in online optimization [Hazan, 2016, Ch. 3]. The learner's task is nontrivial even in the case where the feasible set is time invariant. If the feasible set is time varying, additional assumptions are required that restrict the amount that the feasible set is allowed to change. These two assumptions, see Part 2 i) and ii) below, are described by the following interaction protocol between the learner and the environment:

Assumption 1.2. (Interaction protocol) At each time step t ∈ { 1 , . . . , T } :

the learner chooses x t ;

the environment chooses f t ∈ F and g t ∈ G such that i) || g t ( x ) -g t -1 ( x ) || ∞ = O (1 /t ) , uniformly for all x ∈ B 4 R , with || · || ∞ the ℓ ∞ norm, and ii) C t is contained in Q t := ∩ t -1 ℓ =0 S ℓ , where S t := { x ∈ R n | G ( x t ) ⊤ ( x -x t ) ≥ 0 } is a cone centered at x t for t ≥ 1 and S 0 = R n (the situation is illustrated in Figure 1, more details are presented in Appendix A);

the environment reveals to the learner partial information on cost f t ( x t ) , ∇ f t ( x t ) and all violated constraints g t,i ( x t ) , ∇ g t,i ( x t ) for i ∈ I ( x t ) .

The requirements i) || g t -g t -1 || ∞ = O (1 /t ) ; and ii) C t ⊂ Q t restrict the feasible sets that the environment can choose. We note that despite the fact that || g t -g t -1 || ∞ = O (1 /t ) , || g 1 -g t || ∞ = Θ(ln( t )) , which means that the sequence of functions g t that defines C t does not converge in general. As a result, C t may evolve in such a way that the initial iterates x 1 , x 2 , . . . , x t 0 achieve a large cost compared to min x ∈C T ∑ T t =1 f t ( x ) , as these are constrained by the sets C 1 , C 2 , . . . , C t 0 , which may be far away from C T . The second requirement ii) C t ⊂ Q t avoids this situation and is therefore key for obtaining an O ( √ T ) regret.

Our setup differs from traditional online convex optimization [Zinkevich, 2003] in the following two important ways:

i) The environment chooses not only the functions f t but also the nonlinear constraint functions g t , ii) even if g t is time-invariant, i.e., g t = g for all t the learner has only access to local information about the feasible set.

That is, the information about the feasible set is only revealed piece-by-piece and needs to be acquired by the agent through repeated queries of a constraint violation oracle.

We propose an online algorithm that despite the lack of information about the feasible set, achieves O ( √ T ) regret, and will derive explicit non-asymptotic bounds for the regret and the convergence to C T . We thus conclude that our algorithm matches the performance of traditional online projected gradients or Frank-Wolfe schemes, while requiring substantially less information about the feasible set and allowing it to be time-varying. Perhaps equally important is the fact that instead of performing projections onto the full feasible set at each iteration, our algorithm only optimizes over a local sparse linear approximation. If constraints are nonlinear, which includes norm-constraints or constraints on the eigenvalues of a matrix, optimizing over the full feasible set at each iteration can be computationally challenging.

1.1 Related Work

Online learning has its roots in online or recursive implementations of algorithms, where due to the piece-by-piece availability of data, algorithms are often analyzed in a non i.i.d. setting. A central algorithm is the multiplicative weights scheme [Freud and Schapire, 1997], where a decider repeatedly chooses between a finite or countable number of options with the aim of minimizing regret. This online learning model not only offers a unifying framework for many classical algorithms [Blum, 1998], but represents a starting point for online convex optimization Hazan [2016], and adversarial bandits [Lattimore and Szepesvári, 2020]. Our approach extends this line of work by allowing the environment to not only choose the objective functions f t , but also the constraints g t . Due to the fact that our learner only obtains local information about the feasible set, our work is somewhat related to Levy and Krause [2019], Lu et al. [2023], Garber and Kretzu [2022], Mhammedi [2022], where the aim is to reduce the computational effort of performing online projected gradient steps or Frank-Wolfe updates. More precisely, Levy and Krause [2019] propose an algorithm that directly approximates projections, while requiring multiple queries of the constraint functions and their gradients. A slightly different constraint violation oracle is assumed in Garber and Kretzu [2022], where the learner can query separating hyperplanes between a given infeasible point and the feasible set. Algorithmically, both Garber and Kretzu [2022] and Levy and Krause [2019] depart from online gradient descent, where the latter computes projections via an approximate Frank-Wolf-type scheme. An alternative is provided by Mhammedi [2022] and Lu et al. [2023], where optimizations over the entire feasible set are simplified by querying only a set membership oracle based on the Minkowski functional. While our approach also avoids projections or optimizations over the entire feasible set, we introduce a different constraint violation oracle that returns a local sparse linear approximation of the feasible set. We call the constraint violation oracle only once every iteration and do not require a two-step procedure that involves multiple oracle calls. In addition, we also allow for adversarial time-varying constraints.

In addition, there has been important recent work that developed online optimization algorithms with constraints. In contrast to the primal formulation of our algorithm, these works are based on primal-dual formulations, where the algorithm is required to satisfy constraints on average, so called long-term constraints. The research can be divided into two lines of work Mahdavi et al. [2012], Jenatton et al. [2016], Yu and Neely [2020] and Yuan and Lamperski [2018], Yi et al. [2021] that use a set of weaker and stricter definitions for constraint violations and investigate time-invariant constraints, which contrasts our formulation that includes time-varying constraints. A third line of work by Mannor et al. [2009], Chen et al. [2017], Neely and Yu [2017], Yu et al. [2017], Sun et al. [2017], Cao and Liu [2019], Liu et al. [2022] focuses on time-varying constraints, where, however, the following weaker notion of constraint violation is used: ∑ T t =1 g t ( x t ) ≥ -c √ T , where t refers to time and x t to the learner's decision. This metric allows constraint violations for many iterations, as long as these are compensated by strictly feasible constraints (in the worst case even with a single feasible constraint with a large margin). In contrast, our algorithm satisfies g t ( x t ) ≥ -c/ √ t for all iterations t ∈ { 1 , . . . , T } , where c is an explicit constant independent of the dimension of the decision variable and the number of constraints. This means that we can explicitly bound the constraint violation at every iteration, whereas infeasible and strictly feasible iterates cannot compensate each other.

An important distinction to Neely and Yu [2017] is given by our performance metric (see also the discussion in Neely and Yu [2017] and Liu et al. [2022]). On the one hand, the work by Chen et al. [2017], Cao and Liu [2019], Liu et al. [2022] use ∑ T t =1 f t ( x t ) -∑ T t =1 f t ( x ′ t ) as a performance measure, where the iterates x t are required to satisfy ∑ T t =1 g t ( x t ) ≥ -c √ T and the optimal solutions x ′ t satisfy x ′ t ∈ arg min x ∈C t f t ( x ) . On the other hand, the work by Neely and Yu [2017], Yu et al. [2017], Sun et al. [2017] use ∑ T t =1 f t ( x t ) -∑ T t =1 f t ( x ⋆ 1: T ) as a performance measure, where the iterates x t are required to satisfy ∑ T t =1 g t ( x t ) ≥ -c √ T and the optimal solution x ⋆ 1: T satisfies g t ( x ⋆ 1: T ) ≥ 0 for all t ∈ { 1 , . . . , T } . This leads to a major asymmetry in the way regret is measured: while the iterates of the online algorithm only need to satisfy a cumulative measure of constraint violation, the benchmark x ⋆ 1: T , which represents the best fixed decision in hindsight, is required to satisfy all constraints g t ( x ⋆ 1: T ) ≥ 0 for t = { 1 , . . . , T } . In contrast, the performance metric introduced in (1) is symmetric and imposes the same constraint x ∈ C T (approximately) on the learner's decision x T and (exactly) on the benchmark x ∗ T . These features make our algorithm a valuable addition to the algorithmic toolkit of online constrained optimization.

Castiglioni et al. [2022] studied the following asymmetric setting with adversarial environment, benchmark x ⋆ T belonging to arg min x ∈X ∑ T t =1 f t ( x ) subject to 1 T ∑ T t =1 g t ( x ) ≥ 0 , online iterates x t satisfying 1 T ∑ T t =1 g t ( x t ) ≥ -O (1 / √ T ) , and regret ∑ T t =1 f t ( x t ) -∑ T t =1 f t ( x ⋆ T ) ≤ O ( √ T ) . Their benchmark and regret formulation can be obtained as a special case of our formulation with time-averaged constraints, that is, when our g T ( x ) is chosen as 1 T ∑ T t =1 g t ( x ) . In contrast, our iterate x T satisfies 1 T ∑ T t =1 g t ( x T ) ≥ -O (1 / √ T ) , a constraint that is asymptotically the same as the one satisfied by the benchmark x ⋆ T . We further note that they introduced a parameter ρ = sup x ∈X min t ∈ [ T ] min i ∈ [ m ] g t,i ( x ) , which is required to be positive and known to the algorithm for achieving O ( √ T ) regret. Notably ρ > 0 implies that the intersection of all feasible sets is non-empty, which is a strong assumption (as is knowledge about the parameter ρ ). In our formulation with time-averaged constraints, Assumption 1.2 reduces to the requirement that the feasible set C t belongs to a polyhedral intersection Q t , which does not require a non-emtpy intersection of all C t (has a geometrical interpretation and the assumption || g t -g t -1 || ∞ = O (1 /t ) is automatically satisfied). Thus, there are situations, where the regret bound from Castiglioni et al. [2022] becomes vacuous (for ρ = 0 ), while our method still provably achieves O ( √ T ) regret. Additional differences are that Castiglioni et al. [2022] considers primal-dual methods and assumes that all constraints are revealed after every iteration, whereas our method is primal-only and has only partial information about all violated constraints. The latter point reduces computation and simplifies projections onto the velocity polyhedron, but requires a nontrivial inductive argument for establishing O ( √ T ) regret.

Other relevant related studies have investigated online learning problems with supply/budget constraints. In these settings, the decision maker must choose a sequence of actions that maximizes their expected reward while ensuring that a set of resource constraints are not violated. The process terminates either after a pre-specified time horizon has been reached or when the total consumption of some resource exceeds its budget. Badanidiyuru et al. [2018] introduced the bandits with knapsacks framework, which considers bandit feedback, stochastic objective and constraint functions. They proposed an optimal algorithm for this problem, which was later improved by Agrawal and Devanur [2014, 2019] and Immorlica et al. [2022]. Immorlica et al. [2022] introduced the adversarial bandits with knapsacks setting and showed that an appropriate benchmark for this setting is the best fixed distribution over arms. Since no-regret is no longer possible under this benchmark, they provide noα -regret guarantees for their algorithm.

An important special case of our online learning model arises when the environment is represented by an adversarial player that competes with the learner. This corresponds to a repeated generalized Nash game due to the constraint that couples the decisions of the learner and its adversary. If the adversary plays best response, the resulting equilibria are characterized by quasi-variational inequalities [Facchinei and Kanzow, 2007] and there has been important recent work, for example by Jordan et al. [2023], Kim [2021], Facchinei and Kanzow [2010] that proposes different gradient and penalty methods for solving these inequalities. Our approach adopts a different perspective, rooted in online learning, which allows us to derive non-asymptotic convergence results for a first-order gradient-based algorithm that can be implemented in a straightforward manner. Our approach is also inspired by the recent work of Muehlebach and Jordan [2022], who propose a similar algorithm for the offline setting.

1.2 Main Contributions

We give an online optimization scheme under unknown non-linear constraints that achieves an optimal O ( √ T ) regret and converges to the latest feasible set at a rate of O (1 / √ T ) . There are two variants of our problem formulation: The first deals with situations where constraints are unknown but fixed, the second allows constraints to be chosen in a time-varying and adversarial manner.

Our algorithm, named Constraint Violation Velocity Projection (CVV-Pro), has the following features:

It assumes access to a new type of oracle, which on input x t , returns partial information on all currently violated constraints. Namely, the value g t,i ( x t ) and the gradient ∇ g t,i ( x t ) for all i ∈ I ( x t ) .

It projects an adversarially generated negative cost gradient -∇ f t ( x t ) onto a velocity polyhedron V α ( x t ) := { v ∈ R n | [ ∇ g i ( x t )] ⊤ v ≥ -αg i ( x t ) , ∀ i ∈ I ( x t ) } . Due to the linear and local structure of V α ( x t ) , the projection can be computed efficiently.

In contrast to standard online methods that project in each round a candidate decision onto the feasible set, our method trades off feasibility for efficiency. In particular, it produces a sequence of decisions that converges at a rate of O (1 / √ T ) to the latest feasible set.

Our method handles time-varying adversarial constraints g t , provided a decreasing rate of change || g t +1 -g t || ∞ ≤ O (1 /t ) and that each feasible set C t belongs to Q t (see Assumption 1.2). As we show in Section 3.1, an important special case where the assumption of decreasing rate of change is satisfied is given by g t = 1 t ∑ t j =1 ˜ g j , i.e., when g t represents an average of constraints ˜ g t over time.

1.3 Outline

Section 2 describes our algorithm and considers the situation where g t is time invariant. This sets the stage for our main results in Section 3 that provide regret guarantees for our new online convex optimization setting with non-stationary, nonlinear, and unknown constraints. An important and interesting application of our algorithm are generalized Nash equilibrium problems, as will be illustrated with a numerical experiment in Section 4. The experiment will also highlight that the numerical results agree with the theoretical predictions.

2 Online learning under unknown, time-invariant, and nonlinear constraints

2.1 Online Gradient Descent

Online gradient descent [Hazan, 2016, Ch. 3.1] is a classical and perhaps the simplest algorithm that achieves optimal O ( √ T ) regret for the setting of a compact, convex, time-invariant, and a priori known feasible set. It consists of the following two operations: i) y t +1 = x t -η t ∇ f t ( x t ) takes a step from the previous point in the direction of the previous cost gradient; and ii) x t +1 = Proj C ( y t +1 ) projects y t +1 back to the feasible set C , as y t +1 may be infeasible.

In this section, we generalize the online gradient descent algorithm to the setting where the feasible set is unknown a priori and has to be learned through repeated queries of a constraint violation oracle that only reveals local information.

2.2 Overview

In Section 2.3, we present the pseudo code of our algorithm. In Section 2.4, we give a structural result showing that Algorithm 1 under Assumption 1.1 and a bounded iterate assumption guarantees an optimal O ( √ T ) regret and converges to the feasible set at a rate of O (1 / √ T ) . In Appendix E, we show that the bounded iterate assumption can be enforced algorithmically, by introducing an additional hypersphere constraint that attracts the sequence { x t } t ≥ 1 to a fixed compact set.

2.3 Constraint Violation Velocity Projection (CVV-Pro)

We present below the pseudocode of Algorithm 1 for a fixed horizon length T , as it is standard in the literature [Hazan, 2016]. However, we note that our algorithm is oblivious to the horizon length T , i.e., it can run for any number of iterations without knowing T a priori.

Algorithm 1 Constraint Violation Velocity Projection (CVV-Pro)

1:

Requirements: See Assumption 1.1

2: Input: α > 0

3: Initialization: Step sizes { η t = 1 α √ t } t ≥ 1

4: for t = 1 to T do

5: Play x t and observe :

6: cost information f t ( x t ) , ∇ f t ( x t ) and constraint information {( g i ( x t ) , ∇ g i ( x t ) ) } i ∈ I ( x t )

7: Construct the velocity polyhedron as follows:

8: Solve the velocity projection problem: v t = arg min v ∈ V α ( x t ) 1 2 ∥ v + ∇ f t ( x t ) ∥ 2

9: Update: x t +1 = x t + η t v t

10: end for

Let x ∈ C be an arbitrary decision. We show in Claim 2.2 that α ( x -x t ) ∈ V α ( x t ) . Hence, the velocity polyhedron V α ( x t ) is always non-empty and well defined.

2.4 Structural Result

Here, we show that Algorithm 1 under Assumption 1.1 and a bounded iterate assumption, guarantees an optimal O ( √ T ) regret and converges to the feasible set at a rate O (1 / √ T ) . The bounded iterate assumption will be removed subsequently, which however, will require a more complex analysis.

Theorem 2.1 (Structural) . Suppose Assumption 1.1 holds and in addition x t ∈ B R for all t ∈ { 1 , . . . , T } . Then, on input α = L F /R , Algorithm 1 with step sizes η t = 1 α √ t guarantees the following for all T ≥ 1 : √

2.5 Proof Sketch of Theorem 2.1

Our analysis establishes, in two steps, an important geometric property that connects the convex costs and the concave constraints via the velocity polyhedron V α ( x t ) . More precisely, we show that the inner product -r ⊤ t ( x ⋆ T -x t ) ≤ 0 for all t ∈ { 1 , . . . , T } . This property will be crucial for deriving the regret and feasibility bounds.

In the first step, we leverage the constraints' concavity and show that the vector α ( x ⋆ T -x t ) belongs to the velocity polyhedron V α ( x t ) .

Claim 2.2. Suppose g i is concave for every i ∈ { 1 , . . . , m } . Then α ( x -x t ) ∈ V α ( x t ) for all x ∈ C . In addition, x t ̸∈ int( C ) implies [ ∇ g i ( x t )] ⊤ [ x -x t ] ≥ 0 for all x ∈ C .

Proof. Let x ∈ C be an arbitrary feasible decision, satisfying g i ( x ) ≥ 0 for all i ∈ { 1 , . . . , m } . Since g i is concave, we have g i ( x t ) + [ ∇ g i ( x t )] ⊤ [ x -x t ] ≥ g i ( x ) ≥ 0 and thus [ ∇ g i ( x t )] ⊤ [ x -x t ] ≥ -g i ( x t ) . The second conclusion follows by x t ̸∈ int( C ) , which implies g i ( x t ) ≤ 0 .

In the second step, we show that r ⊤ t ( x t -x ⋆ ) ≤ 0 , where r t = v t + ∇ f t ( x t ) is such that -r t belongs to the normal cone N V α ( x t ) ( v t ) of the velocity polyhedron V α ( x t ) evaluated at the projection v t .

Lemma 2.3 (Main) . Let v t be the projection of -∇ f t ( x t ) onto the polyhedron V α ( x t ) such that v t = r t -∇ f t ( x t ) ∈ V α ( x t ) , where -r t ∈ N V α ( x t ) ( v t ) . Then, -r ⊤ t ( x -x t ) ≤ 0 for all x ∈ C .

Proof. By definition, the normal cone N V α ( x t ) ( v t ) is given by { u ∈ R n | u ⊤ ( v -v t ) ≤ 0 , ∀ v ∈ V α ( x t ) } . Then, by construction -r t ∈ N V α ( x t ) ( v t ) and thus it holds for every v ∈ V α ( x t ) that -r ⊤ t [ v -v t ] ≤ 0 . The proof proceeds by case distinction:

Case 1. Suppose x t is in the interior of C . Then, I ( x t ) = ∅ , which implies -∇ f t ( x t ) ∈ V α ( x t ) = R n and thus r t = 0 .

̸

Case 2. Suppose x t is on the boundary or outside of C , i.e., I ( x t ) = ∅ . By Claim 2.2, we have [ ∇ g i ( x t )] ⊤ [ x -x t ] ≥ 0 for all x ∈ C . By construction, v t ∈ V α ( x t ) and thus v ( x ) = v t + x -x t ∈ V α ( x t ) . The statement follows by applying v = v ( x ) to -r ⊤ t [ v -v t ] ≤ 0 .

Regret. To establish the first conclusion of Theorem 2.1 (regret), we combine the preceding geometric property with the analysis of online gradient descent. Since f t ∈ F is convex, we upper bound the regret in terms of the gradient of f t , namely ∑ T t =1 f t ( x t ) -f t ( x ⋆ T ) ≤ ∑ T t =1 [ ∇ f t ( x t )] ⊤ ( x t -x ⋆ T ) and then we show that the following inequality holds

Moreover, in Appendix D (see Lemma D.2), we upper bound the velocity ∥ v t ∥ ≤ α ∥ x ⋆ T -x t ∥ + 2 ∥∇ f t ( x t ) ∥ . Combining Assumption 1.1 and x t ∈ B R yields a uniform bound ∥ v t ∥ ≤ V α , where for α = L F /R we set V α := 4 L F . The desired regret follows by a telescoping argument and by convexity of the cost functions f t ∈ F .

Feasibility. For the second conclusion of Theorem 2.1 (convergence to the feasible set at a rate of 1 / √ T ), we develop a non-trivial inductive argument that proceeds in two steps. In Appendix D (see Claim D.6), we give a structural result that bounds the constraint functions from below. In particular, for every i ∈ I ( x t ) we have g i ( x t +1 ) ≥ (1 -αη t ) g i ( x t ) -η 2 t V 2 α β G and for every i ̸∈ I ( x t ) it holds that g i ( x t +1 ) ≥ -η t +1 V α [2 L G + V α β G /α ] .

Using an inductive argument, we establish in Appendix D (see Lemma D.5) the following lower bound: g i ( x t ) ≥ -cη t where c = 2 V α ( L G + V α β G /α ) . Choosing α = L F /R implies that V α = 4 L F . Then, the desired convergence rate to the feasible set follows for the step size η t = 1 α √ t , since

3 Online Learning under Adversarial Nonlinear Constraints

3.1 Problem Formulation

In this section, we consider an online optimization problem with adversarially generated time-varying constraints. More precisely, at each time step t , the learner receives partial information on the current cost f t and feasible set C t , and seeks to minimize (1). To make this problem well posed, we restrict the environment such that each feasible set C t is contained in Q t (see Section 1) and the rate of change between consecutive time-varying constraints decreases over time. We quantify a sufficient rate of decay with the following assumption.

Assumption 3.1 (TVC Decay Rate) . We assume that the adversarially generated sequence { g t } t ≥ 1 of time-varying constraints is such that for every x ∈ B 4 R and all t ≥ 1 , the following holds ∥ g t +1 ( x ) -g t ( x ) ∥ ∞ ≤ 98 t +16 [ L G R +3 β G ] R 2 .

We note that Assumption 3.1 essentially only requires ∥ g t +1 ( x ) -g t ( x ) ∥ ∞ ≤ O (1 /t ) , as R can be chosen large enough such that the bound is satisfied. Of course, R will appear in our regret and feasibility bounds, but it will not affect the dependence on t or T (up to constant factors).

An important special case where Assumption 3.1 is satisfied, is summarized in the following Lemma. The proof is included in Appendix F (see Lemma F.7 and Lemma F.8).

Lemma 3.2. Suppose the functions ˜ g t,i satisfy Assumption 1.1 and in addition there is a decision x t,i ∈ B R such that ˜ g t,i ( x t,i ) = 0 for every t ≥ 1 and i ∈ { 1 , . . . , m } . Then the time-averaged constraints g t,i ( x ) := 1 t ∑ t ℓ =1 ˜ g ℓ,i ( x ) satisfy Assumption 1.1 and Assumption 3.1.

3.2 Velocity Projection with Attractive Hypersphere Constraint

We show in Appendix E that the second assumption in Theorem 2.1, namely, ' x t ∈ B R for all t ≥ 1 ' can be enforced algorithmically. We achieve this in two steps.

1) Algorithmically, we introduce an additional hypersphere constraint g m +1 ( x t ) = 1 2 [ R 2 -∥ x t ∥ 2 ] that attracts the decision sequence { x t } t ≥ 1 to a hypersphere B R and guarantees that it always stays inside a hypersphere B 4 R with a slightly larger radius.

More precisely, we augment the velocity polyhedron in Step 3 of Algorithm 1 as follows: V ′ α ( x t ) = V α ( x t ) if ∥ x ∥ ≤ R , otherwise

2) Analytically, we give a refined inductive argument in Appendix F (see Lemma E.5), showing that g m +1 ( x t ) ≥ -27 R 2 / √ t +15 , ∥ x t ∥ ≤ 4 R and ∥ v t ∥ ≤ 7 L F , for all t ≥ 1 .

3.3 Main Contribution

Our main contribution is to show that Algorithm 1 with the augmented velocity polyhedron V ′ α ( x t ) , achieves optimal O ( √ T ) regret and satisfies g T ( x T ) ≥ -O (1 / √ T ) convergence feasibility rate. Due to space limitations, we defer the proof to Appendix F.

Theorem 3.3 (Time-Varying Constraints) . Suppose the functions { f t , g t } t ≥ 1 satisfy Assumptions 1.1, 1.2 and 3.1. Then, on input R,L F > 0 and x 1 ∈ B R , Algorithm 1 applied with α = L F /R , augmented velocity polyhedron V ′ α ( · ) and step sizes η t = 1 α √ t +15 guarantees the following for all T ≥ 1 :

Our regret analysis in Theorem 3.3 builds upon the following key structural result that generalizes Lemma 2.3 to time-varying constraints. In particular, in Appendix F (see Lemma F.3), we show that given the feasible set C T ⊂ Q T , it holds for every x ∈ C T that -r ⊤ t ( x -x t ) ≤ 0 for all t ∈ { 1 , . . . , T } . As a result, a similar argument as in (2) shows that the regret is bounded by O ( √ T ) .

Moreover, we note that the linear and quadratic dependence on R in Theorem 3.3 is consistent in length units. Let the radius R be of length units ℓ , then the Lipschitz constant L F , which can be viewed as the supremum over the ℓ 2 norm of the gradient is of 1 /ℓ units, and the β G smoothness constant (associated with Hessian) is of 1 /ℓ 2 units. This means that the regret bound in Theorem 3.3 has the same units as f t , while the feasibility bound has the same units as g t .

4 Simulation examples

Two-player games with shared resources are an excellent example for demonstrating the effectiveness and importance of our online learning framework. We apply our algorithm and show numerical experiments that support our theoretical findings.

We choose random instances of a two player game with linear utility and constraints. In particular, we consider the following optimization problem

where △ n = { x ∈ R n | ∑ n i =1 x i = 1 , x ≥ 0 } is the probability simplex. Each component of the utility matrix A ∈ R n × n is sampled from the normal distribution and the constraint matrices C x , C y ∈ [0 , 1] m × n have each of their components sampled uniformly at random from [0 , 1] .

(a)

(b)

0

1

,

000

2

,

000

3

,

000

4

,

000

0

50

100

rounds

regret

regret

2

√

t

10

0

10

1

10

2

10

3

10

4

10

-

2

10

-

1

10

0

rounds

two-norm of error

|

x

t

-

x

∞

|

|

y

t

-

y

∞

|

2

/

√

t

(c)

10

0

10

1

10

2

10

3

10

4

10

-

4

10

-

2

10

0

rounds

max avg violation

-

min

i

∈

I

(

x t

)

g

t,i

(

x

t

)

1

/

√

t

4.1 Online Formulation

The problem in (3) can be modeled with our online learning framework (1) by choosing costs f t ( x ) := x ⊤ Ay t and time-averaged resource constraints g T ( x ) := 1 T ∑ T t =1 ˜ g t ( x ) , where the function ˜ g t ( x ) := 1 -C x x -C y y t . Thus, the constraint in (3) is included as an average over the past iterations of y t . The strategy for choosing y t will be described below and, as we will see, the average of y t over the past iterations converges. This ensures that the feasible set C t (defined in (1)) is slowly time-varying, while the averages of x t and y t over past iterates converge to equilibria in (3). Further, by a refined version of Lemma 3.2 (see Lemma F.6 in Appendix F), the time-averaged constraints g T ( x ) satisfy Assumption 3.1.

In each iteration, Algorithm 1 seeks to minimize the online problem and commits to a decision x t . The adversary computes the best response ˆ y t with respect to the decision x t by solving arg max y ∈△ n x ⊤ t Ay . To make the dynamics more interesting, the adversary then commits with probability 0.8 to ˆ y t and with probability 0.2 to a random decision ξ t , i.e., y t = 0 . 8ˆ y t +0 . 2 ξ t where the random variable ξ t is sampled uniformly at random from the probability simplex △ n .

As both players optimize over the probability simplex ( x, y ∈ △ n ), the sequence of decisions { x t } t ≥ 1 is automatically bounded. Thus, we can apply Theorem 3.3 with the original velocity polyhedron, as discussed in Appendix E. We implemented our algorithm with η t = 1 / ( α √ t ) and α = 100 .

4.2 Experimental Results

We report results from numerical simulations with decision dimension n = 100 , m = 10 shared resource constraints, T = 4000 iterations, and five independently sampled instances of the two-player game. The learner's regret, depicted in Figure 3a, shows a clear correspondence with the theoretical prediction of O ( √ T ) . Figure 3b presents the maximal constraint violation -min i ∈ I ( x T ) 1 T ∑ T t =1 ˜ g t,i ( x T ) , which follows the predicted O (1 / √ T ) convergence rate. We also conclude from Figure 3c that the learner's averaged decisions x T = 1 T ∑ T t =1 x t converge at a rate of O (1 / √ T ) . Similarly, the averaged decisions y T of the adversary also converge at a rate of O (1 / √ T ) . We note that there is little variability in the results despite the different realizations of the matrices A,C x , C y .

Contrasting CVV-Pro and Online Gradient Descent In Appendix C, we show that our (CVV-Pro) algorithm outperforms the standard Online Gradient Descent algorithm in the two-player game from above. In particular, our algorithm achieves a lower regret and a runtime improvement of about 60% . Further, the percentage of violated constraints decreases rapidly and plateaus at 20% .

The amount of improvement in execution time is likely to be greater for higher-dimensional problems, where fewer constraints tend to be active at each iteration. Moreover, when the constraints are nonlinear, which includes ℓ p norm or spectral constraints, optimizing over the full feasible set can be computationally challenging. In contrast, the velocity projection step in CCV-Pro is always a convex quadratic program with linear constraints, regardless of the underlying feasible set.

5 Broader Impact

It is important to emphasize that our work is theoretical, and the main contribution is to design and analyze a novel algorithm that combines techniques from the seemingly distant fields of online convex optimization (online gradient descent) and non-smooth mechanics (velocity space, see Muehlebach and Jordan [2022]). Nevertheless, the list of potential applications includes, but is not limited to: adversarial contextual bandits with sequential risk constraints Sun et al. [2017], network resource allocation Chen et al. [2017], logistic regression Cao and Liu [2019], Liu et al. [2022], ridge regression and job scheduling Liu et al. [2022], two-player games with resource constraints (Section 4), system identification and optimal control (Appendix B).

6 Conclusion

We propose an online algorithm that, despite the lack of information about the feasible set, achieves O ( √ T ) regret. We further ensure convergence of violated constraint -min { g T ( x T ) , 0 } at a rate of O (1 / √ T ) and derive explicit constants for all our bounds that hold for all T ≥ 1 . We thus conclude that our algorithm matches the performance of traditional online projected gradients or Frank-Wolfe schemes, while requiring substantially less information about the feasible set and allowing the feasible set to be time-varying. Perhaps equally important is the fact that instead of performing projections onto the full feasible set at each iteration, our algorithm only optimizes over a local sparse linear approximation. We show the applicability of our algorithm in numeric simulations of random two-player games with shared resources.

References

Shipra Agrawal and Nikhil R. Devanur. Bandits with concave rewards and convex knapsacks. In ACM Conference on Economics and Computation , 2014.

Shipra Agrawal and Nikhil R. Devanur. Bandits with global convex constraints and objective. Operations Research , 67(5):1486-1502, 2019.

Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77, 2002.

Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. Journal of the ACM , 65(3):1-55, 2018.

Avrim L. Blum. On-line algorithms in machine learning. Online Algorithms: The State of the Art , pages 306-325, 1998.

Xuanyu Cao and K. J. Ray Liu. Online convex optimization with time-varying constraints and bandit feedback. IEEE Transactions on Automatic Control , 64(7):2665-2680, 2019.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, Giulia Romano, and Nicola Gatti. A unifying framework for online optimization with long-term constraints. In Advances in Neural Information Processing Systems 36 , 2022.

Tianyi Chen, Qing Ling, and Georgios B. Giannakis. An online convex optimization approach to proactive network resource allocation. IEEE Transactions on Signal Processing , 65(24):6350-6364, 2017.

Francisco Facchinei and Christian Kanzow. Generalized Nash equilibrium problems. A Quarterly Journal of Operations Research , 5:173-210, 2007.

Francisco Facchinei and Christian Kanzow. Penalty methods for the solution of generalized Nash equilibrium problems. SIAM Journal on Optimization , 20(5):2228-2253, 2010.

Yoav Freud and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997.

Dan Garber and Ben Kretzu. New projection-free algorithms for online convex optimization with adaptive regret guarantees. In Conference on Learning Theory , 2022.

Elad Hazan. Introduction to online convex optimization. Foundation and Trends in Optimization , 2 (3-4):157-325, 2016.

Nicole Immorlica, Karthik Abinav Sankararaman, Robert E. Schapire, and Aleksandrs Slivkins. Adversarial bandits with knapsacks. Journal of the ACM , 69(6):40:1-40:47, 2022.

Rodolphe Jenatton, Jim Huang, and Cedric Archambeau. Adaptive algorithms for online convex optimization with long-term constraints. In 33th International Conference on Machine Learning , 2016.

I. Jordan. Artificial intelligence - the revolution hasn't happened yet. Harvard Data Science Review , 1(1):1-8, 2019.

I. Jordan and T. M. Mitchell. Machine learning: Trends, perspectives, and prospects. Science , 349(6245):255-260, 2015.

Michael I. Jordan, Tianyi Lin, and Manolis Zampetakis. First-order algorithms for nonlinear generalized Nash equilibrium problems. Journal of Machine Learning Research , 24:38:1-38:46, 2023.

Jong Gwang Kim. Equilibrium computation of generalized nash games: A new Lagrangian-based approach. In 22nd Conference on Economics and Computation , 2021.

Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

Kfir Y. Levy and Andreas Krause. Projection free online learning over smooth sets. In 22nd International Conference on Artificial Intelligence and Statistics , 2019.

Nick Littlestone and Manfred K. Warmuth. The weighted majority algorithm. Information and Computation , 108(2):212-261, 1994.

Qingsong Liu, Wenfei Wu, Longbo Huang, and Zhixuan Fang. Simultaneously achieving sublinear regret and constraint violations for online convex optimization with time-varying constraints. Performance Evaluation , 49(3):4-5, 2022.

Zhou Lu, Nataly Brukhim, Paula Gradu, and Elad Hazan. Projection-free adaptive regret with membership oracles. In International Conference on Algorithmic Learning Theory , 2023.

Mehrdad Mahdavi, Rong Jin, and Tianbao Yang. Trading regret for efficiency: online convex optimization with long term constraints. Journal of Machine Learning Research , 13(81):25032528, 2012.

Shie Mannor, John N. Tsitsiklis, and Jia Yuan Yu. Online learning with sample path constraints. Journal of Machine Learning Research , 10(20):569-590, 2009.

Zakaria Mhammedi. Efficient projection-free online convex optimization with membership oracle. In Conference on Learning Theory , 2022.

Michael Muehlebach and Michael I. Jordan. On constraints in first-order optimization: A view from non-smooth dynamical systems. Journal of Machine Learning Research , 23(256):1-47, 2022.

Michael J. Neely and Hao Yu. Online convex optimization with time-varying constraints. arXiv preprint arXiv:1702.04783 , 2017.

Wen Sun, Debadeepta Dey, and Ashish Kapoor. Safety-aware algorithms for adversarial contextual bandit. In 34th International Conference on Machine Learning , 2017.

Xinlei Yi, Xiuxian Li, Tao Yang, Lihua Xie, Tianyou Chai, and Karl H. Johansson. Regret and cumulative constraint violation analysis for online convex optimization with long term constraints. In 38th International Conference on Machine Learning , 2021.

√

Hao Yu and Michael J. Neely. A low complexity algorithm with o( t) regret and O(1) constraint violations for online convex optimization with long term constraints. Journal of Machine Learning Research , 21(1):1-24, 2020.

Hao Yu, Michael Neely, and Xiaohan Wei. Online convex optimization with stochastic constraints. In Advances in Neural Information Processing Systems 30 , 2017.

Jianjun Yuan and Andrew Lamperski. Online convex optimization for cumulative constraints. In Advances in Neural Information Processing Systems 31 , 2018.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In 20th International Conference on Machine Learning , 2003.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the online convex optimization subject to certain constrains and assumes that such constrains are slowly changing. The main contribution is a $O(\\sqrt{T})$ regret bound under such constrains.\n\nSTRENGTHS:\nThis paper appears to prove some interesting technical results. However, due to its unclear presentation, I find it challenging to fully appreciate the significance of these findings.\n\nWEAKNESSES:\nThe presentation of this paper is very confusing, making it difficult to follow even in the introduction. Let me elaborate on the following points:\n\n1. The purpose of introducing the constraints $\\mathcal{C}_t$ for $t < T$ is unclear. From your regret formulation in (1), it appears to only depend on $\\mathcal{C}_T$. Are you assuming that $x_t$ must be selected from $\\mathcal{C}_t$? This doesn't seem to align with Assumption 2.\n2. There is no clear motivation behind the definition of (1). What is the reason for setting the constraints $g(x_T)\\ge c/\\sqrt{T}$?\n3. The set $I(x_t)$ is not well-defined. Where does $m$ originate from?\n4. What is the rationale for assuming $||g_t-g_{t+1}||\\le O(1/t)$? Why not $1/t^2$, $1/\\log t$, or $1/e^t$? How would the results change under these different bounds?\n5. What novel concept does Algorithm 1 introduce? Is it the construction of the velocity polytope?\n6. Can you highlight the main technical originality in your proofs? They appear to be simple applications of the analysis of online gradient descent.\n7. Could you provide some real-world examples where your results can be applied?\n\nConsidering the issues mentioned above, my recommendation leans towards a \"borderline reject.\" The paper lacks clarity in presentation, and several assumptions and definitions are not adequately justified or explained.\n\nQUESTIONS:\nUnfortually, I do not have any interesting technical questions for the authors at this point.\n\nLIMITATIONS:\nNo issue with negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-24"}

comments: SUMMARY:
The paper studies an online learning problem in which the learner aims at maximizing their reward subject to adversarial time-varying nonlinear constraints. The proposed algorithm guarantees a regret upper bound of order $T^{1/2}$ and converges to the feasible set at a rate of $T^{-1/2}$. Intrestingly, the feasible set may be unknown to the learner and slowly change over time. An interesting application described in the paper is computing equilibria in two-players zero-sum games in which players share a common resource.

STRENGTHS:
The set-up is very interesting. I particularly appreciated the connection to computing GNEs of some kind. The techniques introduced are novel and may find applications also in related problems.

WEAKNESSES:
I would add to the related works, when discussing performance metrics, also some pointers to work on online learning with long-term constraints stemming from the literature on bandits with knapsacks. One example is: "A Unifying Framework for Online Optimization with Long-Term Constraints" by Castiglioni et al. (NeurIPS 22) It would be nice to see a discussion of their baselines and how they relate to yours.

QUESTIONS:
None.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers an online constrained learning scenario, where partial information about the objective and the constraints, which could potentially be time-varying, is available at each time slot. The proposed algorithm, referred to as \"Constraint Violation Velocity Projection,\" constructs a velocity polytope at each time step and updates the decisions based on the solution to a quadratic velocity projection problem. It is shown that the proposed method leads to a $\\mathcal{O}(\\sqrt{T})$ regret and convergence rate of $\\mathcal{O}(1/\\sqrt{T})$ to the feasible set. Numerical results are also provided to corroborate the theoretical contributions.\n\nSTRENGTHS:\n- The considered problem is interesting and challenging from the perspective of the limited information available to the learning algorithm and the variation of constraints over time.\n- The paper is well written, and the theoretical analyses of the proposed method could be of interest to the NeurIPS audience.\n\nWEAKNESSES:\nThis is mostly a theoretical piece of work, but it is slightly unclear to me what the broader implications of the proposed algorithm would be. The two-player game experimental results presented in Section 4 are interesting, but I wonder if the authors could highlight at least another example use-case of the proposed algorithm.\n\nQUESTIONS:\n- How does the proposed method compare with primal-dual Lagrangian approaches? Do these approaches completely fail if the feedback received from the environment is limited, as assumed in this work?\n- Are there applications of the proposed algorithm in the context of constrained reinforcement learning with time-varying constraints and (potentially) partial observability? Following my comment under \"**Weaknesses**,\" this might be an area where the proposed algorithm could shine.\n\nLIMITATIONS:\nPlease refer to my comments under \"**Weaknesses**\" regarding the broader impact and application of the proposed algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a constrained online learning problem. The considered problem has a notion of constraint violation which is different from previous online learning problems. Also, the environment chooses not only a cost function but also a constraint function at each round. The algorithm can observe the cost function and the violated constraint function after playing each action. The authors propose an algorithm called CVV-Pro to solve this problem for both time-invariant and adversarial nonlinear constraints. The algorithm projects the gradients into a constructed velocity polytope in each step. The analysis shows that regret is sublinear and the constraint violation decreases with $O(1/\\sqrt{t})$ for each round $t$.\n\nSTRENGTHS:\nThe paper considers a novel constrained online learning problem where the environment chooses both the cost function and the constraint function. The proposed algorithm does not require the knowledge of the horizon length $T$ in advance. The paper bounds a new notion of the the constraint violation. The conclusions and the proof sketches are clearly presented. The proposed algorithm may be of interest by the community.\n\nWEAKNESSES:\nThe paper can be improved in the following points:\n\n- The time-varying constraints are also considered in [1,2,3]. Thus, it is not appropriate to claim that the setup differs from traditional OCO in terms of the time-varying constraint function.\n\n- Although the considered constraint in (1) requires the constraint for the last round. The previous cumulative constraints [1] should also imply this. \n\n- The bound of constraint violation relies on Assumption 1.2 which requires the change of constraint function shrinks with time and the constraint set $\\mathcal{C}_t$ is contained in $\\mathcal{Q}_t$. It would be better to explain why the two assumptions are reasonable.\n\n\n\n[1] Neely, M.J. and Yu, H., 2017. Online convex optimization with time-varying constraints. arXiv preprint arXiv:1702.04783.\n[2] Yu, H., Neely, M. and Wei, X., 2017. Online convex optimization with stochastic constraints. Advances in Neural Information Processing Systems, 30.\n[3] Cao, X. and Liu, K.R., 2018. Online convex optimization with time-varying constraints and bandit feedback. IEEE Transactions on automatic control, 64(7), pp.2665-2680.\n\nQUESTIONS:\nPlease see the questions in the Weakness part. Other questions are listed as below.\n\n1. Does the constraint violation bound cover the cumulative constraint violation bound in [1]?\n\n2. In Assumption 1.2, why can $\\mathcal{C}_t$ be contained in $\\mathcal{Q}_t$?\n\n3. In Theorem 3, what is the intuition of the attraction?\n\nLIMITATIONS:\nThe authors may need to address why  Assumption 1.2 and Assumption 3.1 are reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCVV-Pro is an online learning framework for non-stationary data streams, with support for handling constraints that may be unknown,  adversarially time-varying, and nonlinear. It doesn't assume that the feasible set is known, but proves that the algorithm still converges to it a rate of 1/sqrt(T). The method is designed to target complex online optimization and adaptive decision making tasks, with applications in two player games with shared constraints. \n\nThe method makes a few assumptions:\n1. It allows constraints to be relaxed and violated. Instead of ensuring feasibility at each step, it makes the violations converge to zero at a rate of O(1/sqrt(T)).\n2. An oracle exists that returns the value and the gradient for all the violated constraints.\n3. Constraints are allowed to vary in time, but at a decreasing rate of change at <= O(1/t)\n\nUnder these constraints, the paper proves that optimal regret at O(sqrt(T)) can be achieved. It is shown to be more efficient than online methods that project the candidates onto the constraint manifold (feasible set), but this may be expected since this method allows for the constraints to be actually violated, which would certainly reduce the loss faster.\n\nSTRENGTHS:\nThis is my understanding of the method: Similar to online gradient descent, the method gets the losses associated with x_t. But instead of projecting on the right constraint manifold (because we may not know it), the paper assumes that we can compute or get the gradients for each constraint. While the gradients could also be used directly (by turning constraints into soft regularizers), instead the method constructs a polytope from points that are shifted versions of x_t, by the value of the feasibility loss (g_t) in the direction of its gradient, scaled by a hyperparameter. So, in essence it is very similar to gradient descent (which would simply take the average of these points), but then it effectively projects the loss gradient onto this polytope to retrieve an update direction, which is then scaled by a step size and added to x_t.\n\nThis may go around the issue of having very complex constraint manifolds, where a direct projection may not be possible.\n\nWEAKNESSES:\n1) The paper is not easy to read, as a lot of the variables and concepts are not introduced linearly.\n2) Assumptions may be making the applicable problems very artificial, which takes away from the impact\n3) Figures need a bit more polish as they help a lot with understanding the main idea behind the paper.\n4) Experiments are pretty weak.\n5) No comparison with pure gradient descent\n\nDetails:\n1) The paper is not easy to read, as it sometimes skips introducing crucial variables. The reader needs to read ahead and jump back and forth to understand everything. For instance, g_t(x) is not explicitly defined on L30 (although clear from context later on). The reader can only guess that it is some sort of feasibility function. Confusingly x^* doesn't have a time index in Eq 1. L_F is not introduced in L50, presumably a smoothness index. B_70R in L50 is not explained (where does 70 come from?). The paper mentions that these assumptions are standard in online optimization literature, but skipping these details makes it hard for a new reader to understand the paper.\n\n2) It's very hard to make sense of all the assumptions the paper makes. Does allowing to violate constraints make sense? Does it make sense to assume that we have access to the gradients of the constraints? Does allowing the constraints to vary at decreasing rates correspond to any realistic setting for a practical problem? It feels like the paper is attempting to come up with an optimal algorithm for a problem that doesn't exist.\n\n3) I didn't understand Figure 1 at all. Figure 2 does almost a good job in explaining the method, but it is too imprecise. The arrows look hand drawn, and the actual area that we should focus on is very small.\n\n4) Experiments consist of artificially created two player \"games\", simulated a handful of times, and compared against a method that does assume constraints should not be violated. It's not clear to me where the real impact lies. The paper mentions tons of related work, including ones it was \"inspired from\", but there is no apple to apple comparison of any sort. The only results show that for the artificial setting the authors picked, their proof is consistent with the simulated results.\n\n5) Since the method has access to the gradients, and allows violating the \"constraints\", a much simpler baseline would be to simply add the constraint gradient losses to the gradient descent. This is already commonly done in other fields, and allows for efficient optimizers like Adam to be used. Since they also have gradients, constraints can also be time varying and nonlinear, with unknown feasible sets. Without comparing against this simple and powerful baseline, I'm not sure I understand the value of this method. \n\nFor instance, spectral normalization of the operators is known to help with training GANs, and it can either be done by explicit normalization, or by adding soft regularizers that pull the solution toward the right manifold. The papers that try these two methods are:\nRegularizer: (Yoshida & Miyato 2017) \" Spectral norm regularization for improving the generalizability of deep learning\"\nNormalizer: (Miyato et. al) \"Spectral normalization for Generative Adversarial Networks\".\nThe second paper mentions how normalization (projection) results in sample data dependent gradients unlike the regularizer (Section 2.1), which proves to be superior. This would be an interesting framework to compare the proposed method to these baselines.\n\nQUESTIONS:\n1) I don't understand 2.ii in L61. If Q_t = U(Q_t, S_t), and Q_0 is already R^n, how does this tell us anything useful? It just keeps getting a union of R^n with its subsets, which makes all Q_t's equal to R^n? I don't understand how Fig 1 clarifies this. I think either the union is wrong, or the initial condition. 2.i in L60: were these supposed to be g_t(x_t) instead of g_t? This makes it look like g_t's are converging, but the paper says in L68 that we should \"Note that ||g_t-g_1|| = O(ln(t))\", so it's not necessarily converging (but how do we \"note\" that?). The paper says that assumption 2.ii avoids this situation, but because I don't understand what 2.ii tries to achieve with the union, I don't get this part. I understand in essence what the paper tries to do, but I can't make what I read match what I understand.\n\n2) What is the difference from using the constraint gradients directly to update x_t a la gradient descent? Why do you think this is a better method?\n\n3) Why not make online gradient descent also relaxed so that projections are allowed to violate the constraints by a threshold, provided losses are better? That would give a more apple to apple comparison.\n\nLIMITATIONS:\nLike I mentioned above, I think the main limitation comes from all the assumptions (constraint have accessible gradients, vary at a decreasing rate, and can be violated). As the paper only provides a couple of artificial two player games in experiments, it's not clear how limited the actual impact of this work is.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}