SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pavel Kolev<sup>1</sup>, Georg Martius<sup>1,2</sup>, and Michael Muehlebach<sup>1</sup>
$^{1}$ Max Planck Institute for Intelligent Systems, Tübingen, Germany  
 $^{2}$ University of Tübingen, Tübingen, Germany  
{pavel.kolev, georg.martius, michael.muehlebach}@tuebingen.mpg.de
In many applications, learning systems are required to process continuous nonstationary data streams. We study this problem in an online learning framework and propose an algorithm that can deal with adversarial time-varying and nonlinear constraints. As we show in our work, the algorithm called Constraint Violation Velocity Projection (CVV-Pro) achieves  $\sqrt{T}$  regret and converges to the feasible set at a rate of  $1 / \sqrt{T}$ , despite the fact that the feasible set is slowly time-varying and a priori unknown to the learner. CVV-Pro only relies on local sparse linear approximations of the feasible set and therefore avoids optimizing over the entire set at each iteration, which is in sharp contrast to projected gradients or Frank-Wolfe methods. We also empirically evaluate our algorithm on two-player games, where the players are subjected to a shared constraint.
Today's machine learning systems are able to combine computation, data, and algorithms at unprecedented scales, which opens up new and exciting avenues in many domains, such as computer vision, computer graphics, speech and text recognition, and robotics [Jordan and Mitchell, 2015]. One of the leading principles that has enabled this progress is the focus on relatively simple pattern recognition and empirical risk minimization approaches, which mostly rely on offline gradient-based optimization and stipulate that the training, validation, and test data are independent and identically distributed.
Somewhat overlooked in these developments is the role of non-stationarity and constraints [Jordan, 2019]. Indeed, emerging machine learning problems involve decision-making in the real world, which typically includes interactions with physical, social, or biological systems. These systems are not only time varying and affected by past interactions, but their behavior is often characterized via fundamental constraints. Examples include cyber-physical systems where constraints are imposed by the laws of physics, multi-agent systems that are subjected to a shared resource constraint, or a reinforcement learning agent that is subjected to safety and reliability constraints. In particular, in their seminal work Auer et al. [2002] gave a reduction for the multi-arm bandit setting to the full information online optimization setting, by employing the multiplicative weights framework [Littlestone and Warmuth, 1994]. This classical reduction was recently extended by Sun et al. [2017] to the contextual bandit setting with sequential (time-varying) risk constraints.
This motivates our work, which is in line with a recent trend in the machine learning community towards online learning, adaptive decision-making, and online optimization. More precisely, we study an online problem with slowly time-varying constraints, governed by the following interaction protocol (see Assumption 1.2). In each time step  $t$ , the learner commits a decision  $x_{t}$  and then in addition to a loss value  $f_{t}(x_{t})$  with its gradient  $\nabla f_{t}(x_{t})$  receives partial information about the current feasible set  $\mathcal{C}_t \coloneqq \{x \in \mathbb{R}^n \mid g_t(x) \geq 0\}$ , where the constraint function  $g_{t}(x)$  is defined as
$[g_{t,1}(x), \ldots, g_{t,m}(x)]$ . The quality of the learner's decision making is measured, for every  $T \geq 1$ , by comparing to the best decision in hindsight  $x_T^\star \in \arg \min_{x \in \mathcal{C}_T} \sum_{t=1}^T f_t(x)$ , that is,
$$
\sum_ {t = 1} ^ {T} f _ {t} \left(x _ {t}\right) - \sum_ {t = 1} ^ {T} f _ {t} \left(x _ {T} ^ {\star}\right) \quad \text {s u b j e c t t o} \quad g _ {T} \left(x _ {T}\right) \geq - \frac {c}{\sqrt {T}}, \tag {1}
$$
which will be shown to be bounded by  $\mathcal{O}(\sqrt{T})$  for our algorithm. The functions  $f_{t}$  and  $g_{t}$  are restricted to  $f_{t}\in \mathcal{F}$  and  $g_{t}\in \mathcal{G}$  (as defined in Assumption 1.1) and  $c > 0$  is an explicit constant.
It is important to note that our performance objective (1) is symmetric in the sense that the constraint  $x \in \mathcal{C}_T$  applies to both the learner's decision  $x_T$  and the benchmark  $x_T^\star$ . This contrasts prior work by Neely and Yu [2017], Yu et al. [2017], Sun et al. [2017], Chen et al. [2017], Cao and Liu [2019] and Liu et al. [2022], where a different notion of constraint violation  $\sum_{t=1}^{T} g_t(x_t) \geq -c_0 \sqrt{T}$  is used for the learner, while either a single benchmark  $x_{1:T}^\star$  satisfies  $g_t(x_{1:T}^\star) \geq 0$  for all  $t \in \{1, \dots, T\}$  or multiple benchmarks  $\{x'_t\}_{t=1}^T$  satisfy  $x'_t \in \arg \min_{x \in \mathcal{C}_t} f_t(x)$ . Unlike (1), different requirements are imposed on the learner and the benchmark(s), which leads to asymmetric regret formulations:  $\sum_{t=1}^{T} f_t(x_t) - \sum_{t=1}^{T} f_t(x_{1:T}^\star)$  and  $\sum_{t=1}^{T} f_t(x_t) - \sum_{t=1}^{T} f_t(x'_t)$ , respectively. Furthermore, as our bound  $g_T(x_T) \geq -c / \sqrt{T}$  applies for all  $T \geq 1$ , it implies the cumulative constraint violation bound in Neely and Yu [2017] up to a constant factor  $\sum_{t=1}^{T} g_t(x_t) \geq -c \sum_{t=1}^{T} 1 / \sqrt{t} \geq -2c \sqrt{T}$ .
Even more intriguing is the fact that our algorithm is unaware of the feasible sets a-priori, and obtains, at each iteration, only a local sparse approximation of  $\mathcal{C}_t$  based on the first-order information of the violated constraints. The indices of all violated constraints at  $x_{t}$  will be captured by the index set  $I(x_{t}) := \{i\in \{1,\dots ,m\} \mid g_{t,i}(x_{t})\leq 0\}$ , while  $G(x_{t}) := [\nabla g_{t,i}(x_{t})]_{i\in I(x_{t})}$  denotes the matrix whose columns store the corresponding gradients. In order to guarantee a regret of  $\mathcal{O}(\sqrt{T})$  in (1) we require the following assumptions.
Assumption 1.1. There exist  $R, L_{\mathcal{F}}, L_{\mathcal{G}} > 0$ : 1)  $\mathcal{F}$  is a class of convex functions, where every  $f \in \mathcal{F}$  satisfies  $||\nabla f(x)|| \leq L_{\mathcal{F}}, \forall x \in \mathcal{B}_{4R}$ , with  $||\cdot||$  the  $\ell_2$  norm and  $\mathcal{B}_R$  the hypersphere of radius  $R$  centered at the origin; 2)  $\mathcal{G}$  is a class of concave  $\beta_{\mathcal{G}}$ -smooth functions, where every  $g$  satisfies  $||\nabla g(x)|| \leq L_{\mathcal{G}}, \forall x \in \mathcal{B}_{4R}$ ; 3) The feasible set  $\mathcal{C}_t$  is non-empty and contained in  $\mathcal{B}_R$  for all  $t$ .
We note that these assumptions are standard in online optimization [Hazan, 2016, Ch. 3]. The learner's task is nontrivial even in the case where the feasible set is time invariant. If the feasible set is time varying, additional assumptions are required that restrict the amount that the feasible set is allowed to change. These two assumptions, see Part 2 i) and ii) below, are described by the following interaction protocol between the learner and the environment:
Assumption 1.2. (Interaction protocol) At each time step  $t \in \{1, \dots, T\}$ :
1) the learner chooses  $x_{t}$ ;
2) the environment chooses  $f_{t} \in \mathcal{F}$  and  $g_{t} \in \mathcal{G}$  such that i)  $||g_t(x) - g_{t - 1}(x)||_\infty = \mathcal{O}(1 / t)$ , uniformly for all  $x \in \mathcal{B}_{4R}$ , with  $||\cdot||_{\infty}$  the  $\ell_{\infty}$  norm, and ii)  $\mathcal{C}_t$  is contained in  $\mathcal{Q}_t := \cap_{\ell=0}^{t-1} S_\ell$ , where  $S_t := \{x \in \mathbb{R}^n \mid G(x_t)^\top (x - x_t) \geq 0\}$  is a cone centered at  $x_t$  for  $t \geq 1$  and  $S_0 = \mathbb{R}^n$  (the situation is illustrated in Figure 1, more details are presented in Appendix A);
3) the environment reveals to the learner partial information on cost  $f_{t}(x_{t})$ ,  $\nabla f_{t}(x_{t})$  and all violated constraints  $g_{t,i}(x_t)$ ,  $\nabla g_{t,i}(x_t)$  for  $i\in I(x_{t})$ .
The requirements i)  $||g_t - g_{t-1}||_\infty = \mathcal{O}(1/t)$ ; and ii)  $\mathcal{C}_t \subset \mathcal{Q}_t$  restrict the feasible sets that the environment can choose. We note that despite the fact that  $||g_t - g_{t-1}||_\infty = \mathcal{O}(1/t)$ ,  $||g_1 - g_t||_\infty = \Theta(\ln(t))$ , which means that the sequence of functions  $g_t$  that defines  $\mathcal{C}_t$  does not converge in general. As a result,  $\mathcal{C}_t$  may evolve in such a way that the initial iterates  $x_1, x_2, \ldots, x_{t_0}$  achieve a large cost compared to  $\min_{x \in \mathcal{C}_T} \sum_{t=1}^T f_t(x)$ , as these are constrained by the sets  $\mathcal{C}_1, \mathcal{C}_2, \ldots, \mathcal{C}_{t_0}$ , which may be far away from  $\mathcal{C}_T$ . The second requirement ii)  $\mathcal{C}_t \subset \mathcal{Q}_t$  avoids this situation and is therefore key for obtaining an  $\mathcal{O}(\sqrt{T})$  regret.
Our setup differs from traditional online convex optimization [Zinkevich, 2003] in the following two important ways:
Figure 1: At each time step, the feasible set  $\mathcal{C}_t$  contained in a polyhedral intersection  $\mathcal{Q}_t$  changes slightly and is only partially revealed.
i) The environment chooses not only the functions  $f_{t}$  but also the nonlinear constraint functions  $g_{t}$ , ii) even if  $g_{t}$  is time-invariant, i.e.,  $g_{t} = g$  for all  $t$  the learner has only access to local information about the feasible set.
That is, the information about the feasible set is only revealed piece-by-piece and needs to be acquired by the agent through repeated queries of a constraint violation oracle.
We propose an online algorithm that despite the lack of information about the feasible set, achieves  $\mathcal{O}(\sqrt{T})$  regret, and will derive explicit non-asymptotic bounds for the regret and the convergence to  $\mathcal{C}_T$ . We thus conclude that our algorithm matches the performance of traditional online projected gradients or Frank-Wolfe schemes, while requiring substantially less information about the feasible set and allowing it to be time-varying. Perhaps equally important is the fact that instead of performing projections onto the full feasible set at each iteration, our algorithm only optimizes over a local sparse linear approximation. If constraints are nonlinear, which includes norm-constraints or constraints on the eigenvalues of a matrix, optimizing over the full feasible set at each iteration can be computationally challenging.
Online learning has its roots in online or recursive implementations of algorithms, where due to the piece-by-piece availability of data, algorithms are often analyzed in a non i.i.d. setting. A central algorithm is the multiplicative weights scheme [Freud and Schapire, 1997], where a decide repeatedly chooses between a finite or countable number of options with the aim of minimizing regret. This online learning model not only offers a unifying framework for many classical algorithms [Blum, 1998], but represents a starting point for online convex optimization Hazan [2016], and adversarial bandits [Lattimore and Szepesvári, 2020]. Our approach extends this line of work by allowing the environment to not only choose the objective functions  $f_{t}$ , but also the constraints  $g_{t}$ . Due to the fact that our learner only obtains local information about the feasible set, our work is somewhat related to Levy and Krause [2019], Lu et al. [2023], Garber and Kretzu [2022], Mhammedi [2022], where the aim is to reduce the computational effort of performing online projected gradient steps or Frank-Wolfe updates. More precisely, Levy and Krause [2019] propose an algorithm that directly approximates projections, while requiring multiple queries of the constraint functions and their gradients. A slightly different constraint violation oracle is assumed in Garber and Kretzu [2022], where the learner can query separating hyperplanes between a given infeasible point and the feasible set. Algorithmically, both Garber and Kretzu [2022] and Levy and Krause [2019] depart from online gradient descent, where the latter computes projections via an approximate Frank-Wolf-type scheme. An alternative is provided by Mhammedi [2022] and Lu et al. [2023], where optimizations over the entire feasible set are simplified by querying only a set membership oracle based on the Minkowski functional. While our approach also avoids projections or optimizations over the entire feasible set, we introduce a different constraint violation oracle that returns a local sparse linear approximation of the feasible set. We call the constraint violation oracle only once every iteration and do not require a two-step procedure that involves multiple oracle calls. In addition, we also allow for adversarial time-varying constraints.
In addition, there has been important recent work that developed online optimization algorithms with constraints. In contrast to the primal formulation of our algorithm, these works are based on primal-dual formulations, where the algorithm is required to satisfy constraints on average, so called long-term constraints. The research can be divided into two lines of work Mahdavi et al. [2012], Jenatton et al. [2016], Yu and Neely [2020] and Yuan and Lamperski [2018], Yi et al. [2021] that use a set of weaker and stricter definitions for constraint violations and investigate time-invariant constraints, which contrasts our formulation that includes time-varying constraints. A third line of work by Mannor et al. [2009], Chen et al. [2017], Neely and Yu [2017], Yu et al. [2017], Sun et al. [2017], Cao and Liu [2019], Liu et al. [2022] focuses on time-varying constraints, where, however, the following weaker notion of constraint violation is used:  $\sum_{t=1}^{T} g_t(x_t) \geq -c\sqrt{T}$ , where  $t$  refers to time and  $x_t$  to the learner's decision. This metric allows constraint violations for many iterations, as long as these are compensated by strictly feasible constraints (in the worst case even with a single feasible constraint with a large margin). In contrast, our algorithm satisfies  $g_t(x_t) \geq -c/\sqrt{t}$  for all iterations  $t \in \{1, \ldots, T\}$ , where  $c$  is an explicit constant independent of the dimension of the decision variable and the number of constraints. This means that we can explicitly bound the constraint violation at every iteration, whereas infeasible and strictly feasible iterates cannot compensate each other.
An important distinction to Neely and Yu [2017] is given by our performance metric (see also the discussion in Neely and Yu [2017] and Liu et al. [2022]). On the one hand, the work by Chen et al. [2017], Cao and Liu [2019], Liu et al. [2022] use  $\sum_{t=1}^{T} f_t(x_t) - \sum_{t=1}^{T} f_t(x_t')$  as a performance measure, where the iterates  $x_t$  are required to satisfy  $\sum_{t=1}^{T} g_t(x_t) \geq -c\sqrt{T}$  and the optimal solutions  $x_t'$  satisfy  $x_t' \in \arg \min_{x \in \mathcal{C}_t} f_t(x)$ . On the other hand, the work by Neely and Yu [2017], Yu et al. [2017], Sun et al. [2017] use  $\sum_{t=1}^{T} f_t(x_t) - \sum_{t=1}^{T} f_t(x_{1:T}^\star)$  as a performance measure, where the iterates  $x_t$  are required to satisfy  $\sum_{t=1}^{T} g_t(x_t) \geq -c\sqrt{T}$  and the optimal solution  $x_{1:T}^\star$  satisfies  $g_t(x_{1:T}^\star) \geq 0$  for all  $t \in \{1, \ldots, T\}$ . This leads to a major asymmetry in the way regret is measured: while the iterates of the online algorithm only need to satisfy a cumulative measure of constraint violation, the benchmark  $x_{1:T}^\star$ , which represents the best fixed decision in hindsight, is required to satisfy all constraints  $g_t(x_{1:T}^\star) \geq 0$  for  $t = \{1, \ldots, T\}$ . In contrast, the performance metric introduced in (1) is symmetric and imposes the same constraint  $x \in \mathcal{C}_T$  (approximately) on the learner's decision  $x_T$  and (exactly) on the benchmark  $x_T^\star$ . These features make our algorithm a valuable addition to the algorithmic toolkit of online constrained optimization.
Castiglioni et al. [2022] studied the following asymmetric setting with adversarial environment, benchmark  $x_{T}^{\star}$  belonging to  $\arg \min_{x\in \mathcal{X}}\sum_{t = 1}^{T}f_{t}(x)$  subject to  $\frac{1}{T}\sum_{t = 1}^{T}g_{t}(x)\geq 0$ , online iterates  $x_{t}$  satisfying  $\frac{1}{T}\sum_{t = 1}^{T}g_{t}(x_{t})\geq -\mathcal{O}(1 / \sqrt{T})$ , and regret  $\sum_{t = 1}^{T}f_{t}(x_{t}) - \sum_{t = 1}^{T}f_{t}(x_{T}^{\star})\leq \mathcal{O}(\sqrt{T})$ . Their benchmark and regret formulation can be obtained as a special case of our formulation with time-averaged constraints, that is, when our  $g_{T}(x)$  is chosen as  $\frac{1}{T}\sum_{t = 1}^{T}g_{t}(x)$ . In contrast, our iterate  $x_{T}$  satisfies  $\frac{1}{T}\sum_{t = 1}^{T}g_{t}(x_{T})\geq -\mathcal{O}(1 / \sqrt{T})$ , a constraint that is asymptotically the same as the one satisfied by the benchmark  $x_{T}^{\star}$ . We further note that they introduced a parameter  $\rho = \sup_{x\in \mathcal{X}}\min_{t\in [T]}\min_{i\in [m]}g_{t,i}(x)$ , which is required to be positive and known to the algorithm for achieving  $\mathcal{O}(\sqrt{T})$  regret. Notably  $\rho >0$  implies that the intersection of all feasible sets is non-empty, which is a strong assumption (as is knowledge about the parameter  $\rho$ ). In our formulation with time-averaged constraints, Assumption 1.2 reduces to the requirement that the feasible set  $C_t$  belongs to a polyhedral intersection  $Q_{t}$ , which does not require a non-empty intersection of all  $C_t$  (has a geometrical interpretation and the assumption  $||g_t - g_{t - 1}||_\infty = \mathcal{O}(1 / t)$  is automatically satisfied). Thus, there are situations, where the regret bound from Castiglioni et al. [2022] becomes vacuous (for  $\rho = 0$ ), while our method still provably achieves  $\mathcal{O}(\sqrt{T})$  regret. Additional differences are that Castiglioni et al. [2022] considers primal-dual methods and assumes that all constraints are revealed after every iteration, whereas our method is primal-only and has only partial information about all violated constraints. The latter point reduces computation and simplifies projections onto the velocity polyhedron, but requires a nontrivial inductive argument for establishing  $\mathcal{O}(\sqrt{T})$  regret.
Other relevant related studies have investigated online learning problems with supply/budget constraints. In these settings, the decision maker must choose a sequence of actions that maximizes their expected reward while ensuring that a set of resource constraints are not violated. The process terminates either after a pre-specified time horizon has been reached or when the total consumption of some resource exceeds its budget. Badanidiyuru et al. [2018] introduced the bandits with knapsacks framework, which considers bandit feedback, stochastic objective and constraint functions. They proposed an optimal algorithm for this problem, which was later improved by Agrawal and Devanur [2014, 2019] and Immorlica et al. [2022]. Immorlica et al. [2022] introduced the adversarial bandits with knapsacks setting and showed that an appropriate benchmark for this setting is the best fixed distribution over arms. Since no-regret is no longer possible under this benchmark, they provide no- $\alpha$ -regret guarantees for their algorithm.
An important special case of our online learning model arises when the environment is represented by an adversarial player that competes with the learner. This corresponds to a repeated generalized Nash game due to the constraint that couples the decisions of the learner and its adversary. If the adversary plays best response, the resulting equilibria are characterized by quasi-variational inequalities [Facchinei and Kanzow, 2007] and there has been important recent work, for example by Jordan et al. [2023], Kim [2021], Facchinei and Kanzow [2010] that proposes different gradient and penalty methods for solving these inequalities. Our approach adopts a different perspective, rooted in online learning, which allows us to derive non-asymptotic convergence results for a first-order gradient-based algorithm that can be implemented in a straightforward manner. Our approach is also inspired by the recent work of Muehlebach and Jordan [2022], who propose a similar algorithm for the offline setting.
We give an online optimization scheme under unknown non-linear constraints that achieves an optimal  $\mathcal{O}(\sqrt{T})$  regret and converges to the latest feasible set at a rate of  $\mathcal{O}(1 / \sqrt{T})$ . There are two variants of our problem formulation: The first deals with situations where constraints are unknown but fixed, the second allows constraints to be chosen in a time-varying and adversarial manner.
Our algorithm, named Constraint Violation Velocity Projection (CVV-Pro), has the following features:
1. It assumes access to a new type of oracle, which on input  $x_{t}$ , returns partial information on all currently violated constraints. Namely, the value  $g_{t,i}(x_t)$  and the gradient  $\nabla g_{t,i}(x_t)$  for all  $i\in I(x_{t})$ .
2. It projects an adversarially generated negative cost gradient  $-\nabla f_{t}(x_{t})$  onto a velocity polyhedron  $V_{\alpha}(x_{t})\coloneqq \left\{v\in \mathbb{R}^{n}\mid [\nabla g_{i}(x_{t})]^{\top}v\geq -\alpha g_{i}(x_{t}),\forall i\in I(x_{t})\right\}$ . Due to the linear and local structure of  $V_{\alpha}(x_{t})$ , the projection can be computed efficiently.
3. In contrast to standard online methods that project in each round a candidate decision onto the feasible set, our method trades off feasibility for efficiency. In particular, it produces a sequence of decisions that converges at a rate of  $\mathcal{O}(1 / \sqrt{T})$  to the latest feasible set.
4. Our method handles time-varying adversarial constraints  $g_{t}$ , provided a decreasing rate of change  $\| g_{t + 1} - g_t\|_\infty \leq \mathcal{O}(1 / t)$  and that each feasible set  $\mathcal{C}_t$  belongs to  $\mathcal{Q}_t$  (see Assumption 1.2). As we show in Section 3.1, an important special case where the assumption of decreasing rate of change is satisfied is given by  $g_{t} = \frac{1}{t}\sum_{j = 1}^{t}\tilde{g}_{j}$ , i.e., when  $g_{t}$  represents an average of constraints  $\tilde{g}_t$  over time.
1.3 Outline
Section 2 describes our algorithm and considers the situation where  $g_{t}$  is time invariant. This sets the stage for our main results in Section 3 that provide regret guarantees for our new online convex optimization setting with non-stationary, nonlinear, and unknown constraints. An important and interesting application of our algorithm are generalized Nash equilibrium problems, as will be illustrated with a numerical experiment in Section 4. The experiment will also highlight that the numerical results agree with the theoretical predictions.
2 Online learning under unknown, time-invariant, and nonlinear constraints
2.1 Online Gradient Descent
Online gradient descent [Hazan, 2016, Ch. 3.1] is a classical and perhaps the simplest algorithm that achieves optimal  $\mathcal{O}(\sqrt{T})$  regret for the setting of a compact, convex, time-invariant, and a priori known feasible set. It consists of the following two operations: i)  $y_{t + 1} = x_t - \eta_t\nabla f_t(x_t)$  takes a step from the previous point in the direction of the previous cost gradient; and ii)  $x_{t + 1} = \mathrm{Proj}_{\mathcal{C}}(y_{t + 1})$  projects  $y_{t + 1}$  back to the feasible set  $\mathcal{C}$ , as  $y_{t + 1}$  may be infeasible.
In this section, we generalize the online gradient descent algorithm to the setting where the feasible set is unknown a priori and has to be learned through repeated queries of a constraint violation oracle that only reveals local information.
2.2 Overview