SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Section 2.3, we present the pseudo code of our algorithm. In Section 2.4, we give a structural result showing that Algorithm 1 under Assumption 1.1 and a bounded iterate assumption guarantees an optimal  $\mathcal{O}(\sqrt{T})$  regret and converges to the feasible set at a rate of  $\mathcal{O}(1 / \sqrt{T})$ . In Appendix E, we show that the bounded iterate assumption can be enforced algorithmically, by introducing an additional hypersphere constraint that attracts the sequence  $\{x_{t}\}_{t\geq 1}$  to a fixed compact set.
2.3 Constraint Violation Velocity Projection (CVV-Pro)
We present below the pseudocode of Algorithm 1 for a fixed horizon length  $T$ , as it is standard in the literature [Hazan, 2016]. However, we note that our algorithm is oblivious to the horizon length  $T$ , i.e., it can run for any number of iterations without knowing  $T$  a priori.
Figure 2: Illustration of the proposed (CVV-Pro) algorithm. Left: the constraint  $g_{t,j}$  is violated by the current solution  $x_{t}$ . The cost gradient  $-\nabla f_{t}(x_{t})$  is projected onto the hyperplane (moved by  $-\alpha g_{t,j}(x_{t})$ ) with normal vector  $\nabla g_{t,j}(x_{t})$ . This yields  $r_t$  (see Section 2.5), and results in the velocity projection  $v_{t}$  ( $\eta = 1$  for clarity). Right: next iteration with updated  $x$ , where both  $f$  and  $g$  are changed. Then the procedure is applied recursively.
Algorithm 1 Constraint Violation Velocity Projection (CVV-Pro)
1: Requirements: See Assumption 1.1  
2: Input:  $\alpha >0$   
3: Initialization: Step sizes  $\{\eta_t = \frac{1}{\alpha\sqrt{t}}\}_{t\geq 1}$   
4: for  $t = 1$  to  $T$  do  
5: Play  $x_{t}$  and observe:  
6: cost information  $f_{t}(x_{t}),\nabla f_{t}(x_{t})$  and constraint information  $\{(g_i(x_t),\nabla g_i(x_t))\}_{i\in I(x_t)}$   
7: Construct the velocity polyhedron as follows:  
[ V_{\alpha}(x_{t}):= \{v\in \mathbb{R}^{n}\mid [\nabla g_{i}(x_{t})]^{\top}v\geq -\alpha g_{i}(x_{t}),\forall i\in I(x_{t})\} , ]  
8: Solve the velocity projection problem:  $v_{t} = \arg \min_{v\in V_{\alpha}(x_{t})}\frac{1}{2}\| v + \nabla f_{t}(x_{t})\|^{2}$   
9: Update:  $x_{t + 1} = x_{t} + \eta_{t}v_{t}$   
10: end for
Let  $x \in \mathcal{C}$  be an arbitrary decision. We show in Claim 2.2 that  $\alpha(x - x_t) \in V_\alpha(x_t)$ . Hence, the velocity polyhedron  $V_\alpha(x_t)$  is always non-empty and well defined.
2.4 Structural Result
Here, we show that Algorithm 1 under Assumption 1.1 and a bounded iterate assumption, guarantees an optimal  $\mathcal{O}(\sqrt{T})$  regret and converges to the feasible set at a rate  $\mathcal{O}(1 / \sqrt{T})$ . The bounded iterate assumption will be removed subsequently, which however, will require a more complex analysis.
Theorem 2.1 (Structural). Suppose Assumption 1.1 holds and in addition  $x_{t} \in \mathcal{B}_{R}$  for all  $t \in \{1, \ldots, T\}$ . Then, on input  $\alpha = L_{\mathcal{F}} / R$ , Algorithm 1 with step sizes  $\eta_{t} = \frac{1}{\alpha \sqrt{t}}$  guarantees the following for all  $T \geq 1$ :
( regret)
(feasibility)  $g_{i}(x_{t}) \geq -8\left[\frac{L_{\mathcal{G}}}{R} + 2\beta_{\mathcal{G}}\right]\frac{R^{2}}{\sqrt{t}}$ , for all  $t \in \{1, \dots, T\}$  and  $i \in \{1, \dots, m\}$ .
2.5 Proof Sketch of Theorem 2.1
Our analysis establishes, in two steps, an important geometric property that connects the convex costs and the concave constraints via the velocity polyhedron  $V_{\alpha}(x_t)$ . More precisely, we show that the inner product  $-r_t^\top (x_T^\star - x_t) \leq 0$  for all  $t \in \{1, \dots, T\}$ . This property will be crucial for deriving the regret and feasibility bounds.
In the first step, we leverage the constraints' concavity and show that the vector  $\alpha (x_T^\star - x_t)$  belongs to the velocity polyhedron  $V_{\alpha}(x_t)$ .
Claim 2.2. Suppose  $g_{i}$  is concave for every  $i\in \{1,\ldots ,m\}$ . Then  $\alpha (x - x_{t})\in V_{\alpha}(x_{t})$  for all  $x\in \mathcal{C}$ . In addition,  $x_{t}\notin \operatorname {int}(\mathcal{C})$  implies  $[\nabla g_i(x_t)]^\top [x - x_t]\geq 0$  for all  $x\in \mathcal{C}$ .
Proof. Let  $x \in \mathcal{C}$  be an arbitrary feasible decision, satisfying  $g_{i}(x) \geq 0$  for all  $i \in \{1, \dots, m\}$ . Since  $g_{i}$  is concave, we have  $g_{i}(x_{t}) + [\nabla g_{i}(x_{t})]^{\top}[x - x_{t}] \geq g_{i}(x) \geq 0$  and thus  $[\nabla g_{i}(x_{t})]^{\top}[x - x_{t}] \geq -g_{i}(x_{t})$ . The second conclusion follows by  $x_{t} \notin \operatorname{int}(\mathcal{C})$ , which implies  $g_{i}(x_{t}) \leq 0$ .
In the second step, we show that  $r_t^\top (x_t - x^\star)\leq 0$ , where  $r_t = v_t + \nabla f_t(x_t)$  is such that  $-r_t$  belongs to the normal cone  $N_{V_{\alpha}(x_t)}(v_t)$  of the velocity polyhedron  $V_{\alpha}(x_{t})$  evaluated at the projection  $v_{t}$ .
Lemma 2.3 (Main). Let  $v_{t}$  be the projection of  $-\nabla f_{t}(x_{t})$  onto the polyhedron  $V_{\alpha}(x_{t})$  such that  $v_{t} = r_{t} - \nabla f_{t}(x_{t}) \in V_{\alpha}(x_{t})$ , where  $-r_{t} \in N_{V_{\alpha}(x_{t})}(v_{t})$ . Then,  $-r_{t}^{\top}(x - x_{t}) \leq 0$  for all  $x \in \mathcal{C}$ .
Proof. By definition, the normal cone  $N_{V_{\alpha}(x_t)}(v_t)$  is given by  $\{u \in \mathbb{R}^n \mid u^\top(v - v_t) \leq 0, \forall v \in V_{\alpha}(x_t)\}$ . Then, by construction  $-r_t \in N_{V_{\alpha}(x_t)}(v_t)$  and thus it holds for every  $v \in V_{\alpha}(x_t)$  that  $-r_t^\top[v - v_t] \leq 0$ . The proof proceeds by case distinction:
Case 1. Suppose  $x_{t}$  is in the interior of  $\mathcal{C}$ . Then,  $I(x_{t}) = \emptyset$ , which implies  $-\nabla f_{t}(x_{t}) \in V_{\alpha}(x_{t}) = \mathbb{R}^{n}$  and thus  $r_t = 0$ .
Case 2. Suppose  $x_{t}$  is on the boundary or outside of  $\mathcal{C}$ , i.e.,  $I(x_{t}) \neq \emptyset$ . By Claim 2.2, we have  $[\nabla g_{i}(x_{t})]^{\top}[x - x_{t}] \geq 0$  for all  $x \in \mathcal{C}$ . By construction,  $v_{t} \in V_{\alpha}(x_{t})$  and thus  $v(x) = v_{t} + x - x_{t} \in V_{\alpha}(x_{t})$ . The statement follows by applying  $v = v(x)$  to  $-r_t^\top [v - v_t] \leq 0$ .
Regret. To establish the first conclusion of Theorem 2.1 (regret), we combine the preceding geometric property with the analysis of online gradient descent. Since  $f_{t} \in \mathcal{F}$  is convex, we upper bound the regret in terms of the gradient of  $f_{t}$ , namely  $\sum_{t=1}^{T} f_{t}(x_{t}) - f_{t}(x_{T}^{\star}) \leq \sum_{t=1}^{T} [\nabla f_{t}(x_{t})]^\top (x_{t} - x_{T}^{\star})$  and then we show that the following inequality holds
$$
\begin{array}{l} [ \nabla f _ {t} (x _ {t}) ] ^ {\top} (x _ {t} - x _ {T} ^ {\star}) - \frac {\eta_ {t}}{2} \| v _ {t} \| ^ {2} = r _ {t} ^ {\top} (x _ {t} - x _ {T} ^ {\star}) + \frac {\| x _ {t} - x _ {T} ^ {\star} \| ^ {2} - \| x _ {t + 1} - x _ {T} ^ {\star} \| ^ {2}}{2 \eta_ {t}} \\ \leq \frac {\left\| x _ {t} - x _ {T} ^ {\star} \right\| ^ {2} - \left\| x _ {t + 1} - x _ {T} ^ {\star} \right\| ^ {2}}{2 \eta_ {t}}. \tag {2} \\ \end{array}
$$
Moreover, in Appendix D (see Lemma D.2), we upper bound the velocity  $\| v_t\| \leq \alpha \| x_T^\star -x_t\| + 2\| \nabla f_t(x_t)\|$ . Combining Assumption 1.1 and  $x_{t}\in \mathcal{B}_{R}$  yields a uniform bound  $\| v_{t}\| \leq \mathcal{V}_{\alpha}$ , where for  $\alpha = L_{\mathcal{F}} / R$  we set  $\mathcal{V}_{\alpha} := 4L_{\mathcal{F}}$ . The desired regret follows by a telescoping argument and by convexity of the cost functions  $f_{t}\in \mathcal{F}$ .
Feasibility. For the second conclusion of Theorem 2.1 (convergence to the feasible set at a rate of  $1 / \sqrt{T}$ ), we develop a non-trivial inductive argument that proceeds in two steps. In Appendix D (see Claim D.6), we give a structural result that bounds the constraint functions from below. In particular, for every  $i\in I(x_{t})$  we have  $g_{i}(x_{t + 1})\geq (1 - \alpha \eta_{t})g_{i}(x_{t}) - \eta_{t}^{2}\mathcal{V}_{\alpha}^{2}\beta_{\mathcal{G}}$  and for every  $i\notin I(x_{t})$  it holds that  $g_{i}(x_{t + 1})\geq -\eta_{t + 1}\mathcal{V}_{\alpha}[2L_{\mathcal{G}} + \mathcal{V}_{\alpha}\beta_{\mathcal{G}} / \alpha ]$
Using an inductive argument, we establish in Appendix D (see Lemma D.5) the following lower bound:  $g_{i}(x_{t}) \geq -c\eta_{t}$  where  $c = 2\mathcal{V}_{\alpha}(L_{\mathcal{G}} + \mathcal{V}_{\alpha}\beta_{\mathcal{G}} / \alpha)$ . Choosing  $\alpha = L_{\mathcal{F}} / R$  implies that  $\mathcal{V}_{\alpha} = 4L_{\mathcal{F}}$ . Then, the desired convergence rate to the feasible set follows for the step size  $\eta_t = \frac{1}{\alpha\sqrt{t}}$ , since
$$
- c \eta_ {t} = - \frac {2 \mathcal {V} _ {\alpha}}{\alpha \sqrt {t}} \Big [ L _ {\mathcal {G}} + \frac {\beta_ {\mathcal {G}} \mathcal {V} _ {\alpha}}{\alpha} \Big ] = - 8 \Big [ \frac {L _ {\mathcal {G}}}{R} + 4 \beta_ {\mathcal {G}} \Big ] \frac {R ^ {2}}{\sqrt {t}}.
$$
3 Online Learning under Adversarial Nonlinear Constraints
In this section, we consider an online optimization problem with adversarially generated time-varying constraints. More precisely, at each time step  $t$ , the learner receives partial information on the current cost  $f_{t}$  and feasible set  $\mathcal{C}_t$ , and seeks to minimize (1). To make this problem well posed, we restrict the environment such that each feasible set  $\mathcal{C}_t$  is contained in  $\mathcal{Q}_t$  (see Section 1) and the rate of change between consecutive time-varying constraints decreases over time. We quantify a sufficient rate of decay with the following assumption.
Assumption 3.1 (TVC Decay Rate). We assume that the adversarially generated sequence  $\{g_t\}_{t \geq 1}$  of time-varying constraints is such that for every  $x \in \mathcal{B}_{4R}$  and all  $t \geq 1$ , the following holds  $\| g_{t+1}(x) - g_t(x) \|_{\infty} \leq \frac{98}{t+16} \left[ \frac{L_{\mathcal{G}}}{R} + 3\beta_{\mathcal{G}} \right] R^2$ .
We note that Assumption 3.1 essentially only requires  $\| g_{t + 1}(x) - g_t(x)\|_\infty \leq \mathcal{O}(1 / t)$ , as  $R$  can be chosen large enough such that the bound is satisfied. Of course,  $R$  will appear in our regret and feasibility bounds, but it will not affect the dependence on  $t$  or  $T$  (up to constant factors).
An important special case where Assumption 3.1 is satisfied, is summarized in the following Lemma. The proof is included in Appendix F (see Lemma F.7 and Lemma F.8).
Lemma 3.2. Suppose the functions  $\tilde{g}_{t,i}$  satisfy Assumption 1.1 and in addition there is a decision  $x_{t,i} \in \mathcal{B}_R$  such that  $\tilde{g}_{t,i}(x_{t,i}) = 0$  for every  $t \geq 1$  and  $i \in \{1, \dots, m\}$ . Then the time-averaged constraints  $g_{t,i}(x) \coloneqq \frac{1}{t} \sum_{\ell=1}^{t} \tilde{g}_{\ell,i}(x)$  satisfy Assumption 1.1 and Assumption 3.1.
3.2 Velocity Projection with Attractive Hypersphere Constraint
We show in Appendix E that the second assumption in Theorem 2.1, namely, “ $x_{t} \in \mathcal{B}_{R}$  for all  $t \geq 1$ ” can be enforced algorithmically. We achieve this in two steps.
1) Algorithmically, we introduce an additional hypersphere constraint  $g_{m + 1}(x_t) = \frac{1}{2} [R^2 - \| x_t\|^2]$  that attracts the decision sequence  $\{x_{t}\}_{t\geq 1}$  to a hypersphere  $\mathcal{B}_R$  and guarantees that it always stays inside a hypersphere  $\mathcal{B}_{4R}$  with a slightly larger radius.
More precisely, we augment the velocity polyhedron in Step 3 of Algorithm 1 as follows:  $V_{\alpha}'(x_t) = V_{\alpha}(x_t)$  if  $\| x \| \leq R$ , otherwise
$$
V _ {\alpha} ^ {\prime} (x _ {t}) = \left\{v \in V _ {\alpha} (x _ {t}) \mid [ \nabla g _ {m + 1} (x _ {t}) ] ^ {\top} v \geq - \alpha g _ {m + 1} (x _ {t}) \right\}.
$$
2) Analytically, we give a refined inductive argument in Appendix F (see Lemma E.5), showing that  $g_{m + 1}(x_t) \geq -27R^2 / \sqrt{t + 15}$ ,  $\| x_t \| \leq 4R$  and  $\| v_t \| \leq 7L_{\mathcal{F}}$ , for all  $t \geq 1$ .
3.3 Main Contribution
Our main contribution is to show that Algorithm 1 with the augmented velocity polyhedron  $V_{\alpha}^{\prime}(x_t)$ , achieves optimal  $\mathcal{O}(\sqrt{T})$  regret and satisfies  $g_{T}(x_{T}) \geq -\mathcal{O}(1 / \sqrt{T})$  convergence feasibility rate. Due to space limitations, we defer the proof to Appendix F.
Theorem 3.3 (Time-Varying Constraints). Suppose the functions  $\{f_t, g_t\}_{t \geq 1}$  satisfy Assumptions 1.1, 1.2 and 3.1. Then, on input  $R$ ,  $L_{\mathcal{F}} > 0$  and  $x_1 \in \mathcal{B}_R$ , Algorithm 1 applied with  $\alpha = L_{\mathcal{F}} / R$ , augmented velocity polyhedron  $V_{\alpha}'(\cdot)$  and step sizes  $\eta_t = \frac{1}{\alpha \sqrt{t + 15}}$  guarantees the following for all  $T > 1$ :
(regret)  $\sum_{t=1}^{T} f_{t}(x_{t}) - \min_{x \in \mathcal{C}_{T}} \sum_{t=1}^{T} f_{t}(x) \leq 246L_{\mathcal{F}}R\sqrt{T}$ ;
(feasibility)  $g_{t,i}(x_t)\geq -265\left[\frac{L_{\mathcal{G}}}{R} +4\beta_{\mathcal{G}}\right]\frac{R^2}{\sqrt{t + 15}},$  for all  $t\in \{1,\ldots ,T\}$  and  $i\in \{1,\dots ,m\}$
(attraction)  $g_{m + 1}(x_t)\geq -27\frac{R^2}{\sqrt{t + 15}}$  for all  $t\in \{1,\ldots ,T\}$
Our regret analysis in Theorem 3.3 builds upon the following key structural result that generalizes Lemma 2.3 to time-varying constraints. In particular, in Appendix F (see Lemma F.3), we show that given the feasible set  $\mathcal{C}_T\subset \mathcal{Q}_T$ , it holds for every  $x\in \mathcal{C}_T$  that  $-r_t^\top (x - x_t)\leq 0$  for all  $t\in \{1,\ldots ,T\}$ . As a result, a similar argument as in (2) shows that the regret is bounded by  $\mathcal{O}(\sqrt{T})$ .
Moreover, we note that the linear and quadratic dependence on  $R$  in Theorem 3.3 is consistent in length units. Let the radius  $R$  be of length units  $\ell$ , then the Lipschitz constant  $L_{\mathcal{F}}$ , which can be viewed as the supremum over the  $\ell_2$  norm of the gradient is of  $1/\ell$  units, and the  $\beta_{\mathcal{G}}$  smoothness constant (associated with Hessian) is of  $1/\ell^2$  units. This means that the regret bound in Theorem 3.3 has the same units as  $f_t$ , while the feasibility bound has the same units as  $g_t$ .
4 Simulation examples
Two-player games with shared resources are an excellent example for demonstrating the effectiveness and importance of our online learning framework. We apply our algorithm and show numerical experiments that support our theoretical findings.
We choose random instances of a two player game with linear utility and constraints. In particular, we consider the following optimization problem
$$
\min  _ {x \in \triangle_ {n}} \max  _ {y \in \triangle_ {n}} x ^ {\top} A y \quad \text {s u b j e c t t o} \quad C _ {x} x + C _ {y} y \leq 1, \tag {3}
$$
where  $\triangle_n = \{x\in \mathbb{R}^n\mid \sum_{i = 1}^n x_i = 1,x\geq 0\}$  is the probability simplex. Each component of the utility matrix  $A\in \mathbb{R}^{n\times n}$  is sampled from the normal distribution and the constraint matrices  $C_x,C_y\in [0,1]^{m\times n}$  have each of their components sampled uniformly at random from [0, 1].
Figure 3: (Setup) The CVV-Pro algorithm is executed on five random instances of the two-player game with shared resources (Section 4.1). The thick line is the mean and the thin lines indicate the minimum and the maximum over the five runs. (a) The regret follows the predicted  $\mathcal{O}(\sqrt{T})$  slope. (b) The CVV-Pro algorithm achieves a convergence rate of  $\mathcal{O}(1 / \sqrt{t})$  for the averaged decisions  $\overline{x}_t \coloneqq \frac{1}{t}\sum_{\ell=1}^t x_\ell$  towards  $\overline{x}_\infty$ . In our experiment, we set  $\overline{x}_\infty \coloneqq \overline{x}_{10000}$ . Similar behavior is reported for the averaged decisions  $\overline{y}_t$  of the adversary. (c) The maximal constraint violation expressed by  $-\min_{i \in I(x_t)} g_{t,i}(x_t)$  converges at a rate of  $\mathcal{O}(1 / \sqrt{t})$ , as predicted by our theoretical results. (Compute) For the implementation of CVV-Pro we have used the MATLAB R2019a numerical computing software. The computation of the experiment takes about 4 hours on a machine with CPU: Intel(R) i7-6800K 3.40 GHz with 6 cores, GPU: NVIDIA GeForce GTX 1080, and RAM: 32 GB.
4.1 Online Formulation
The problem in (3) can be modeled with our online learning framework (1) by choosing costs  $f_{t}(x) \coloneqq x^{\top}Ay_{t}$  and time-averaged resource constraints  $g_{T}(x) \coloneqq \frac{1}{T}\sum_{t = 1}^{T}\widetilde{g}_{t}(x)$ , where the function  $\widetilde{g}_t(x)\coloneqq 1 - C_xx - C_yy_t$ . Thus, the constraint in (3) is included as an average over the past iterations of  $y_{t}$ . The strategy for choosing  $y_{t}$  will be described below and, as we will see, the average of  $y_{t}$  over the past iterations converges. This ensures that the feasible set  $\mathcal{C}_t$  (defined in (1)) is slowly time-varying, while the averages of  $x_{t}$  and  $y_{t}$  over past iterates converge to equilibria in (3). Further, by a refined version of Lemma 3.2 (see Lemma F.6 in Appendix F), the time-averaged constraints  $g_{T}(x)$  satisfy Assumption 3.1.
In each iteration, Algorithm 1 seeks to minimize the online problem and commits to a decision  $x_{t}$ . The adversary computes the best response  $\hat{y}_{t}$  with respect to the decision  $x_{t}$  by solving  $\arg \max_{y\in \triangle_{n}}x_{t}^{\top}Ay$ . To make the dynamics more interesting, the adversary then commits with probability 0.8 to  $\hat{y}_{t}$  and with probability 0.2 to a random decision  $\xi_{t}$ , i.e.,  $y_{t} = 0.8\hat{y}_{t} + 0.2\xi_{t}$  where the random variable  $\xi_{t}$  is sampled uniformly at random from the probability simplex  $\triangle_{n}$ .
As both players optimize over the probability simplex  $(x, y \in \triangle_{n})$ , the sequence of decisions  $\{x_{t}\}_{t \geq 1}$  is automatically bounded. Thus, we can apply Theorem 3.3 with the original velocity polyhedron, as discussed in Appendix E. We implemented our algorithm with  $\eta_{t} = 1 / (\alpha \sqrt{t})$  and  $\alpha = 100$ .
We report results from numerical simulations with decision dimension  $n = 100$ ,  $m = 10$  shared resource constraints,  $T = 4000$  iterations, and five independently sampled instances of the two-player game. The learner's regret, depicted in Figure 3a, shows a clear correspondence with the theoretical prediction of  $\mathcal{O}(\sqrt{T})$ . Figure 3b presents the maximal constraint violation  $-\min_{i \in I(x_T)} \frac{1}{T} \sum_{t=1}^{T} \widetilde{g}_{t,i}(x_T)$ , which follows the predicted  $\mathcal{O}(1 / \sqrt{T})$  convergence rate. We also conclude from Figure 3c that the learner's averaged decisions  $\overline{x}_T = \frac{1}{T} \sum_{t=1}^{T} x_t$  converge at a rate of  $\mathcal{O}(1 / \sqrt{T})$ . Similarly, the averaged decisions  $\overline{y}_T$  of the adversary also converge at a rate of  $\mathcal{O}(1 / \sqrt{T})$ . We note that there is little variability in the results despite the different realizations of the matrices  $A, C_x, C_y$ .
Contrasting CVV-Pro and Online Gradient Descent In Appendix C, we show that our (CVV-Pro) algorithm outperforms the standard Online Gradient Descent algorithm in the two-player game from above. In particular, our algorithm achieves a lower regret and a runtime improvement of about  $60\%$ . Further, the percentage of violated constraints decreases rapidly and plateaus at  $20\%$ .
The amount of improvement in execution time is likely to be greater for higher-dimensional problems, where fewer constraints tend to be active at each iteration. Moreover, when the constraints are nonlinear, which includes  $\ell_p$  norm or spectral constraints, optimizing over the full feasible set can be computationally challenging. In contrast, the velocity projection step in CCV-Pro is always a convex quadratic program with linear constraints, regardless of the underlying feasible set.
It is important to emphasize that our work is theoretical, and the main contribution is to design and analyze a novel algorithm that combines techniques from the seemingly distant fields of online convex optimization (online gradient descent) and non-smooth mechanics (velocity space, see Muehlebach and Jordan [2022]). Nevertheless, the list of potential applications includes, but is not limited to: adversarial contextual bandits with sequential risk constraints Sun et al. [2017], network resource allocation Chen et al. [2017], logistic regression Cao and Liu [2019], Liu et al. [2022], ridge regression and job scheduling Liu et al. [2022], two-player games with resource constraints (Section 4), system identification and optimal control (Appendix B).
We propose an online algorithm that, despite the lack of information about the feasible set, achieves  $\mathcal{O}(\sqrt{T})$  regret. We further ensure convergence of violated constraint  $-\min \{g_T(x_T),0\}$  at a rate of  $\mathcal{O}(1 / \sqrt{T})$  and derive explicit constants for all our bounds that hold for all  $T\geq 1$ . We thus conclude that our algorithm matches the performance of traditional online projected gradients or Frank-Wolfe schemes, while requiring substantially less information about the feasible set and allowing the feasible set to be time-varying. Perhaps equally important is the fact that instead of performing projections onto the full feasible set at each iteration, our algorithm only optimizes over a local sparse linear approximation. We show the applicability of our algorithm in numeric simulations of random two-player games with shared resources.
We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039B). Georg Martius is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645. Pavel Kolev was supported by the Cyber Valley Research Fund and the Volkswagen Stiftung (No 98 571). Michael Muehlebach thanks the German Research Foundation and the Branco Weiss Fellowship, administered by ETH Zurich, for the support. We thank anonymous reviewers for comments, which helped improve the presentation of the paper.