Title: Beyond Primal-Dual Methods in Bandits with Stochastic and Adversarial Constraints
Venue: NeurIPS 2024
Accepted: yes
Keywords: bandits with knapsacks

long-term constraints
Abstract: We address a generalization of the bandit with knapsacks problem, where a learner aims to maximize rewards while satisfying an arbitrary set of long-term constraints. Our goal is to design best-of-both-worlds algorithms that perform optimally under both stochastic and adversarial constraints. Previous works address this problem via primal-dual methods, and require some stringent assumptions, namely the Slater's condition, and in adversarial settings, they either assume knowledge of a lower bound on the Slater's parameter, or impose strong requirements on the primal and dual regret minimizers such as requiring weak adaptivity. We propose an alternative and more natural approach based on optimistic estimations of the constraints. Surprisingly, we show that estimating the constraints with an UCB-like approach guarantees optimal performances.
Our algorithm consists of two main components: (i) a regret minimizer working on moving strategy sets and (ii) an estimate of the feasible set as an optimistic weighted empirical mean of previous samples. The key challenge in this approach is designing adaptive weights that meet the different requirements for stochastic and adversarial constraints. Our algorithm is significantly simpler than previous approaches, and has a cleaner analysis. Moreover, ours is the first best-of-both-worlds algorithm providing bounds logarithmic in the number of constraints. Additionally, in stochastic settings, it provides $\widetilde O(\sqrt{T})$ regret without Slater's condition.

Beyond Primal-Dual Methods in Bandits with Stochastic and Adversarial Constraints

Abstract

We address a generalization of the bandit with knapsacks problem, where a learner aims to maximize rewards while satisfying an arbitrary set of long-term constraints. Our goal is to design best-of-both-worlds algorithms that perform optimally under both stochastic and adversarial constraints. Previous works address this problem via primal-dual methods, and require some stringent assumptions, namely the Slater's condition, and in adversarial settings, they either assume knowledge of a lower bound on the Slater's parameter, or impose strong requirements on the primal and dual regret minimizers such as requiring weak adaptivity. We propose an alternative and more natural approach based on optimistic estimations of the constraints. Surprisingly, we show that estimating the constraints with an UCBlike approach guarantees optimal performances. Our algorithm consists of two main components: (i) a regret minimizer working on moving strategy sets and (ii) an estimate of the feasible set as an optimistic weighted empirical mean of previous samples. The key challenge in this approach is designing adaptive weights that meet the different requirements for stochastic and adversarial constraints. Our algorithm is significantly simpler than previous approaches, and has a cleaner analysis. Moreover, ours is the first best-of-both-worlds algorithm providing bounds logarithmic in the number of constraints. Additionally, in stochastic settings, it provides ˜ O ( √ T ) regret without Slater's condition.

1 Introduction

We address the problem faced by a decision maker who aims to maximize its cumulative reward over a time horizon T , while satisfying an arbitrary set of m long-term constraints. At each round t , the learner selects an action a t from a finite set of K actions, and then observes a reward f t ( a t ) and some costs g t ( a t ) ∈ [ -1 , 1] m . The goal is to design best-of-both-worlds algorithms for this problem that perform optimally under both stochastic and adversarial constraints. We always assume rewards are generated adversarially. This is because the real complexity of the problem is captured by the nature of the constraints, so that transitioning from adversarial to stochastic rewards under the same type of constraints does not affect our results.

The first works on bandits with constraints focus on budget constraints, a.k.a bandit with knapsack ( BwK ) [7] study the settings in which both rewards and constraints are i.i.d. and propose an UCBbased approach, combined with primal-dual method. Agrawal and Devanur [2] provide an UCB-like approach for more general rewards and costs. Immorlica et al. [21], Kesselheim and Singla [22]

analyse settings with adversarial constraints and rewards, providing a primal-dual algorithm to tackle the problem. Castiglioni et al. [15] show that a similar primal-dual approach provides best-of-bothworlds guarantees. Many subsequent works extend the setting to more general constraints, mostly employing primal-dual methods [16, 17, 28, 11, 13, 10, 18]. Primal-dual methods have been the only effective method that provides best-of-both-worlds guarantees for bandits with constraints [16, 17, 11, 13, 10]. However, such methods require assumptions that are particularly stringent in settings beyond knapsack constraints. First, they require the existence of a strictly feasible solution ( i.e., Slater's condition) to avoid a regret of order O ( T 3 / 4 ) [16, 28]. While this assumption always holds in bandits with knapsack setting (where 'doing nothing' incurs in a negative cost equal to the per-round budget), this assumption is far more stringent with general constraints. Moreover, some works require the knowledge of a lower bound on the Slater's parameter [16, 28]. Subsequent works circumvent this assumption at the expense of strong requirements on the primal and dual regret minimizers [17, 11, 13, 1]. In particular, such approaches require weakly-adaptive primal and dual regret minimizers. The challenge of applying such primal-dual algorithms to bandit beyond knapsack constraint is reflected in regret bounds that exhibit non-optimal dependencies on some parameters. For instance, a polynomial (instead of logarithmic) dependence on the number of constraints [17, 11, 13]. For further pointers to the literature, we refer to Appendix A.

1.1 Our contribution

We propose an alternative and insightful approach to design best-of-both-worlds algorithms for bandit with long-term constraints. Our method relies on optimistic estimations of the constraints through a weighted empirical mean of past samples. Surprisingly, we demonstrate that using a UCB-based approach to estimate the constraints ensures optimal performance under both stochastic and adversarial constraints. Our algorithm differs significantly from previous UCB-based approaches. For instance, it guarantees no-regret even with adversarial rewards and stochastic constraints, unlike previous works [2, 7, 23]. Moreover, it is the first UCB-like approach that provides an optimal competitive ratio of 1 + 1 / ρ with adversarial constraints, where ρ is the unknown Slater's parameter.

Our algorithm consists of two simple components. The first is an adversarial regret minimizer working on moving strategy sets . In particular, at each round, the regret minimizer chooses a strategy in the current optimistic estimation of the feasible set, and is required to achieve no-regret with respect to any strategy in the intersection of all feasibility set estimations. The second component is a tool for estimating the feasible set using an optimistic weighted mean of previous samples. The key challenge in this approach is designing adaptive weights that meet the different requirements for stochastic and adversarial constraints. Intuitively, in stochastic settings, we aim to converge to the (unweighted) empirical mean of the observed constraints. Conversely, in adversarial settings, we should assign larger weights to recent samples to address time-dependent constraints.

Not only is our algorithm significantly simpler than previous approaches, with a clean and insightful analysis, but it also provides better theoretical performance than primal-dual methods. Indeed, it is the first best-of-both-worlds algorithm to provide bounds logarithmic in the number of constraints. Moreover, in stochastic settings, it is the first algorithm to provide ˜ O ( √ T ) regret without requiring Slater's condition. Finally, it guarantees that the expected violation in the current round converges to zero, making our algorithm 'converge" to strategies that are feasible in expectation. This provides a more stable and consistent control on the violations.

2 Model and Preliminaries

We address the problem faced by an agent aiming at maximizing its cumulative reward over a time horizon T , while satisfying J m K long-term constraints. 1 The agent has a set J K K of available actions and, at each round t ∈ J T K , selects a t ∈ J K K . The agent then observes the corresponding reward f t ( a t ) ∈ [0 , 1] and a cost g ( i ) t ( a t ) ∈ [ -1 , 1] , for each constraint i ∈ J m K . We define the cumulative violation of the i th constraint as

while V T := max i ∈ J m K V ( i ) T is the maximum violation across all constraints. At a high level, we want to minimize the regret while keeping the violation of each constraint V ( i ) T sublinear in T .

The focus of this paper is on handling both stochastic and adversarial constraints. Conversely, we always assume the rewards to be generated up-front by an adversary; we do not treat explicitly the situation where the rewards are generated i.i.d. because our guarantees are already tight for the harder case of adversarial rewards. 2 In the stochastic setting, we assume that g t = { g ( i ) t } i ∈ J m K is drawn i.i.d. from a fixed but unknown distribution G , and we let ¯ g ( i ) ( a ) = E g ∼G [ g ( i ) ( a )] be the expected cost of action a for the i th constraint. On the other hand, in the adversarial setting { g t } t ∈ J T K is an arbitrary sequence of cost functions.

Let ∆ K to be the set of discrete probability distributions over the set J K K . Then, at round t ∈ J T K , given a randomized strategy x t ∈ ∆ K , the expected learner reward is ∑ a ∈ K f t ( a ) x t ( a ) = 〈 x t , f t 〉 .

J K Similarly, 〈 x t , g ( i ) t 〉 denotes the expected cost of the i th constraint. Finally, we use n t ( a ) to denote the number of times arm a was played up to time t , i.e., n t ( a ) = ∑ t τ =1 I ( a τ = a ) .

We want to design algorithms which achieve good performances in both the adversarial and the stochastic setting. As it is customary in the literature, we compare our learning algorithm with different benchmarks according to the setting.

Stochastic Benchmark In the stochastic setting, the constraints g ( i ) t are i.i.d. samples with mean ¯ g ( i ) and thus we consider as benchmark the best fixed randomized strategy that satisfies the constraints in expectation, which is a standard choice in bandits with constraints [11, 28, 16]. Formally, in the stochastic setting, we can define the feasible sets X glyph[star] i and X glyph[star] as follows:

Then, we can define the stochastic baseline as:

We naturally assume the existence of safe mixed strategies, i.e., that X glyph[star] = ∅ . This is equivalent to assume the existence of a randomized strategy x ∅ such that 〈 x ∅ , ¯ g ( i ) 〉 ≤ 0 for all i . Notice that this is a weaker assumption than the one commonly assumed by best-of-both-worlds algorithms in which 〈 x ∅ , ¯ g ( i ) 〉 ≤ -ρ , where ρ is a strictly positive constant (see, e.g., [16, 11, 10]).

glyph[negationslash]

Adversarial Benchmark In the adversarial setting, { g t } t ∈ J T K is an arbitrary sequence of constraints. We consider as benchmark the best unconstrained strategy:

While this baseline has already been used [e.g., 11, 13]), other works on adversarial bandit with constraints employ weaker baselines [e.g., 21, 16]. For instance, Castiglioni et al. [16] consider the best fixed strategy which is feasible on average. However, we show that, despite using a stronger baseline, we obtain a competitive ratio that is optimal even for the weaker baselines commonly adopted in the literature [16, 10, 9].

2.1 Best-Of-Both-Worlds Guarantees

Our goal is to design learning algorithms that exhibit optimal guarantees both in the stochastic and adversarial settings. In the stochastic setting, we are interested in minimizing the regret R T w.r.t. OPTS :

√

Algorithm 1

Require: bonuses b t ( a ) , weights w ( i ) t,a and parameter β

1: Initialize regret minimizer R with β

2: for each step t = 1 , . . . , T do

3: Estimation:

5: ̂ X ( i ) t ←{ x ∈ ∆ K : 〈 x, ˆ g ( i ) t -b t 〉 ≤ 0 }

4: ˆ g ( i ) t ( a ) ← ∑ τ ∈T t -1 ,a w ( i ) t,a ( τ ) g ( i ) τ ( a ) for all a ∈ J K K and i ∈ J m K

6: ̂ X t ←∩ i ∈ m ̂ X ( i ) t

J K 7: Regret minimization:

8: Get prediction from R on set ̂ X t : x t ←R ( ̂ X t )

9: Sample a t ∼ x t and receive { g ( i ) t ( a t ) } i ∈ J m K and f t ( a t )

and specifically we require both R T and V T to be in ˜ O ( √ T ) with high probability. This clearly matches the standard Ω( √ T ) lower bound that holds even without constraints [5].

In the (harder) adversarial setting, we pose the less ambitious goal of achieving a constant competitive ratio with respect to OPTA , or equivalently sublinear α -regret with constant α . Formally, given an α < 1 , we define the α -regret as:

As it is customary in the literature [15], the competitive ratio α obtained by our algorithms depends on the following Slater's parameter ρ :

The parameter ρ is related to the existence of strictly-feasible actions, and only depends on the constraints. Our definition is slightly stronger than the one in most previous works where the inf is over randomized strategies. To guarantee the existence of a feasible strategy we assume that ρ ≥ 0 . Then, our goal is to guarantee that both V T and the α -regret, with α = ρ / ρ +1 , belong to ˜ O ( √ T ) with high probability. Note that this matches the lower bound of Bernasconi et al. [11].

3 Our Approach

In this section, we present the main components of our algorithm, while the following sections will describe the specific components in details. We refer to Algorithm 1 for the pseudocode. At each step t , the algorithm works in two phases: i) it estimates the feasible set, and ii) it plays a strategy in the estimated set. Each phase requires a specific ingredient:

An estimator ˆ g ( i ) t of the costs functions g ( i ) t that is used together with the optimistic bonus b t to define the estimation of the feasible set defined as ̂ X t := ∩ i ∈ J m K ̂ X ( i ) t . In the stochastic case, we would like ̂ X t ⊇ X glyph[star] , while in the adversarial case our goal is to maintain a sequence of sets that always contains a version of the action set X , properly scaled around a ∅ (see Equation (2) for a formal definition).

ii) A regret minimizer R for adversarial linear reward function that, at each round, takes in input a convex set of feasible strategies ̂ X t ⊆ ∆ K , and then selects a strategy x t ∈ ̂ X t . We require the regret minimizer to achieve ˜ O ( √ KT ) regret with respect to any x ∈ ∩ t ∈ J T K ̂ X t ;

In the following we define the two phases more in details. Let T t,a := { τ ≤ t : a t = a } be the set of rounds in which the algorithm plays action a . Then, at each round t , Algorithm 1 computes the estimate

Algorithm 2 No Regret on Moving Sets

Require: Parameter β > 0

1: Set γ = β / 2

2: for each step t = 1 , . . . , T do

3: Receive ̂ X t

5: x t ← Π ̂ X t (ˆ x t ) := arg min x ∈ ̂ X t B ( x || ˆ x t )

4: ˆ x t ( a ) ← x t -1 ( a ) e β ( ˆ f t -1 ( a ) -1) for all a ∈ J K K

6: Sample a t ∼ x t

7: Observe f t ( a t ) and set ˆ f t ( a ) ← 1 for all a = a t and ˆ f t ( a t ) ← 1 -1 -f t ( a t ) x t ( a t )+ γ

glyph[negationslash]

as the weighted mean of available past observations { g ( i ) τ ( a ) } τ ∈ J t -1 K for each actions a ∈ J K K and constraint i ∈ J m K , for some weights w ( i ) t,a . 3 Then, the estimates together with the optimistic bonus { b t ( a ) } a ∈ J K K are used to define the moving sets ̂ X t , which are fed to the regret minimizer R which in turn selects a point x t ∈ ̂ X t .

glyph[negationslash]

One crucial property that is required for the execution of the regret minimizer R is that all the sets ̂ X t are non-empty (as otherwise the regret minimizer has no feasible strategies). To simplify exposition, in the following sections we assume that the clean event C := { ̂ X t = {∅} ∀ t ∈ J T K } holds. In Corollary 6.3, we prove that this event holds with high probability in the stochastic setting, while in Theorem 5.2 we argue that it holds deterministically in the adversarial one.

In Algorithm 1 we left unspecified two crucial parts of our approach. The first is how to build the regret minimizer R , and the second concerns how to actually generate the sets ̂ X t , i.e., the weights w ( i ) t,a and the bonus b t ( a ) . We delve into these details in Section 4 and Section 5, respectively.

4 No-regret on moving sets

We describe the regret minimizer R that exhibits no-regret with respect to any x ∈ ∩ t ∈ J T K ̂ X t . We achieve this via a simple modification to the EXP-IX algorithm of Neu [25] that provides high probability results for multi-armed bandits via implicit exploration. More specifically, our algorithm maintains a randomized strategy x t ∈ ∆ K which is updated using the biased reward estimate ˆ f t ( a ) as in Neu [25] and then projected onto ̂ X t according to the negative entropy Bregman divergence B ( x || y ) = ∑ a ∈ J K K [ x ( a ) log ( x ( a ) / y ( a ) ) -x ( a ) + y ( a )] . We refer to Algorithm 2 for the pseudocode, and present here the main result of the Section.

Theorem 4.1. Let x t be selected accordingly to Algorithm 2 run with arbitrary sequence of convex sets ̂ X t ⊆ ∆ K with γ = β 2 and β = √ log( K / δ 1 ) KT . Then, with probability at least 1 -δ 1 it holds that

This result establishes no-regret in the case of moving sets, taking as benchmark the optimal strategy in the intersection of all sets. To exploit this result in Algorithm 1, we have to make sure that in both the stochastic and adversarial setting the intersection of the sets ̂ X t contains 'good' strategies. In the stochastic setting, we show that with high probability it includes X glyph[star] , while, in the adversarial setting, it includes a strategy with utility ρ / 1+ ρ · OPTA .

5 How to build the sets ̂ X t

In this section, we show how to design estimations ̂ X t of the feasible sets that, surprisingly, are effective both in stochastic and adversarial settings. Indeed, the main challenge is to design sets ̂ X t

that accommodate the different requirements of the two settings. First, in Section 5.1, we discuss how to set the optimistic bonuses b t and then in Section 5.2 we focus on how to set the weights w ( i ) t,a .

5.1 How to set the optimistic bonus

The optimistic bonuses have the main purpose of balancing the estimation error in the stochastic setting. As the following lemma show, we simply need that | ˆ g ( i ) t ( a ) -¯ g ( i ) ( a ) | ≤ b t ( a ) with high probability. Indeed, this is sufficient to show that X glyph[star] ⊆ ∩ t ∈ J T K ̂ X t in the stochastic setting.

Theorem 5.1. Consider the stochastic setting. Given any δ > 0 , let b t ( a ) be such that with probability at least 1 -δ it holds:

glyph[negationslash]

Then, it holds X glyph[star] ⊆ ∩ t ∈ J T K ̂ X t with probability at least 1 -δ .

Even tough it is crucial in the stochastic setting, it turns out that in the adversarial setting the optimistic bonus b t is not really needed. Indeed, as we will show in the following, we are interested in obtaining no-regret with respect to the set X glyph[star] ∅ which is obtained via interpolation of points in X and the strictly feasible actions a ∅ . Let x ∅ be such that x ∅ ( a ∅ ) = 1 and x ∅ ( a ) = 0 for all a = a ∅ . Formally:

where A + B is the Minkowski sum between sets and αA indicates the set that contains each element of A multiplied by α . 4 The following theorem proves that X glyph[star] ∅ ⊆ ̂ X t for all t .

Theorem 5.2. In the adversarial setting, it holds X glyph[star] ∅ ⊆ ̂ X t for all t ∈ J T K .

Notice that having no-regret with respect to the set X glyph[star] ∅ is not sufficient to achieve no-regret in the adversarial setting. Nonetheless, we will show that this is sufficient to guarantee noα -regret, for α = ρ / 1+ ρ with respect to any strategy x ∈ ∆ K .

5.2 How to set the weights

We focus on the design of estimators ˆ g ( i ) t that are good approximations of the real functions g ( i ) t . Algorithm 1 computes the estimators ˆ g ( i ) t by using a weighted mean of all past observations:

However, to simplify the exposition, we use the following equivalence between online gradient descent ( OGD ) on quadratic losses ˆ g ( i ) t ( a t ) ↦→ 1 2 ( g ( i ) t ( a t ) -ˆ g ( i ) t ( a t ) ) 2 and weighted means. In particular this equivalence is realized by observing that such loss has gradient g ( i ) t ( a t ) -ˆ g ( i ) t ( a t ) .

Lemma 5.3. Given any sequence { y t } t ∈ J T K such that y 1 = 0 and any sequence of learning rates { η t } t ∈ J T K such that η 1 = 1 , let { ˆ y t } t ∈ J T K be the estimator updated as:

Then, it holds that ˆ y t = ∑ t -1 τ =1 y τ w t ( τ ) where w t ( τ ) = η τ ∏ t -1 k = τ +1 (1 -η k ) . Moreover, ∑ t -1 τ =1 w t ( τ ) = 1 for any t ≥ 2 .

Clearly, in the OGD interpretation of our update, we only update ˆ g ( i ) t ( a ) only when a t = a , and thus we only need to define learning rates for action a for the times t in which a t = a . Based on this observation, we are going to update ˆ g ( i ) t ( a ) as glyph[negationslash]

Thus, given an action a ∈ J K K and a time t ∈ J T K the corresponding weights { w ( i ) t,a ( τ ) } τ ∈T t -1 ,a are:

We now proceed to give two notable examples on how to instantiate the learning rates and recover commonly used estimators such as the empirical mean and the exponentially weighted mean. 5

Proposition 5.4. If η ( i ) t ( a t ) = 1 n t ( a t ) for each τ ∈ T t -1 ,a , then w ( i ) t,a ( τ ) = 1 n t -1 ( a ) and we recover the empirical mean estimator for ˆ g ( i ) t ( a ) = 1 n t -1 ( a ) ∑ τ ∈T t -1 ,a g ( i ) τ ( a ) .

Proposition 5.5. If η ( i ) t ( a t ) = η then

for each τ ∈ T t -1 ,a and we recover an exponentially weighted average estimator for ˆ g ( i ) t ( a ) .

As it will turns out, these are the two extreme cases that we want to interpolate between. Indeed, the empirical mean estimator is particularly effective in the stochastic case but ineffective in the adversarial case, while the converse happens with the exponentially weighted estimator.

Now, we show that the OGD interpretation is particularly useful to bounds the violations suffered by the algorithm. First, we define the violations in an interval [ t 1 , t 2 ] := { t ∈ J T K : t 1 ≤ t ≤ t 2 } as:

Then, in the following lemma we show that the violations in the interval are related to the variation of the estimates ˆ g ( i ) t ( a ) .

Theorem 5.6. Given an interval [ t 1 , t 2 ] ⊆ J T K , an i ∈ J m K , and a δ > 0 , with probability at least 1 -δ it holds:

By a simple telescoping argument, we have the following corollary, which holds whenever the learning rates are non-increasing within a time interval. Let glyph[lscript] ( a, [ t 1 , t 2 ]) be the last rounds in the interval [ t 1 , t 2 ] in which action a is played.

Corollary 5.7. Given an interval [ t 1 , t 2 ] ⊆ J T K , a i ∈ J m K , and a δ > 0 , assume that for any a ∈ J K K it holds η ( i ) τ ( a ) ≥ η ( i ) τ ′ ( a ) ∀ τ < τ ′ ∈ T t 2 ,a ∩ [ t 1 , t 2 ] . Then, with probability at least 1 -δ it holds:

Corollary 5.7 shows how to bound the violation as a function of the learning rates η ( i ) t and the bonus terms b τ . The following lemma shows how to bound the second term of the violations depending on the structure of the bonus terms.

Lemma 5.8. Given a c > 0 , an α ∈ (0 , 1) , a t ∈ J T K , and a δ > 0 , let b t ( a ) = c n t ( a ) α for all a ∈ J K K . Then, with probability at least 1 -δ , it holds:

In this section, we saw how the choice of the learning rates of the estimator affects the estimators. In the following section, we will see how to adaptively set those learning rates to handle both stochastic and adversarial settings.

6 Adaptive learning rates

The previous section highlights the main difficulties of obtaining best-of-both-world algorithms: we need to set the weights w ( i ) t,a (or equivalently - by Lemma 5.3 - the learning rates η ( i ) t ( a t ) ) and the optimistic bonuses b t so that they meet, at the same time, the requirements needed by the stochastic and the adversarial settings.

We start presenting two possible choices and show that they fail either in the stochastic or the adversarial setting. Then, we show how adaptive learning rates combine the strengths of both approaches. The first, natural, choice of setting the learning rate is to use an exponentially weighted estimator, i.e., choose η ( i ) t ( a t ) = 1 / √ T . With this choice, we can apply a weighted version of AzumaHoeffding inequality and find that | ˆ g ( i ) t ( a ) -¯ g ( i ) ( a ) | ∈ ˜ O ( n t ( a ) -1 / 4 ) , with high probability. Thus, as discussed in Section 5.1, we would need to define b t ( a ) ∈ ˜ O ( n t ( a ) -1 / 4 ) , which, by Corollary 5.7 and Lemma 5.8 would imply a suboptimal ˜ O ( T 3 / 4 ) rate for the violations.

The second option is to set η ( i ) t ( a t ) = 1 / n t ( a t ) . In the stochastic setting, we have an optimal rate of concentration of the terms | ˆ g ( i ) t ( a ) -¯ g ( i ) ( a ) | ∈ ˜ O ( n t ( a ) -1 / 2 ) as, by Proposition 5.4, this is equivalent to compute the empirical mean. However, this second option fails disastrously in the adversarial setting as highlighted in Corollary 5.7, where the first component of the violations becomes linear in T . Intuitively, a learning rate of order 1 / n t ( a ) makes the update of the estimates too slow when the underlying constraints change, as it does happen in the adversarial setting.

This trade-off forces us to employ adaptive learning rates . Our idea is to use learning rates of the order 1 / n t ( a ) with an adaptive multiplicative term that depends on the current violation of the constraint. Formally, we use learning rates:

where Γ ( i ) t is a bonus term defined as

and [ x ] b a := min(max( x, a ) , b ) is the clipping of x between a and b . Moreover, we set the exploration bonus as

The following theorem shows that such approach guarantees ˜ O ( √ KT ) violations in both adversarial and stochastic settings.

Theorem 6.1. Both in the stochastic and the adversarial setting, with probability at least 1 -2 mT 2 δ 2 it holds that

The previous theorem shows that this choice of learning rates is sufficient to guarantee optimal bounds on the violations. However, to achieve this result we are setting b t ( a ) ∈ ˜ O ( n t ( a ) -1 / 2 ) . As we showed in theorem 5.1, this requires a concentration on the estimates | ˆ g ( i ) t ( a ) -¯ g ( i ) t ( a ) | of the same magnitude (in the stochastic setting). This is crucially needed to ensure that the regret minimizer R provides the desired guarantees and that the event C defined in Section 3 actually holds with high probability.

Lemma 6.2. In the stochastic setting, with probability at least 1 -5 mKTδ 2 , it holds that:

The proof of the previous result relies on the fact that in the stochastic case the bonus Γ ( i ) t does not 'kick in' ensuring that η ( i ) t ( a ) = 1 / n t ( a ) . Thus, ˆ g ( i ) t is the empirical average of past observations. The previous result, together with Theorem 5.1 proves the following corollary.

Corollary 6.3. In the stochastic setting, with probability at least 1 -5 mKTδ 2 , it holds that X glyph[star] ∈ ̂ X t for all t ∈ J T K .

This proves that the clean event C holds with high probability, as promised in Section 3.

7 Putting everything together

Now, we have everything in place to easily prove the our main theorems. First, we define the parameters δ 1 = δ 1 ( glyph[epsilon1] ) and δ 2 = δ 2 ( glyph[epsilon1] ) in order to guarantee that our theorems hold with probability at least 1 -glyph[epsilon1] . In particular, we set δ 1 ( glyph[epsilon1] ) = glyph[epsilon1] / 2 , where we recall that δ 1 is the parameter used to set β and γ in Algorithm 2, and δ 2 ( glyph[epsilon1] ) = glyph[epsilon1] / (14 mKT 2 ) , where δ 2 is used to set the optimistic bonus and learning rate of Algorithm 1.

In the stochastic setting, the violation guarantees directly follow from Theorem 6.1, while the regret guarantee follows by combining Theorem 4.1 and Corollary 6.3. Formally:

Theorem 7.1. In the stochastic setting, for any glyph[epsilon1] > 0 Algorithm 1 guarantees that with probability at least 1 -glyph[epsilon1] :

Now, we turn to the adversarial setting. Theorem 6.1 guarantee ˜ O ( √ T ) violations even with adversarial constraint, while the regret guarantees follows by combining Theorem 5.2 and Theorem 4.1

Theorem 7.2. In the adversarial setting, for any glyph[epsilon1] > 0 Algorithm 1 guarantees that with probability at least 1 -glyph[epsilon1] :

where α = ρ / (1+ ρ ) .

Note that in both settings, the regret upper bound is of order ˜ O ( √ KT ) and it is independent from the number of constraints m , while the violations are of order ˜ O ( √ KT log( m )) and depend only logarithmically on m . This is in contrast to the other best-of-both-world algorithms for bandits with long term constraints, based on primal-dual methods, in which both the regret and the violations depends polynomially in m .

Another interesting characteristic of our methodology is that we guarantee an anytime bound on the constraint violation. Indeed, this matches the guarantees provided by the most recent primal-dual methods [11, 1] that, however, require weakly-adaptive underlying regret minimizers.

7.1 Convergence rate in the stochastic setting

To conclude, we point to a nice byproduct of our analysis. In the stochastic setting, we can easily prove a sort of 'convergence rate' of x t to the set X glyph[star] . Formally, we can prove that positive violations are bounded by ˜ O ( √ Kt log m ) as long as we consider expected violations. Let us define x + := max( x, 0) and

Then, we can state the following theorem:

Theorem 7.3. Algorithm 1, in the stochastic setting, guarantees that with probability at least 1 -glyph[epsilon1] , it holds that:

Intuitively, our result shows that our algorithm plays only a sublinear number of times 'far' from the set X glyph[star] , or that our algorithm plays a linear number of times 'close' to the set X glyph[star] . This is a much stronger result then just guaranteeing that V T is sublinear, as in that case it might be a linear number of times the algorithm plays 'far' from X glyph[star] as long as it plays strictly inside of X glyph[star] often enough.

Acknowledgments

MB, MC, AC, FF are partially supported by the FAIR (Future Artificial Intelligence Research) project PE0000013, funded by the NextGenerationEU program within the PNRR-PE-AI scheme (M4C2, investment 1.3, line on Artificial Intelligence). FF is also partially supported by ERC Advanced Grant 788893 AMDROMA 'Algorithmic and Mechanism Design Research in Online Markets', and PNRR MURproject IR0000013-SoBigData.it. MC is also partially supported by the EU Horizon project ELIAS (European Lighthouse of AI for Sustainability, No. 101120237). AC is partially supported by MUR- PRIN 2022 project 2022R45NBB funded by the NextGenerationEU program.

References

Gagan Aggarwal, Giannis Fikioris, and Mingfei Zhao. No-regret algorithms in non-truthful auctions with budget and ROI constraints. arXiv preprint , abs/2404.09832, 2024.

Shipra Agrawal and Nikhil R. Devanur. Bandits with concave rewards and convex knapsacks. In EC , pages 989-1006. ACM, 2014.

Shipra Agrawal and Nikhil R Devanur. Bandits with global convex constraints and objective. Operations Research , 67(5):1486-1502, 2019.

Peter Auer and Chao-Kai Chiang. An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits. In COLT , volume 49 of JMLR Workshop and Conference Proceedings , pages 116-120. JMLR.org, 2016.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, FOCS 2013 , pages 207-216. IEEE, 2013.

Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. J. ACM , 65(3), 2018.

Santiago Balseiro, Christian Kroer, and Rachitesh Kumar. Online resource allocation under horizon uncertainty. SIGMETRICS Perform. Eval. Rev. , 51(1):63-64, 2023.

Santiago R Balseiro and Yonatan Gur. Learning in repeated auctions with budgets: Regret minimization and equilibrium. Management Science , 65(9):3952-3968, 2019.

Santiago R Balseiro, Haihao Lu, and Vahab Mirrokni. The best of many worlds: Dual mirror descent for online allocation problems. Operations Research , 2022.

Martino Bernasconi, Matteo Castiglioni, and Andrea Celli. No-regret is not enough! bandits with general constraints through adaptive regret minimization. arXiv preprint , abs/2405.06575, 2024.

Martino Bernasconi, Matteo Castiglioni, Andrea Celli, and Federico Fusco. No-regret learning in bilateral trade via global budget balance. In STOC . ACM, 2024.

Martino Bernasconi, Matteo Castiglioni, Andrea Celli, and Federico Fusco. Bandits with replenishable knapsacks: the best of both worlds. In International Conference on Learning Representations (ICLR) , 2024.

Sébastien Bubeck and Aleksandrs Slivkins. The best of both worlds: Stochastic and adversarial bandits. In COLT , volume 23 of JMLR Proceedings , pages 42.1-42.23. JMLR.org, 2012.

Matteo Castiglioni, Andrea Celli, and Christian Kroer. Online learning with knapsacks: the best of both worlds. In International Conference on Machine Learning , pages 2767-2783. PMLR, 2022.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, Giulia Romano, and Nicola Gatti. A unifying framework for online optimization with long-term constraints. In Advances in Neural Information Processing Systems , volume 35, pages 33589-33602, 2022.

Matteo Castiglioni, Andrea Celli, and Christian Kroer. Online learning under budget and ROI constraints via weak adaptivity. In ICML . OpenReview.net, 2024.

Giannis Fikioris and Éva Tardos. Approximately stationary bandits with knapsacks. In Proceedings of Thirty Sixth Conference on Learning Theory , volume 195, pages 3758-3782, 12-15 Jul 2023.

Elad Hazan et al. Introduction to online convex optimization , volume 2. Now Publishers, Inc., 2016.

Nicole Immorlica, Karthik Abinav Sankararaman, Robert Schapire, and Aleksandrs Slivkins. Adversarial bandits with knapsacks. In 60th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2019 , pages 202-219. IEEE Computer Society, 2019.

Nicole Immorlica, Karthik Sankararaman, Robert Schapire, and Aleksandrs Slivkins. Adversarial bandits with knapsacks. J. ACM , 69(6), 2022. ISSN 0004-5411.

Thomas Kesselheim and Sahil Singla. Online learning with vector costs and bandits with knapsacks. In Conference on Learning Theory , pages 2286-2305. PMLR, 2020.

Raunak Kumar and Robert Kleinberg. Non-monotonic resource utilization in the bandits with knapsacks problem. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Shang Liu, Jiashuo Jiang, and Xiaocheng Li. Non-stationary bandits with knapsacks. Advances in Neural Information Processing Systems , 35:16522-16532, 2022.

Gergely Neu. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. Advances in Neural Information Processing Systems , 28, 2015.

Yevgeny Seldin and Gábor Lugosi. An improved parametrization and analysis of the EXP3++ algorithm for stochastic and adversarial bandits. In COLT , volume 65 of Proceedings of Machine Learning Research , pages 1743-1759. PMLR, 2017.

Yevgeny Seldin and Aleksandrs Slivkins. One practical algorithm for both stochastic and adversarial bandits. In ICML , volume 32 of JMLR Workshop and Conference Proceedings , pages 1287-1295. JMLR.org, 2014.

Aleksandrs Slivkins, Karthik Abinav Sankararaman, and Dylan J Foster. Contextual bandits with packing and covering constraints: A modular lagrangian approach via regression. In The Thirty Sixth Annual Conference on Learning Theory , pages 4633-4656. PMLR, 2023.

Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In COLT , volume 75 of Proceedings of Machine Learning Research , pages 1263-1291. PMLR, 2018.

Julian Zimmert and Yevgeny Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. J. Mach. Learn. Res. , 22:28:1-28:49, 2021.

A Further Related Works

Best-of-Both-Worlds. A long line of work has investigated Best-of-Both-Worlds algorithms for bandits without constraints. These algorithms aim to achieve an instance-dependent logarithmic regret bound in stochastic environments, while also ensuring the worst-case Θ( √ T ) regret bound that characterizes the adversarial settings [14, 4, 27, 26, 29, 30]. Although our focus is on the generation model of the constraints , our motivation in this paper is affine: retaining the best of the stochastic (sublinear regret with respect to the optimal dynamic policy) and adversarial world (tight competitive ratio with respect to the adversarial benchamrk). Furthermore, our idea of setting an adaptive learning rate that forces the learning algorithm to interpolate between an adversarial and a stochastic routine is reminescent of some of the techniques adopted in, e.g., Bubeck and Slivkins [14].

Bandits with Knapsacks. The (stochastic) BwK problem, where the rewards f t as well as the g i t are drawn i.i.d. from a non-negative distribution (so that the budget available for each resource can only decrease over time) is formally introduced and solved in Badanidiyuru et al. [6] (see also its journal version [7]). Agrawal and Devanur [2] studies a more general stochastic setting, which subsumes knapsack and exhibit optimal guarantees via optimism in the face of uncertainty [see also 3]. Moving to the adversarial BwK problem (which corresponds to our model when the g i t are all non-negative), an optimal solution is proposed in Immorlica et al. [20] [see also 21]; there, the authors propose the LagrangeBwK framework, which has a natural interpretation: arms can be thought of as primal variables, and resources as dual variables. The framework works by setting up a repeated two-player zero-sum game between a primal and a dual player, and by showing convergence to a Nash equilibrium of the expected Lagrangian game. Differently from the stochastic version, the adversarial BwK does not admit no-regret algorithms, but Θ(log T ) competitive ratio. In a subsequent work, [22] provides a new analysis obtaining a O (log m log T ) competitive ratio, which is optimal both in the time horizon T and in the number of resources m (and improves on the O ( m log T ) of Immorlica et al. [20, 21]). In the special case in which budgets are Ω( T ) , Castiglioni et al. [15] further improves the competitive ratio to 1 / ρ where ρ is the per-iteration budget.

More general constraints. Castiglioni et al. [15] studies a setting with general constraints, and show how to adapt the LagrangeBwK framework to obtain best-of-both-worlds guarantees when Slater's parameter is known a priori. Similar guarantees are also provided, in the stochastic setting, by Slivkins et al. [28], which then extend the results to the contextual model. Finally, Castiglioni et al. [17] introduces the use of weakly adaptive regret minimizers within the LagrangeBwK framework, and provides guarantees in the specific case of one budget constraint and one return-on-investments constraint.

Other related works. In an effort to bridge the results for adversarial and stochastic BwK , Fikioris and Tardos [18] investigates a data generation model that interpolate between the fully stochastic and the fully adversarial setting, depending on the magnitude of fluctuations in expected rewards and resources consumption across rounds. A similar effort is undertaken in Liu et al. [24], that study a non-stationary setting and provide no-regret guarantees against the best dynamic policy through a UCB-based algorithm. A recent line of work also investigates the natural situation where resources can be replenished in certain rounds (as also captured in our model) [23, 13, 12]. Finally, a related line of works is the one on online allocation problems with fixed per-iteration budget, where the input pair of reward and costs is observed before the learner makes a decision [10, 8].

B Proofs omitted from Section 4

Theorem 4.1. Let x t be selected accordingly to Algorithm 2 run with arbitrary sequence of convex sets ̂ X t ⊆ ∆ K with γ = β 2 and β = √ log( K / δ 1 ) KT . Then, with probability at least 1 -δ 1 it holds that

Proof. Let us define the negative entropy for a vector x ∈ R K ≥ 0 as:

and the Bregman divergence using Ψ can be written as

For the Bregman divergence it holds the following:

Claim B.1 ([19]) . For any z 1 , z 2 , and z 3 , it holds:

Moreover, given z , define z ′ = arg min ¯ z ∈K B (¯ z || z ) . Then:

At this point, is more convenient to work with losses rather then rewards. Define glyph[lscript] t ( a ) := 1 -f t ( a ) and ˆ glyph[lscript] t ( a ) := 1 -ˆ f t ( a ) . Note that:

glyph[negationslash]

Then, it is easy to verify that ∇ Ψ( x ) = log( x ) in which log( x ) has to be interpreted to be applied entry-wise. Simple calculations also show that β ˆ glyph[lscript] t = log( x t ) -log(ˆ x t +1 ) . Thus, we can apply Claim B.1 with z 1 = x, z 2 = x t and z 3 = ˆ x t +1 and this gives us the following:

Moreover using the second part of Claim B.1 in which z = ˆ x , z ′ = x t , ˜ z = x , and K = ̂ X t , we can conclude that B ( x || x t ) ≤ B ( x || ˆ x t ) . Notice that here we use x ∈ ̂ X t for each t . Then, we have the following chain of inequalities:

Combining the two we can find that:

Now we analyze the term B ( x t || ˆ x t +1 ) .

where, in the last inequality, we use that x t ( a ) / ( x t ( a )+ γ ) is at most 1 . Thus, by choosing x 1 ( a ) = 1 / K for all a , we have that B ( x || x 1 ) ≤ log( K ) and thus:

Form [25, Corollary 1 ] we know that with probability at least 1 -δ 1 we have:

Moreover, it is easy to verify that:

The regret is with probability at least 1 -δ 1 :

where in the last inequality we used that β = 2 γ . By taking β = √ log( K / δ 1 ) KT we obtain, that with probability at least 1 -δ 1 :

as desired.

C Proofs omitted from Section 5.1: How to set the optimistic bonus

Theorem 5.1. Consider the stochastic setting. Given any δ > 0 , let b t ( a ) be such that with probability at least 1 -δ it holds:

Then, it holds X glyph[star] ⊆ ∩ t ∈ J T K ̂ X t with probability at least 1 -δ .

Proof. In the following, we assume that the condition in the statement of the theorem holds. Hence, our result with hold with probability 1 -δ as promised. Let x ∈ X glyph[star] i . Consider a t ∈ J T K and an i ∈ J m K . Then, consider the following inequalities:

Thus, 〈 x, ˆ g ( i ) t -b t 〉 ≤ 0 which, by definition, proves that x ∈ ̂ X ( i ) t . This concludes the proof.

Theorem 5.2. In the adversarial setting, it holds X glyph[star] ∅ ⊆ ̂ X t for all t ∈ J T K .

Proof. In the adversarial setting, by Equation (1) we have that

for all t ∈ J T K and constraint i ∈ J m K . Moreover, for each t ∈ J T K , i ∈ J m K , and a ∈ J K K , it holds

and ∑ τ ∈T t -1 ,a w ( i ) t,a ( τ ) = 1 . Then, for all t ∈ J T K and constraint i ∈ J m K , ˆ g ( i ) t ( a ∅ ) ≤ -ρ and ˆ g ( i ) t ( a ) ≤ 1 for each a = a ∅ . 6 Thus, we can consider the following inequalities for any ˜ x ∈ X glyph[star] ∅ :

glyph[negationslash]

thus proving that ˜ x ∈ ̂ X t .

D Proofs omitted from Section 5.2: How to set the weights

Lemma 5.3. Given any sequence { y t } t ∈ J T K such that y 1 = 0 and any sequence of learning rates { η t } t ∈ J T K such that η 1 = 1 , let { ˆ y t } t ∈ J T K be the estimator updated as:

Then, it holds that ˆ y t = ∑ t -1 τ =1 y τ w t ( τ ) where w t ( τ ) = η τ ∏ t -1 k = τ +1 (1 -η k ) . Moreover, ∑ t -1 τ =1 w t ( τ ) = 1 for any t ≥ 2 .

Proof. The first part of the statement is trivial as it can be easily checked that:

Then, we prove the second part of the lemma by induction on t . The base case holds trivially as w 2 (1) = η 1 = 1 . Moreover, assuming ∑ t -2 τ =1 w t τ = 1 , it holds:

where in the second-to-last equality we use the inductive hypothesis. This concludes the proof.

Proposition 5.4. If η ( i ) t ( a t ) = 1 n t ( a t ) for each τ ∈ T t -1 ,a , then w ( i ) t,a ( τ ) = 1 n t -1 ( a ) and we recover the empirical mean estimator for ˆ g ( i ) t ( a ) = 1 n t -1 ( a ) ∑ τ ∈T t -1 ,a g ( i ) τ ( a ) .

Proof. Consider an a ∈ J K K , an i ∈ J m K , and a t ∈ J T K . Then, by applying Lemma 5.3 to the set of rounds T t -1 ,a we have that:

Now, we show that

and thus w ( i ) t,a ( τ ) = 1 n t -1 ( a ) ,

Theorem 5.6. Given an interval [ t 1 , t 2 ] ⊆ J T K , an i ∈ J m K , and a δ > 0 , with probability at least 1 -δ it holds:

Proof. First, applying Lemma G.1, we have that with probability 1 -δ it holds:

Consider the following chain of inequalities:

where the last equality follows by the definition of the update:

This concludes the proof.

Corollary 5.7. Given an interval [ t 1 , t 2 ] ⊆ J T K , a i ∈ J m K , and a δ > 0 , assume that for any a ∈ J K K it holds η ( i ) τ ( a ) ≥ η ( i ) τ ′ ( a ) ∀ τ < τ ′ ∈ T t 2 ,a ∩ [ t 1 , t 2 ] . Then, with probability at least 1 -δ it holds:

Proof. We assume that Theorem 5.6 holds, and hence our statement holds with probability 1 -δ . Then, to prove the statement it is sufficient to show that

Fix any a ∈ J K K , and let k = |T t 2 ,a ∩ [ t 1 , t 2 ] | be the number of times action a is played in the interval [ t 1 , t 2 ] . Moreover, let τ ( j ) be the rounds in which action a is played the j -th time in the interval [ t 1 , t 2 ] . Then:

Summing over all the actions we obtain the desired inequality.

Lemma 5.8. Given a c > 0 , an α ∈ (0 , 1) , a t ∈ J T K , and a δ > 0 , let b t ( a ) = c n t ( a ) α for all a ∈ J K K . Then, with probability at least 1 -δ , it holds:

Proof. Consider the following inequalities:

The proof is concluded by using Lemma G.1.

E Proofs omitted from Section 6

Theorem 6.1. Both in the stochastic and the adversarial setting, with probability at least 1 -2 mT 2 δ 2 it holds that

Proof. We prove that given an i ∈ J m K , it holds:

with probability 1 -2 T 2 δ 2 . Then, a union bound over i completes the proof.

Given an i ∈ J m K , we first assume some high-probability events. In particular, we assume that Corollary 5.7 with δ = δ 2 holds for any interval, and that Lemma 5.8 with δ = δ 2 holds for all t ∈ J T K . This happens with probability at least 1 -2 T 2 δ 2 . Weconsider two cases. If V ( i ) t ≤ 53 √ KT log( 2 / δ 2 ) for all t ∈ J T K , then the statement it is trivially satisfied. Otherwise, there exists an a time ¯ t for which V ( i ) ¯ t ≥ 53 √ Kt log( 1 / δ 2 ) . Clearly, this implies that there exists a t < ¯ t such that V ( i ) t ≥ 42 √ Kt log( 2 / δ 2 ) for all t ∈ [ t, ¯ t ] and V ( i ) t -1 ≤ 42 √ Kt log( 1 / δ 2 ) . Since V ( i ) t ≥ 42 √ Kt log( 1 / δ 2 ) for all t ∈ [ t, ¯ t ] we have that:

and thus Γ ( i ) t = 21 √ Kt log( 1 / δ 2 ) for all t ∈ [ t, ¯ t ] . Hence, on the interval t ∈ [ t, ¯ t ] we known that the learning rate can be lower bounded by a non-increasing function of time as

This let us use Corollary 5.7 (that we assumed to hold) to show that:

Now, V ( i ) ¯ t ≤ V t + V ( i ) [ t , ¯ t ] ≤ (42 + 1 / 10 +10) √ Kt log( 2 / δ ) < 53 √ Kt log( 2 / δ ) . We thus reached a contradiction and there is no such a ¯ t . The union bound on all i ∈ J m K concludes the proof.

Lemma 6.2. In the stochastic setting, with probability at least 1 -5 mKTδ 2 , it holds that:

Proof. First, we show some concentration inequalities that will be useful in the following. By an Hoeffding's inequality and an union bound with probability at least 1 -mKTδ 2 , it holds:

Moreover, by Lemma Lemma G.1 and an union bound, with probability at least 1 -mTδ 2 , it holds:

Similarly, by Lemma Lemma G.1 and an union bound, with probability at least 1 -mTδ 2 , it holds:

By Lemma G.2 and an union bound, with probability at least 1 -mTδ 2

Finally, by Lemma 5.8 and an union bound, with probability 1 -Tδ 2 , it holds:

In the following, we will assume the the previous events hold, and hence our result holds with probability at least 1 -5 mKTδ 2 .

First, we show that V i t ≤ 21 √ Kt log( 2 / δ 2 ) for each t and i . Our proof works by induction on t . Clearly, the inequality holds for t = 1 . Now, assume that it holds for all τ ≤ t -1 . By the definition of Γ ( i ) τ , the induction assumption implies that η ( i ) τ ( a ) = 1 n τ ( a ) for all a ∈ J K K , i ∈ J m K and τ ≤ t -1 . Then, thanks to Proposition 5.4 we have that:

Hence, by Equation (10), it holds that:

and thus that ̂ X ( i ) τ = {∅} for all τ ≤ t -1 . Assuming that the events above holds, consider now the following inequalities:

glyph[negationslash]

and thus V ( i ) t ≤ 21 √ Kt log( 2 / δ 2 ) .

Thus Γ ( i ) t = 0 and ˆ g ( i ) t ( a ) is the empirical mean of past observations. This concludes the induction step, showing that V i t ≤ 21 √ Kt log( 1 / δ 2 ) for all t ∈ J T K , and Γ ( i ) t = 0 for all t ∈ J T K and i ∈ J m K .

Now, we proved that with probability 1 -3 mKTδ 2 , all Γ ( i ) t = 0 , and hence by Equation (10) we have that:

as desired.

F Proofs omitted from Section 7

Theorem 7.1. In the stochastic setting, for any glyph[epsilon1] > 0 Algorithm 1 guarantees that with probability at least 1 -glyph[epsilon1] :

Proof. To prove the upper bound on the regret, we simply have to combine Corollary 6.3 with Theorem 4.1. By Corollary 6.3 which probability at least 1 -5 mKTδ , it holds X glyph[star] ⊆ ∩ t ∈ J T K ̂ X t .

Moreover, by Theorem 4.1, we have that for each x ∈ X glyph[star] with probability at least 1 -δ 1 :

Let x glyph[star] = arg max x ∈X glyph[star] ∑ t ∈ J T K 〈 x, f t 〉 . Then, by union bound we have that with probability at least 1 -5 mKTδ 2 -δ 1 it holds:

proving the bound on the regret. The bound on the violations holds with probability at least 1 -2 mT 2 δ 2 by Theorem 6.1, and guarantees:

By an union bounds on all events, the guarantees hold with probability at least 1 -7 mKT 2 δ 2 -δ 1 . Thus by taking δ 1 = glyph[epsilon1] / 2 and δ 2 = glyph[epsilon1] / (14 mKT 2 ) we obtain the desired result.

Theorem 7.2. In the adversarial setting, for any glyph[epsilon1] > 0 Algorithm 1 guarantees that with probability at least 1 -glyph[epsilon1] :

where α = ρ / (1+ ρ ) .

Proof. Combining Theorem 5.2 and Theorem 4.1 readily proves that with probability at least 1 -δ 1 we have that for all ˜ x ∈ X glyph[star] ∅ ⊆ ̂ X t , we have:

glyph[negationslash]

Let x glyph[star] = arg max x ∈ ∆ K ∑ t ∈ J T K 〈 x, f t 〉 . Then, observe that ¯ x = 1 1+ ρ x ∅ + ρ 1+ ρ x ∗ ∈ X ∅ , where x ∅ ( a ∅ ) = 1 and x ∅ ( a ) = 0 for each a = a ∅ . Then, we have that:

since f t ( a ∅ ) ≥ 0 . This proves that with probability at least 1 -δ 1 :

Similarly to the proof of Theorem 7.1, we can prove that the bound on the violations holds with probability at least 1 -2 mT 2 δ 2 by Theorem 6.1, and give:

Overall these events hold with probability at least 1 -2 mT 2 δ 2 -δ 1 . By defining δ 1 = glyph[epsilon1] / 2 and δ 2 = glyph[epsilon1] / (14 mKT 2 ) we have that the desired results hold with probability at least 1 -glyph[epsilon1] .

Theorem 7.3. Algorithm 1, in the stochastic setting, guarantees that with probability at least 1 -glyph[epsilon1] , it holds that:

Proof. Define for each i ∈ J m K and t ∈ J T K

Then, given an i and a t consider the following chain of inequalities:

where last inequality both hold with probability 1 -5 mKTδ 2 jointly for each i and t .

Since b t = √ 2 log( 2 / δ 2 ) n t -1 ( a ) we can apply Lemma 5.8 and an union bound on all t to find that with probability at least 1 -Tδ 2 -5 mKTδ 2 :

Thus, we can conclude that:

K with probability at least 1 -6 mKTδ 2 . Recalling that δ 2 = glyph[epsilon1] / (14 mKT 2 ) we obtain the result.

G Further technical lemmas

Lemma G.1. For any sequence of function r t : J K K → [ -1 , 1] which is t -1 predictable and any sequence of randomized strategy x t ∈ ∆ K , it holds that with probability at least 1 -δ :

Proof. By definition E a ∼ x t [ r t ( a )] = ∑ a ∈ J K K r t ( a ) x t ( a ) = 〈 x t , r t 〉 . Thus the sequence X t = ∑ t τ =1 [ r τ ( a τ ) -〈 x τ , r τ 〉 ] is a martingale and | X t -X t -1 | ≤ 2 . Thus we can apply Azuma inequality and find that with probability at least 1 -δ :

Lemma G.2. For any sequence of randomized strategy x t ∈ ∆ K and any function ¯ r ( a ) such that r t ( a ) are sampled from a distribution with mean ¯ r ( a ) , i.e., E [ r t ( a )] = ¯ r ( a ) and P ( | r t ( a ) | ≤ 1) = 1 , it holds that with probability at least 1 -δ :

Proof. This holds by simple application of Azuma's inequality, similarly to the proof of Lemma G.1.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We included the main contributions and scope in the abstract

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, the paper discusses limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are explicitly discussed and all the proofs are provided.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper is theoretical and we do not have any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper is theoretical and we do not have any experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper is theoretical and we do not have any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper is theoretical and we do not have any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper is theoretical and we do not have any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conforms.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is theoretical without any immediate societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No data has been used in this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No data has been used in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets have been introduced in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing have been used in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing have been used in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article tackles the problem of multi-armed bandits with general constraints: at each time step, the learner receives a reward and $m$ costs, corresponding to $m$ independent constraints, and its goal is to maximize the reward while maintaning the cumulative cost corresponding to each constraint at a sub-linear level in the time horizon. To choose the action played at each time step (in the sense of the probability to play each arm), the authors propose an adaptation of Exp-IX, that chooses an action in an optimistic feasible set. Their main contribution consists in proving *best-of-both-worlds* guarantees for this algorithm *without knowing* a slackness parameter, usually assumed as known in the related literature. In this setting, best-of-both world guarantees consist in establishing sub-linear regret with respect to the best feasible probability on one side, and establishing simultaneously a competitive ratio determined by the slackness parameter (which is, again, not known and thus not used by the algorithm) on the other hand, if rewards are adversarial. This is the first result of this kind without knowledge of the Slater’s parameter.\n\nSTRENGTHS:\nI found the paper very clear and well written. The setting and the algorithm are well presented, as well as the results. Clear intuitions are provided, especially in the presentation of the technical results.\n\nWEAKNESSES:\nIn my opinion, there is a major issue with the results,which is that they are obtained with a modified definition of the Slater’s parameter. As stated by the authors, in the literature it is generally defined with the set of probabilities allocated to each arm, while in the paper it is defined with the arm set. While the author claim that this definition is « slightly stronger », I disagree with this claim. It seems to me that on the contrary the problem becomes much simpler if we know in advance a set of K atoms in the simplex that contain for sure a safe policy (even the safest in a sense). To me, this seems like trading a strong assumption (knowing the Slater’s parameter) by another strong assumption. This assumptions also considerably restricts the class of problems that can be considered. Furthermore, from my current understanding of the arguments it seems that the assumption is not just an artifact of the analysis and that the results would not be guaranteed to work if the infimum in the Slater’s condition was achieved by a mixed strategy.\n\nIn itself, I find the paper interesting despite this limitation, but I believe that this should be discussed more thouroughly because it reduces the potential impact of the paper.\n\nQUESTIONS:\n* Please elaborate on the definition of the Slater’s condition. In particular, I am interested in understanding if the definition is indeed necessary for the guarantees to hold of if it is an artifact of the analysis.\n\n* You claim that the competitive ratio matches the lower bound obtained in another paper. However, this seems inexact to me because the lower bound do not consider the same definition of the slackness parameter. I am missing something?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of multi-armed bandits under long-term constraints.\nThey give an algorithm with regret and violation guarantees simultaneously for both stochastic and adversarial constraints (i.e. best-of-both-worlds guarantees).\nSpecifically, they guarantee $\\tilde{O}(\\sqrt{T})$ regret and violation under stochastic constraints (without Slater's condition), as well as $\\tilde{O}(\\sqrt{T})$ $\\alpha$-regret and violation under adversarial constraints (with Slater's condition with parameter $\\rho$ where $\\alpha=\\frac{\\rho}{\\rho +1}$).\n\nSTRENGTHS:\n- The paper appears to improve on [11] in that it's regret scales logarithmic with the number of constraints and doesn't require Slater's condition for stochastic constraints.\n- The exposition is clear and there are nice insights from the theoretical results.\n\nWEAKNESSES:\nMy main concern is that the paper is missing key literature that render the contributions rather limited.\nMore details in the following:\n- [R1] gave a similar algorithm that uses an \"optimistic set\" to overestimate the constraint and then playing regret minimizer within this set. They showed $\\tilde{O}(\\sqrt{T})$ regret and violation without Slater's condition. In light of this, it appears the only contribution is extending [R1] to adversarial costs (and more restrictive finite action set). Although useful, this result is more limited than claimed.\n- The line of literature on stochastic bandits with round-wise constraints (e.g. [R2],[R3],[R4]) is directly relavant because it considers stochastic constraints with stronger constraint satisfaction guarantees (i.e. no violation).\n- The abstract claims that their particular problem setting generalizes Bandits with Knapsacks, which is unclear to me. Without this being the case, it seems that the only improvement on the literature would be for the recent arXiv paper [11].\n\n[R1] Gangrade et al, \"Safe Linear Bandits over Unknown Polytopes\", COLT 2024 (first appeared on arXiv in 2022 as 2209.13694).\n\n[R2] Amani et al, \"Linear Stochastic Bandits Under Safety Constraints\", NeurIPS 2019.\n\n[R3] Pacchiano et al, \"Stochastic Bandits with Linear Constraints\", AISTATS 2021.\n\n[R4] Moradipari et al, \"Safe Linear Thompson Sampling With Side Information\", IEEE TSP 2021.\n\nQUESTIONS:\n- Could you provide more clarification on how this setting generalizes Bandits with Knapsacks?\n- Given the references I gave, could you provide any insight on how this work contributes to this literature?\n\nLIMITATIONS:\nI did not see clear discussion of limitations. See the \"Weaknesses\" section for my discussion on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the bandit with constraints problem where the authors consider two possible settings for the constraints -- one where the constraints are stochastic, sampled i.i.d. from some unknown distribution, and one where the constraints are adversarial. The rewards are always assumed to be generated by an oblivious adversary. The authors present a novel algorithm which is able to achieve the optimal competitive ratio, up to O(\\sqrt{T}) regret in the constraints and objective, in the adversarial constraints setting and O(\\sqrt{T}) regret in the constraints and objective in the stochastic setting which is also min-max optimal.\n\nSTRENGTHS:\nTo the best of my knowledge the proposed algorithms and regret bounds are novel. The results are nearly min-max optimal (up to small logarithmic factors in K). The presentation of the work is overall good and there seems to be good technical novelty in developing the algorithms. The challenge in this work for achieving best of both worlds regret for the constraints is in constructing the right empirical estimators for the constraints. This is explained well in Section 6. In particular there is tension between tightly bounding V_t and having an estimator which constraints well in the stochastic setting. The main novelty seems to be the choice of adaptive step size in the definition of the empirical constraints which leads to the neat proof of Lemma 6.2 and Theorem 6.1.\n\nWEAKNESSES:\nI do not find any major weaknesses in this work. Perhaps the authors can do slightly better in highlighting the tension between bounding V_t and concentration of the empirical estimators. It was also not entirely clear to me what is the benefit of viewing the estimator update as a variant of gradient descent.\n\nMinor typos:\nline 517: This let us...\nline 517: The underscored t subscript is larger\nEquation 10: \\bar g^{i}a -- brackets are missing\n\nQUESTIONS:\nIs it possible to show some type of gap-dependent regret guarantees for the regret of the constraint violation using the same type of estimator? Intuitively it seems like there would still be fast enough concentration of the empirical estimator of the constraints, so that infeasible actions are eliminated quickly in case of a large gap, however, it's not completely obvious how to handle some parts of bound for V_t.\n\nLIMITATIONS:\nLimitations are appropriately addressed and there is no obvious negative societal impact as this work is theoretical.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies multi-armed bandits with cumulative cost connstraints, with the focus of designing a 'best of both worlds' algorithm that has small constraint violations in both the stochastic and adverasarial settings (where the constraints as well as rewards can vary with time arbitrarily), and attain either low-regret for the stochastic setting, or low $alpha$-regret for the adversarial setting (i.e., ensure that the reward accrued is at least $\\alpha$ times the best-in-hindsight reward accrued by a constant unconstrained assignment probability x, up to $o(T)$ terms). The main desiderata is to design methods that avoid explicit knowledge of a slater parameter, and avoid polynomial dependence on m, the number of unknown constraints, in the regret and violations, both of which are cons of prior work using the  Langrangian BwK design.\n\nThroughout, the setup adopted is linearised, i.e., at each time, the authors select a distribution $x_t$ over the actions [1:K], and the action $a_t$ is selected by sampling from $x_t$. A bandit feedback of both the reward, $f_t(a)$, and the constaint violations $g_{t}^i(a), i \\in [1:m]$ is assumed, where these functions are stochastic perturbations of an expected function in the stochastic setting, and are selected arbitrarily in the adversarial setting. Throughout, it is assumed that $f_t \\in [0,1]$ and $g\\_t^i \\in [-1,1]$. The constraint is of the form $\\forall i, g_t^i \\le 0$, Net violations are defined as $V_T = \\max_{i} V_T^{i}, $ where $V_T^i = \\sum_t g_t^i(a_t).$ Note that this equals $\\sum_t \\langle x_t, g_t^i\\rangle,$ up to a $\\sqrt{T}$ concentration term. In the adverarial case, the paper studies the regret $\\frac{\\rho}{1+\\rho} \\max_{x} \\sum f_t(x) - \\sum f_t(a_t),$ where $\\rho$ is the minimax Slater parameter $\\rho:=  \\max_a \\min_{i,t} (-g_{t}^i(a))$. \n\nThe overall strategy adopted by the paper is very natural: for each time, the method constructs a set of \"plausibly\" feasible distributions as $\\widehat{\\mathcal{X}}\\_t = \\{x \\in \\Delta\\_K : \\forall i, \\langle x, \\hat{g}\\_t^i - b\\_t\\rangle \\le 0\\}$, where $\\hat{g}\\_t^i$ serve as \"estimates\" for $g\\_t^i,$ and $b$ is a nonnegative 'bonus' that ensures optimism in the stochastic scenario. The method then just selects actions from $\\widehat{\\mathcal{X}}\\_t$, using a (small modification of) the EXP-IX strategy, thus ensuring low regret relative to any constant $x \\in \\bigcap \\widehat{\\mathcal{X}}\\_t$. The main results then need to select $\\hat{g}_t^i$ and $b_t$ in such a way that \n\n1. In the stochastic case, $\\mathcal{X}^* = \\{x : \\forall i, \\langle \\mathbb{E}[g_t^i], x\\rangle \\le 0\\}$ remains within this intersection of $\\widehat{\\mathcal{X}}s$, and  \n2. In the adverasrial case, $\\frac{\\delta\\_{a^{\\emptyset}}}{1+\\rho} + \\frac{\\rho}{1+\\rho} \\mathcal{X}$ remains within the same.\n3. The net violation is small.\n\nAs is natural, the $b_t(a)$ is set to just $\\sqrt{ \\log(mT^2/\\delta)/n_{t}(a)},$ where $n_t(a)$ is the number of times action $a$ has been played up to time $t$. The $\\hat{g}\\_t^i$ are estimated using OGD as $$ \\hat{g}\\_{t+1}^i(a)  =  \\hat{g}\\_t^i(a)\\mathbf{1}\\{a\\_t \\neq a\\} + ( (1-\\eta\\_{t}^i(a) \\hat{g}\\_t^i(a) + \\eta\\_t^i(a) g\\_t^i(a))\\mathbf{1}\\{a\\_t = a\\}.$$ The chief question becomes how to select the learning rates $\\eta\\_t^i(a)$. The basic tension is illustrated thus: in the stochastic case, we'd like $\\eta\\_t^i$ to be roughly $1/n\\_{t}(a),$ so that $\\hat{g}\\_t^i(a)$ is essentially the empirical average of the feedback. However, in the adversarial case, such a rate would be way too slow, and one needs quicker adaptation. The authors elegantly resolve this dilemma by setting $ \\eta\\_t^{i}(a) =(1+\\Gamma\\_t^i)/n\\_t^i,$ where $$ \\Gamma\\_t^t = \\min( \\zeta\\_t , \\max( 0, V\\_t^i - \\zeta\\_t )), \\textrm{ where } \\zeta\\_t = 21\\sqrt{Kt \\log(mT^2/\\delta) }$$\n\nFor the stochastic case, the analysis directly shows that, whp, $V\\_t^i$ never crosses the $21\\sqrt{Kt}$ threshold, and thus the desired $\\eta\\_t = 1/n\\_t(a)$ is retained. For the adversarial case, the analysis proceeds by first controlling the violation over any interval in terms of the updates to $\\hat{g}\\_t$ over the same, and then arguing via contradition by showing that that if $V_\\tau/\\sqrt{K\\tau}$ is ever too large, then $\\eta_t$ is large over a significantly sized interval before it, which in turn implies strong adaptation and a small $V_\\tau$. Finally, the point 1 follows by the choice of $b_t(a) \\sim 1/\\sqrt{n\\_t(a)}$, and 2 follows directly from the boundedness of the $g\\_t^i$, by interpreting $\\hat{g}_t^i$ as a weighted sum of the observed $g\\_t^i(a_t)$s with net weight $1$.\n\nSTRENGTHS:\nThe subject of the paper is of course very peritnent to the online learning subcommunity at Neurips. I think the technical contribution of the paper, i.e., the design of a BOBW method for the bandits with cumulative constraints that avoids knowledge of Slater parameters is quite interesting. Furthermore, the removal of the poly(m) factors from the ensuing bounds is an important improvement in the same.\nexplained\nI find the paper to be quite well written. The thought process behind the design of the method is well explained, and the intuition behind the design of the learning rate is clear. Prior work is contextualised well, and the contribution of the paper is clearly outlined. To my understanding, the proofs are correct.\n\nThe design of the scheme, and the analysis, are simple, but well motivated and elegant. Altogether, while I don't see this paper as especially groundbreaking, I do find it to be a well executed contribution that develops a simple idea and does interesting things with it to clean up the theory of bandits with cumulative constraints.\n\nWEAKNESSES:\nPerhaps the main weakness is that the structure of $\\Gamma\\_t^i$ bakes in the large constant 21 (and this of course leads to the large constant 53 in the bound). I find it natural to wonder if this constant is necessary, or if can be set in a path-dependent way that avoids both the large constant, and potentially offers improvements if the $g$s are somehow 'nice'. That said, I think that such a lacuna is not a major weakness, and such questions are find being left to follow-up work.\n\nQUESTIONS:\nComment: Comment: the roundiwse violation bound on V_T^+ of theorem 7.3 has previously been observed for the stochastic setup in the \"safe bandit\" literature, and this point of contact should perhaps be acknoweldged. See, e.g., [1,2]. The method these papers study is Agarwal and Devanur's style of \"doubly-optmistic\" selection.\n\n[1]: Chen et al., Strategies for Safe Multi-Armed Bandits with Logarithmic Regret and Risk, ICML 22\n[2]: Gangrade et al., Safe Linear Bandits over Unknown Polytopes, COLT 24\n\nLIMITATIONS:\nThis is fine.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-29"}