Title: Contextual Bandits with Knapsacks beyond Worst Cases via Re-Solving
Venue: NeurIPS 2023
Accepted: no
Keywords: Contextual bandits with knapsacks

Re-solving

Regularity
Abstract: Contextual Bandits with Knapsacks (CBwK) is a fundamental and essential framework for modeling a dynamic decision-making scenario with resource constraints. 
Under this framework, an agent selects an action in each round upon observing a request, leading to a reward and resource consumption that are further associated with an unknown external factor. 
The agent's target is to maximize the total reward under the initial inventory. 
While previous research has already established an $\widetilde{O}(\sqrt{T})$ worst-case regret for this problem, this work offers two results that go beyond the worst-case perspective, one for worst-case locations, and another for logarithmic regret rates.
We start by demonstrating that the unique-optimality and degeneracy of the fluid LP problem, which is both succinct and easily verifiable, is a sufficient condition for the existence of an $\Omega(\sqrt{T})$ regret lower bound. 
To supplement this worst-case location result, we merge the re-solving heuristic with distribution estimation skills and propose an algorithm that achieves an $\widetilde{O}(1)$ regret as long as the fluid LP has a unique and non-degenerate solution. 
This condition is mild as it is satisfied for most problem instances. 
Furthermore, we prove our algorithm maintains a near-optimal $\widetilde{O}(\sqrt{T})$ regret even in the worst cases, and extend these results to the setting where request and external factor are continuous. 
Regarding information, our regret results are obtained under two feedback models, respectively, where the algorithm accesses the external factor at the end of each round and at the end of a round only when a non-null action is executed. 

Abstract

Contextual Bandits with Knapsacks (CBwK) is a fundamental and essential framework for modeling a dynamic decision-making scenario with resource constraints. Under this framework, an agent selects an action in each round upon observing a request, leading to a reward and resource consumption that are further associated with an unknown external factor. The agent's target is to maximize the total reward under the initial inventory. While previous research has already established an ˜ O ( √ T ) worst-case regret for this problem, this work offers two results that go beyond the worst-case perspective, one for worst-case locations, and another for logarithmic regret rates. We start by demonstrating that the unique-optimality and degeneracy of the fluid LP problem, which is both succinct and easily verifiable, is a sufficient condition for the existence of an Ω( √ T ) regret lower bound. To supplement this worst-case location result, we merge the re-solving heuristic with distribution estimation skills and propose an algorithm that achieves an ˜ O (1) regret as long as the fluid LP has a unique and non-degenerate solution. This condition is mild as it is satisfied for most problem instances. Furthermore, we prove our algorithm maintains a near-optimal ˜ O ( √ T ) regret even in the worst cases, and extend these results to the setting where request and external factor are continuous. Regarding information, our regret results are obtained under two feedback models, respectively, where the algorithm accesses the external factor at the end of each round and at the end of a round only when a non-null action is executed.

1 Introduction

In the contextual bandits problem with knapsack constraints (CBwK problem for short), an agent is required to make sequential decisions over a finite time horizon to maximize the accumulated reward under initial resource constraints. To be more specific, in each round t = 1 , · · · , T , a request θ t and an external factor γ t are independently generated from two distributions, and only θ t is revealed to the agent. Based on the request, the agent should irrevocably choose an action a t , which would result in a reward r ( θ t , a t , γ t ) and a consumption vector c ( θ t , a t , γ t ) of resources. The agent's target is to optimize the sum of rewards ∑ T t =1 r ( θ t , a t , γ t ) before the resources are depleted.

The CBwK problem presents two key challenges when compared to closely related problems (e.g., network revenue management problem) : (1) choices are made without observing external factors, and (2) distributions of requests and external factors are unknown . However, the complexity of CBwK makes it a suitable mathematical abstraction for many real-life scenarios, such as dynamic bidding in repeated second-price auctions with budgets [Balseiro et al., 2021, Balseiro and Gur, 2019]. In this circumstance, an advertiser (the agent) acquires the value of the ad slot (the request) at the start of each auction, and would choose a bid (the action) accordingly. The agent's utility and payment in

this auction, as a consequence, are collaboratively determined by the value, the bid, and the highest 36

competing bid (the external factor). It is to be noted that the highest competing bid is inaccessible 37

to the agent before committing to the bid, as all advertisers bid simultaneously. Meanwhile, its 38

is ˜ O ( √ T ) when the initial resources are linearly proportional to the horizon length T [Slivkins and 43 Foster, 2022, Han et al., 2022]. 1 However, it is still unclear where worst-case scenarios occur, 44 meaning under which condition(s) an ˜ Ω( √ T ) regret is inevitable. Furthermore, we do not know 45 whether we can achieve a better regret guarantee for the CBwK problem beyond worst-case scenarios. 46 In particular, can we design algorithms to obtain an o ( √ T ) regret only under mild assumptions that 47 hold for almost all possible CBwK instances? This work takes the first step in addressing these 48 questions. 49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

1.1 Our Contributions

This work mainly makes three contributions, summarized as follows.

A precise sufficient condition for an ˜ Ω( √ T ) regret lower bound. To move beyond worst-case analysis, we establish a precise sufficient condition for the ˜ Ω( √ T ) regret lower bound to hold. Specifically, we demonstrate that when the fluid benchmark (also known as the deterministic LP) has a unique and degenerate solution, then an Ω( √ T ) regret is unavoidable for any online decision strategy (Theorem 2.1). While Han et al. [2022] have also provided a regret lower bound result for the CBwK problem, their condition depends on the inseparability of the possible expected reward/consumption function set. In other words, their condition may not perform well when this feasible set is small. Furthermore, their condition is rather complicated to verify. In contrast, our condition only depends on the underlying problem instance, is concise, and is easy to check. The proof of our result extends the approach of Vera and Banerjee [2021] to the CBwK problem.

An ˜ O (1) regret via re-solving under mild assumptions with full/partial information feedback. With the above result, we investigate how well an online algorithm can perform beyond worst cases, by applying the re-solving heuristic in conjunction with distribution estimation techniques, as given in Algorithm 1. Although this method has been considered in the problem of bandits with knapsacks (BwK) [Flajolet and Jaillet, 2015], to the best of our knowledge, we are the first to extend this method to the CBwK problem, which poses new challenge as decisions should be made according to the request. To avoid worst cases, we explicitly suppose that the fluid problem has a unique and non-degenerate solution (Assumption 3.1). This assumption is mild in three aspects: (1) it captures almost all CBwK problem instances, as slightly perturbing any LP can help it satisfy the unique optimality and non-degeneracy conditions; (2) it is almost necessary for an o ( √ T ) regret bound to establish by Theorem 2.1 as we just discussed, only left the case that the fluid problem has multiple optimal solutions; and (3) it is far less restrictive than the assumptions given in Sankararaman and Slivkins [2021], which require that there are at most two resources and the best-arm optimality, and almost surely excludes all problem instances. Under the assumption, our main results show that the re-solving heuristic reaches an O (1) regret with full information (Theorem 3.1) and an O (log T ) regret with partial information (Theorem 4.1). To our knowledge, these are the first ˜ O (1) regret results in the CBwK problem beyond the worst case with only mild assumptions. Importantly, these regret bounds are also independent to the number of actions, unlike previous results.

Within our results, the full information model assumes that the agents sees the external factor at 80 the end of each round, while in the partial information model, the agent acquires the external factor 81 only when a non-null action is adopted. Other state-of-the-art results consider bandit information 82 feedback, in which the agent only sees the reward and the consumption rather than the external factor. 83

However, they explicitly assume a specific (e.g., linear) relationship between the conditional expected 84 reward-consumption pair and the request [Agrawal and Devanur, 2016, Sankararaman and Slivkins, 85 2021, Han et al., 2022, Slivkins and Foster, 2022], whereas our results do not impose any underlying 86 distribution structures. On this side, our information model are comparable to those in existing work. 87

88

89

90

91

92

93

94

95

96

97

A near-optimal regret even in worst cases with full/partial information feedback, and an extension to continuous randomness. We further explore how well our Algorithm 1 performs even in worst-case scenarios. With full information feedback, we show that an O ( √ T log T ) regret is achieved (Theorem 5.1). This bound is asymptotically equal to the state-of-the-arts with this information model [Han et al., 2022, Slivkins and Foster, 2022]. Even with partial information, we can still guarantee a universal O ( √ T log T ) regret (Theorem 5.2), which is optimal up to a logarithmic factor. These results demonstrate the applicability of the re-solving heuristic in CBwK problems, regardless of the specific instance. For completeness, we also extend our algorithm and analysis to the situation in which the randomness of request and external factor are continuous, and derive corresponding regret results (Theorems A.1 and A.2).

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

1.2 Literature Review

Contextual bandits with knapsacks. The contextual bandits with knapsacks framework was introduced by Agrawal and Devanur [2016]. Along this research line, two main methodologies have been proposed to solve the problem. The first approach aims to select the best probabilistic strategy within the policy set [Badanidiyuru et al., 2014], and Agrawal et al. [2016] adopts this approach to achieve an ˜ O ( √ T ) regret. This heuristic originates from the subject of contextual bandits [Dudik et al., 2011, Agarwal et al., 2014], and requires a cost-sensitive classification oracle to achieve computation efficiency.

On the other hand, another approach views the problem from the perspective of the Lagrangian dual space. It uses a dual update method that reduces the CBwK problem to the online convex optimization (OCO) problem. In particular, some work [Agrawal and Devanur, 2016, Sankararaman and Slivkins, 2021, Sivakumar et al., 2022, Liu and Grigas, 2022] assumes a linear relationship between the conditional expectation of the reward-consumption pair and the request-action pair. This line adopts techniques for estimating linear function classes [Abbasi-Yadkori et al., 2011, Auer, 2002, Sivakumar et al., 2020, Elmachtoub and Grigas, 2022] and combines them with OCO methods to achieve sub-linear regret. Among these works, [Sankararaman and Slivkins, 2021] shows that when there are only two resources and a best-arm, this method can obtain an O (log T ) regret. Compared with their results, our assumptions are much milder, as we only assume non-degeneracy.

From another angle, depending on the difficulty of overcoming the lack of distribution knowledge on the external factor, there are two types of feedback models in the literature: full or bandit information. In the former [Liu and Grigas, 2022], the agent sees the external factor at the end of each round and can derive the reward and consumption of each possible decision in the round ex-post. Meanwhile, in the latter, the agent can only observe the reward-consumption pair brought by the decision. Apparently, the bandit information feedback is harder to deal with since less information can be accessed. Our work further considers a partial feedback model, in which the agent observes the external factor when a non-null action is chosen. This model acts as an intermediate between full and partial information feedback models.

Apart from the above work, two results [Han et al., 2022, Slivkins and Foster, 2022] concurrent with this work are not restricted to linear expectation functions. To deal with more general problems with bandit feedback, they plug model-reliable online regression methods [Foster et al., 2018, Foster and Rakhlin, 2020] into the dual update framework. As a result, the regret of their algorithms is the sum of the regret on online regression and online convex optimization, respectively. Nevertheless, the online regression technique still limits the conditionally expected reward-consumption functions.

Network revenue management and the re-solving heuristic. Unlike the above approaches, our work adopts the re-solving method, also known as the 'certainty equivalence' (CE) heuristic. Under this approach, the agent frequently solves the fluid optimization problem with the remaining resources to obtain a probability control in each round. This method comes from the literature on the network revenue management problem, which can be seen as a simplification of the CBwK problem without the existence of external factors, or that the external factor not getting involved in the resource

consumption [Wu et al., 2015]. Some work in this setting also assumes known request distributions. 137 This line of research originates from Jasin and Kumar [2012], and also includes Jasin [2015], Ferreira 138 et al. [2018], Bumpensanti and Wang [2020], Li and Ye [2021], Chen et al. [2022], Besbes et al. 139 [2022]. They show that the re-solving method can obtain constant regret under certain non-degeneracy 140 assumptions and can generally obtain square root regret [Chen et al., 2022]. Recently, the re-solving 141 method is also extended to the general dynamic resource-constrained reward collection (DRCRC) 142 problem in Balseiro et al. [2021], which assumes the knowledge of request and external factor 143 distributions and achieves O (1) to O (log T ) regret for different action space cardinalities. 144

145

146

147

148

149

We should mention that the re-solving technique has also been adopted to the bandits with knapsacks (BwK) problem [Flajolet and Jaillet, 2015] to achieve an O (log T ) regret. However, CBwK is a more challenging problem than BwK in the sense that the decision has to be made based on the received request. Thus, there is no optimal static action mode that is irrelevant to the round, which adds a layer of complexity to the re-solving method.

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

2 Preliminaries

We consider an agent interacting with the environment for T rounds. There are n kinds of resources, with an average amount of ρ i for resource i in each round, resulting in a total of ρ i T amount of resource i . We suppose that 0 < ρ = ρ 1 = ( ρ i ) i ∈ [ n ] ≤ 1 is independent of T , with a maximum entry of ρ max and a minimum entry of ρ min . At the beginning of each round t ≥ 1 , the agent observes a request θ t ∈ Θ drawn i.i.d. from a distribution U , and should choose an action a t from a set of actions A . Given the request θ t and the action a t , the agent receives a random reward r t ∈ [0 , 1] and consumption vector of resources c t ∈ [0 , 1] n , both of which are related to an external factor γ t ∈ Γ drawn i.i.d. from a distribution V . In other words, there is a reward function r : Θ × A × Γ → [0 , 1] and a consumption vector function c : Θ × A × Γ → [0 , 1] n , such that r t = r ( θ t , a t , γ t ) and c t = c ( θ t , a t , γ t ) . We suppose these two functions are pre-known to the agent. We further define R ( θ, a ) := E γ [ r ( θ, a, γ )] , and C ( θ, a ) := E γ [ c ( θ, a, γ )] .

We impose minimum restrictions on the distributions U and V . Specifically, in the main body of this work, we suppose that both distributions are discrete without any further assumptions. In other words, Θ and Γ are finite. We denote the mass function of U and V by u ( θ ) and v ( γ ) , respectively. We will extend to the situation that these two distributions can be continuous in Appendix A.

The agent's objective is to maximize her cumulative rewards over the period under initial resource constraints, which is a sequential decision-making problem. To ensure feasibility, we assume the existence of a null action (denoted by 0 ) in the action set A . Under the null action, the reward and the consumption of any resource are zero, regardless of the request and the external factor. In other words, we have r ( θ t , 0 , γ t ) = 0 and c ( θ t , 0 , γ t ) = 0 for any ( θ t , γ t ) ∈ Θ × Γ . We use A + := A \ { 0 } to denote the set of non-null actions, and let m := | A + | be its size.

We consider the set of non-anticipating strategies Π . In particular, let H t be the history the agent could access at the start of round t . Then, for any non-anticipating strategy π ∈ Π , a t should depend only on ˜ H t := ( θ t , H t ) , that is, a t = a π t ( θ t , H t ) . For abbreviation, we write a π t = a π t ( θ t , H t ) when there is no confusion.

Therefore, we can define the agent's optimization problem as below: 176

Benchmark. In practice, however, computing the expected reward of the optimal online strategy 177 would require high-dimension (probably infinite) dynamic programming, which is intractable. Hence, 178

we turn to consider the fluid benchmark to measure the performance of a strategy, which is defined as 179

̸

̸

Full information

Agent's observation

t

= 1

t

= 2

t

= 3

t

= 4

t

= 5

γ

1

γ

2

γ

3

γ

4

γ

5

a

1

= 0

a

2

= 0

a

3

= 0

a

4

= 0

a

5

= 0

Partial information

Agent's observation

t

= 1

t

= 2

t

= 3

t

= 4

t

= 5

γ

1

γ

2

γ

5

a

1

= 0

a

2

= 0

a

3

= 0

a

4

= 0

a

5

= 0

̸

̸

follows: 180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

For a better understanding, V FL reflects the maximum expected total rewards an agent can win when a static strategy is adopted and the resource constraints are only to be satisfied in expectation. Therefore, this optimization problem is a linear programming, in which the decision variable ϕ ( θ, a ) represents the probability that the agent chooses action a upon seeing request θ . It is a well-known result that V FL gives an upper bound on V ON .

Proposition 2.1 ([Balseiro et al., 2021]) . V FL ≥ V ON .

Thus, we evaluate the performance of a non-anticipating strategy π by comparing its expected accumulated reward Rew π with the fluid benchmark V FL . We call their difference the regret of π for convenience. In this context, we prove that an Ω( √ T ) regret lower bound is inevitable as long as V FL is degenerate.

Theorem 2.1 (worst-case location) . When V FL has a unique and degenerate optimal solution, V FL -V ON = Ω( √ T ) .

Despite the worst-case lower bound, we prove in this work that for any CBwK instance in which V FL has a unique non-degenerate optimal solution (Assumption 3.1), we can obtain an O (1) regret via the re-solving approach.

Information feedback model. In this work, we consider two types of information feedback models, with increasing levels of difficulty in obtaining a sample of the external factor γ .

[Full information feedback.] The agent is able to observe γ t at the end of each round t .

̸

[Partial information feedback.] The agent can observe γ t at the end of round t only if a t = 0 .

The above two information feedback models are illustrated in Figure 1. In general, with full information feedback, the agent can observe an i.i.d. sample from V each round, which is the optimal scenario for learning the distribution. Nevertheless, such an assumption may be overly strong since the reward and consumption vector are irrelevant to the external factor when the agent chooses the

̸

̸

204

205

206

207

208

209

210

211

212

213

214

215

̸

̸

null action a = 0 . Thereby, a more realistic information model is partial feedback, where the external factor is only accessible when a = 0 . This limitation also increases the difficulty of learning the distribution V since the agent observes fewer samples under this model than under full information feedback. It is important to note that the partial information model represents a transition from full to bandit information feedback, under which only the reward and consumption vector are accessible in each round, rather than the external factor.

3 The Re-Solving Heuristic

In this work, we introduce the re-solving heuristic to the CBwK problem. The resulting algorithm is presented in Algorithm 1.

To briefly describe the algorithm, we start by defining an optimization problem that captures the optimal fluid control for each round, assuming complete knowledge of U and V . For any κ ∈ [0 , 1] n , we define J ( κ ) be the following optimization problem:

Evidently, we have V FL = T · J ( ρ ) = T · J ( ρ 1 ) by definition. Intuitively, in each round t , the best 216 fluid choice of the agent is given by the optimal solution ϕ ∗ t of LP J ( ρ t ) , where ρ t is the average 217

budget of the remaining rounds, including round t . Nevertheless, since the agent lacks full knowledge 218 of the exact distributions U and V , she can only solve an estimated programming ̂ J ( ρ t , H t ) as 219 outlined in Algorithm 1, with the following realization: 220

Here, ̂ U t and ̂ V t represent the empirical distribution of θ and γ , respectively, according to the sample 221 history given by H t . Specifically, the mass functions of these two estimated distributions are standard 222 as follows: 223

It is worth noting that the estimated distribution of θ , ̂ U t , is always based t -1 samples since the agent received an independent sample from U

̸

the empirical distribution of the external factor γ , ̂ V t , is estimated from |I t | With full information feedback, |I t | = t -1 ; whereas with partial information feedback, |I t | ≤ t -1 equals the number of times the agent chooses an action a = 0 before round t . For brevity, for the estimated programming, we write ̂ C t ( θ, a ) := E [ c ( θ, a, γ )] and ̂ R t ( θ, a ) := E [ r ( θ, a, γ )]

As per Algorithm 1, the agent's decision mode in round t is given by the optimal solution ̂ ϕ t of programming ̂ J ( ρ t , H t ) . B i ≤ 1 for some resource i ∈ [ n ] , and we use T 0 T 0 := min { T, min { t : ∃ i ∈ [ n ] , B i < 1 }} .

at the beginning of each round. On the other hand, independent samples. γ ∼ ̂ V t γ ∼ ̂ V t . ∗ The algorithm stops when the resources are near depletion, that is, to denote the stopping time of Algorithm 1, i.e., t +1

For an analysis beyond the worst-case scenario, a crucial assumption we will make is that the fluid problem possesses good regularity properties, i.e., it is an LP with a unique and non-degenerate solution.

Assumption 3.1. The optimal solution to J ( ρ 1 ) is unique and non-degenerate.

The regularity assumption made Assumption 3.1 is commonplace in the linear programming literature [Chen et al., 2022, Li and Ye, 2021]. Further, any LP can easily avoid non-uniqueness or degeneracy through a slight perturbation [Megiddo and Chandrasekaran, 1989].

With the assumption, below we present the main result of this work, which is proved in Appendix C.1.

Theorem 3.1. Under Assumption 3.1, with full information feedback, the expected accumulated reward Rew brought by Algorithm 1 when T →∞ satisfies:

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

which is independent of T .

One of the key implications of Theorem 3.1 is that the re-solving heuristic's regret is independent of the number of rounds beyond the worst-case with full information. This result represents a significant improvement over previous state-of-the-art results under mild assumptions, surpassing the solutions proposed by Slivkins and Foster [2022], Han et al. [2022]. In particular, their solutions come from the bandits with knapsacks (BwK) literature and rely on dual update and upper confidence bound (UCB) heuristics, which only provide a worst-case regret of O ( √ T log T ) even with full information feedback. Furthermore, the reduction proposed by Sankararaman and Slivkins [2021] can only grant an O (log T ) regret for linear CBwK problems, and relies on the strong assumption that there is a

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

Θ(log

T

)

O

(

T

)

Overall

O

(1)

frequency

Uniform

O

(1

/

log

T

)

frequency

universal best action and only n = 2 resources. In contrast, our assumption is more common and less restrictive.

Additionally, as pointed out in Theorem 2.1, an Ω( T ) lower bound is established when the primal LP J ( ρ 1 ) has a unique and degenerate optimal solution, while Theorem 3.1 provides an O (1) upper bound on the optimal regret of CBwK with full information under the uniqueness and non-degeneracy condition. It is interesting to consider the regret bound in the remaining cases when J ( ρ 1 ) has multiple optimal solutions.

√

It is worth noting the relationship between our theoretical regret and the number of resources n and number of actions m . Generally, our analysis shows that the regret scales with (at most) the square of n . Further, a surprising result is that the regret is not explicitly related to m . This is superior to existing results, which report an ˜ O ( √ m ) reliance [Slivkins and Foster, 2022, Han et al., 2022, Badanidiyuru et al., 2014, Agrawal et al., 2016]. As an intuitive reason, the number of actions does not explicitly appear in the re-solving algorithm but only contributes to the dimension of the linear programming. However, this is not the case for other existing algorithms, which explicitly incorporate the number of actions m into their algorithms, resulting in a correlation between the regret and m .

4 Partial Information Feedback

̸

We now shift to consider the re-solving method's performance with partial information feedback, under which the agent only sees the external factor γ t when her choice is non-null in round t , i.e., a t = 0 . Apparently, with less information, the learning speed of the distribution V decreases, hindering the re-solving procedure's quick convergence to an optimal solution. Nevertheless, we demonstrate that the performance of the re-solving method only faces an O (log T ) multiplicative degradation under partial information feedback. Our primary theorem in this section is as follows:

Theorem 4.1. Under Assumption 3.1, with partial information feedback, the expected accumulated reward Rew brought by Algorithm 1 satisfies:

Before we come to the technical parts, we first place Theorem 4.1 within the literature. As previously mentioned, Ω( √ T ) is a worst-case lower bound on the regret even with full information feedback, and thus also serves as a lower bound with partial information feedback. However, Theorem 4.1 steps beyond the worst-case by providing an O (log T ) upper bound for most regular problem instances. This result outperforms the universal O ( √ T log T ) regret by Slivkins and Foster [2022], Han et al. [2022]. Although the result is asymptotically equivalent to that of Sankararaman and Slivkins [2021], it imposes fewer restrictions on the problem structure, as previously discussed. Moreover, the regret result's dependence on the number of resources n and number of actions m is inherited from Theorem 3.1.

̸

We now provide an intuitive understanding of the proof of Theorem 4.1. The crux lies in analyzing the frequency that Algorithm 1 can access an independent sample of the external factor. To this end, we use Y t = |I t | ≤ t -1 to denote the times of choosing action a = 0 before time t , or equivalently, the number of i.i.d. samples from V observed by the agent before time t , under partial information feedback. We have the following important lemma that presents a lower bound on Y t .

Lemma 4.1. There is a constant 0 < C b < 1 / 2 , such that with probability 1 -O (1 /T ) , the following hold for Algorithm 1:

1. For any Θ(log T ) ≤ t ≤ C b · T , Y t ≥ C f · ( t -1) / log T for some constant C f ;

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

2. For any t > C

b

·

T

,

Y

t

C

r

·

T

for some constant

C

r

.

The proof of Lemma 4.1 is deferred to Appendix D.2, and an illustration is displayed in Figure 2. In simple terms, during the first Θ(log T ) rounds (the shaded segment), the re-solving method cannot guarantee the accessing frequency since the learning of the request distribution U has not converged sufficiently. However, after Θ(log T ) rounds, Algorithm 1 ensures a constant probability of obtaining a new example in each round, provided that the remaining resources are sufficient. As a consequence, before O ( T ) rounds, we can guarantee an O (1 / log T ) accessing frequency at any time step and an overall O (1) frequency with high probability, by a concentration inequality. The remaining proof of Theorem 4.1 is provided in Appendix D.1.

5 Relaxing the Regularity Assumption - A Worst-Case Guarantee

In Sections 3 and 4, we have proved that can achieve an ˜ O (1) regret for CBwK problems under full or partial information feedbacks, assuming certain regular conditions (Assumption 3.1). Put differently, the re-solving heuristic nicely deals with regular scenarios. In this section, we complement this by showing that this method can also attain nearly optimal regret in the worst cases. Furthermore, we extend our analysis to cases where the context and external factor distributions can be continuous in Appendix A.

Our main results are given below, and their proofs are provided in Appendices E.1 and E.2, respectively.

Theorem 5.1. With full information feedback, the expected accumulated reward Rew brought by Algorithm 1 satisfies:

Theorem 5.2. With partial information feedback, the expected accumulated reward Rew brought by Algorithm 1 satisfies:

As given by Theorem 2.1, the worst-case regret of any online CBwK algorithm is Ω( √ T ) , while Theorems 5.1 and 5.2 indicate that the re-solving heuristic reaches near-optimality in such cases. Further, state-of-the-art algorithms [Han et al., 2022, Slivkins and Foster, 2022]) can at most obtain an ˜ O ( √ T ) regret with full/partial information feedback. Our algorithm also achieves this regret bound in worst cases.

6 Concluding Remarks

This work establishes the effectiveness of the re-solving heuristic in the contextual bandits with knapsacks problem. We first prove that any online algorithm incurs a regret of Ω( √ T ) when the fluid LP has a unique and degenerate optimal solution. Building on this, we demonstrate that the re-solving method reaches an O (1) regret with full information and an O (log T ) regret with partial information when the fluid LP has a unique and non-degenerate optimal solution. Considering the sufficient condition for the Ω( √ T ) lower bound, our non-degeneracy assumption is mild, especially when combined with the two-resource and best-arm-optimality condition required in Sankararaman and Slivkins [2021].

Further, we show that even in the worst-case, the re-solving method achieves an O ( √ T log T ) regret with full information feedback and an O ( √ T log T ) regret with partial information feedback. These results are comparable to start-of-the-art results [Slivkins and Foster, 2022, Han et al., 2022]. We also extend our analysis to the continuous randomness case for completeness.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems , 24, 2011.

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646. PMLR, 2014.

Shipra Agrawal and Nikhil Devanur. Linear contextual bandits with knapsacks. Advances in Neural Information Processing Systems , 29, 2016.

Shipra Agrawal, Nikhil R Devanur, and Lihong Li. An efficient algorithm for contextual bandits with knapsacks, and an extension to concave objectives. In Conference on Learning Theory , pages 4-18. PMLR, 2016.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Ashwinkumar Badanidiyuru, John Langford, and Aleksandrs Slivkins. Resourceful contextual bandits. In Conference on Learning Theory , pages 1109-1134. PMLR, 2014.

Santiago Balseiro, Omar Besbes, and Dana Pizarro. Survey of dynamic resource constrained reward collection problems: Unified model and analysis. Available at SSRN 3963265 , 2021.

Santiago R Balseiro and Yonatan Gur. Learning in repeated auctions with budgets: Regret minimization and equilibrium. Management Science , 65(9):3952-3968, 2019.

Omar Besbes, Yash Kanoria, and Akshit Kumar. The multisecretary problem with many types. arXiv preprint arXiv:2205.09078 , 2022.

Pornpawee Bumpensanti and He Wang. A re-solving heuristic with uniformly bounded loss for network revenue management. Management Science , 66(7):2993-3009, 2020.

Guanting Chen, Xiaocheng Li, and Yinyu Ye. An improved analysis of lp-based control for revenue management. Operations Research , 2022.

Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence , pages 169-178, 2011.

Adam N Elmachtoub and Paul Grigas. Smart 'predict, then optimize'. Management Science , 68(1): 9-26, 2022.

Kris Johnson Ferreira, David Simchi-Levi, and He Wang. Online network revenue management using thompson sampling. Operations research , 66(6):1586-1602, 2018.

Arthur Flajolet and Patrick Jaillet. Logarithmic regret bounds for bandits with knapsacks. arXiv preprint arXiv:1510.01800 , 2015.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning , pages 3199-3210. PMLR, 2020.

Dylan Foster, Alekh Agarwal, Miroslav Dudik, Haipeng Luo, and Robert Schapire. Practical contextual bandits with regression oracles. In International Conference on Machine Learning , pages 1539-1548. PMLR, 2018.

Yuxuan Han, Jialin Zeng, Yang Wang, Yang Xiang, and Jiheng Zhang. Optimal contextual bandits with knapsacks under realizibility via regression oracles. arXiv preprint arXiv:2210.11834 , 2022.

Stefanus Jasin. Performance of an lp-based control for revenue management with unknown demand parameters. Operations Research , 63(4):909-915, 2015.

Stefanus Jasin and Sunil Kumar. A re-solving heuristic with bounded revenue loss for network revenue management with customer choice. Mathematics of Operations Research , 37(2):313-345, 2012.

Xiaocheng Li and Yinyu Ye. Online linear programming: Dual convergence, new algorithms, and regret bounds. Operations Research , 2021.

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

Heyuan Liu and Paul Grigas. Online contextual decision-making with a smart predict-then-optimize method. arXiv preprint arXiv:2206.07316 , 2022.

Olvi L Mangasarian and T-H Shiau. Lipschitz continuity of solutions of linear inequalities, programs and complementarity problems. SIAM Journal on Control and Optimization , 25(3):583-595, 1987.

Nimrod Megiddo and Ramaswamy Chandrasekaran. On the ε -perturbation method for avoiding degeneracy. Operations Research Letters , 8(6):305-308, 1989.

Karthik Abinav Sankararaman and Aleksandrs Slivkins. Bandits with knapsacks beyond the worst case. Advances in Neural Information Processing Systems , 34:23191-23204, 2021.

Gerard Sierksma. Linear and integer programming: theory and practice . CRC Press, 2001.

Vidyashankar Sivakumar, Steven Wu, and Arindam Banerjee. Structured linear contextual bandits: A sharp and geometric smoothed analysis. In International Conference on Machine Learning , pages 9026-9035. PMLR, 2020.

Vidyashankar Sivakumar, Shiliang Zuo, and Arindam Banerjee. Smoothed adversarial linear contextual bandits with knapsacks. In International Conference on Machine Learning , pages 2025320277. PMLR, 2022.

Aleksandrs Slivkins and Dylan Foster. Efficient contextual bandits with knapsacks via regression. arXiv preprint arXiv:2211.07484 , 2022.

Alberto Vera and Siddhartha Banerjee. The bayesian prophet: A low-regret framework for online decision making. Management Science , 67(3):1368-1391, 2021.

Larry Wasserman. Density estimation - lecture note for 36-708 statistical methods for machine learning, Carnegie Mellon University, 2019.

Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the l1 deviation of the empirical distribution. Hewlett-Packard Labs, Tech. Rep , 2003.

Huasen Wu, Rayadurgam Srikant, Xin Liu, and Chong Jiang. Algorithms with logarithmic or sublinear regret for constrained contextual bandits. Advances in Neural Information Processing Systems , 28, 2015.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the CBwK problem beyond the worst case scenario, presenting two results regarding the worst-case locations and log rate respectively. The proposed algorithm utilizes a re-solving heuristic that achieves $O(1)$ under full-information and $O(\\log(T))$ regret under partial-information with some regularity conditions. The worst-case guarantee is also presented.\n\nSTRENGTHS:\nThis is a well written paper. The key ideas and main contributions are clearly presented. The proofs seems complete and easy to follow.\n\nWEAKNESSES:\nThis reviewer has doubts on the exact contribution of this paper. \nFirstly, the problem studied (CBwK) is well-established in the literature. And the setting that the authors choose might be not so realistic and sort of marginal. For example, one major concern from my understanding is that the budget is \"soft\", since the budget constraint is only required in expectation, and in summation. However, in reality, \"hard\" constraints should be considered prevalently, where the constraint should not be compensated, and the expectation might also be unnecessary. The assumption of null action might not be met in reality as well.\n\nSecondly, the algorithm and the re-solving heuristic are from literature. There's something new in the proofs since extending from BwK to CBwK requires a more complicated estimation.\n\nTo add to these, there are existing work on constrained reinforcement learning, e.g. [1] that considers similar setting. Since RL is typically considered more general a setting than CB, the results there should also be at least comparable.\n\nLastly, the assumption for full-information seems overly strong, and the results does not seem very pertinent to the main claims. Maybe it could be better moved to the appendix to make the contribution more clear.\n\n[1] Sobhan Miryoosefi, Chi Jin, A Simple Reward-free Approach to Constrained Reinforcement Learning\n\nQUESTIONS:\nSince this paper is dealing with a subtle topic, a thorough comparison with the existing work should be expected, which seems missing in the current version.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works considers a general setup of Contextual Bandits with Knapsacks (CBwK). The authors identify sufficient conditions under which constant of logarithmic regrets are possible (while previous authors were mainly dealing with $\\sqrt{T}$ worst-case regrets). The studied algorithm is rather intuitive and is based on a sequential refinement of the underlying best static LP problem.\n\nSTRENGTHS:\n1. Identification of the conditions for $o(\\sqrt{T})$ regret is a great contribution.\n2. The algorithm does not need to know whether the conditions are satisfied---it is adaptive.\n3. The main body exposition is rather clear, but more details would be appreciated.\n4. The approach is intuitive and computationally tractable.\n5. Overall I liked the proofs, they are a bit hard to read due to heavy notation, but once the reader is comfortable, the presentation is rather good.\n\nWEAKNESSES:\nHere are the weaknesses in no particular order\n\n1. Bounds are in expectation (which is fine per-se, but from my experience the results are often formulated with high probability)\n2. Little to no comments on the actual stopping time of the algorithm, is it much smaller than T? (I only found this information in the appendix)\n3. O(1) is great, but what kind of 1 it involves is not present in the main body.\n4. Regret is not the only part of the story: e.g., Agrawal and Devour 2016 allow o(1) budgets, instead of constant budgets. In this case comparing to their result is not really fair. \n5. While computationally tractable, the algorithm is not very efficient, one needs to solve a potentially huge LP problem at each step.\n\nQUESTIONS:\n1. Proofs of all the results rely heavily on the stability Lemma C.2. This lemma involves a constant $D$, which eventually appears in the upper bounds. The actual dependency on the parameters of the problem is not discussed. Imagine I am in the setup of non-constant budget e.g. 1 / \\sqrt{T} budget. Shall $D$ start to depend on $T$ as well in this case? In general, expliciting exact dependency of $D$ on the problem's parameters would be great. Otherwise $O(1)$ is a bit too opaque.\n\n2. Say I want to derive bounds with high probability, then, I feel that the same results are possible, but they will involve additional condition like $T \\geq T_0$ with $T_0$ being non-explicit and potentially very large. Is it the case?\n\n3. Could the authors provide a comment on potential extension of the results to algorithms based on OCO?\n\n4. The extension to the continuous case is slightly unsatisfactory. Personally, I expected a more different type of modelling condition e.g. logistic model (instead of smoothness type assumption). Would it be possible to extend the analysis to continuous case, maintaining the $\\sqrt{T}$ worst-case regret, assuming that both reward and costs can be modelled by a logistic model?\n\nP.S. I am open to increase my score.\n\nLIMITATIONS:\nI was not aware of the work of Chen et al 2022, but upon skimming it looks very similar to the current submission (tools and techniques-wise). A broader discussion would be appreciated.\n\nWhile being honest and correct, the extension to the continuous case, is not very satisfactory (but maybe it is a matter of taste).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper studies stochastic contextual Bandits with Knapsacks. The authors provide an algorithm that guarantees regret smaller than the worst-case $\tilde O(\sqrt{T})$ in non-degenerate instances. In particular, they provides an algorithm that achieves $\tilde O(1)$ regret when the fluid LP has a unique and non-degenerate solution. Moreover, in the worst-case the algorithm maintains $\tilde O(\sqrt{T})$ regret.

STRENGTHS:
The main paper is clear and well written. The problem under study is interesting. The technical results are not straightforward.

WEAKNESSES:
Most of the results in the paper are based on the existence of an unique and non-degenerate solution (Assumption 3.1). The formal definition of this assumption is missing. Moreover, while this assumption is commonplace in the linear programming literature, it is not clear whether it makes sanse in the bandit with knapsack problem. Finally, previous work suggests that large regret is unavailable in many standard cases.

The main paper does not give a clear intuition of the techniques used to prove the theoretical results and the appendix is difficult to follows. Moreover, the appendix is heavily based on results in other papers, e.g., Chen et al. [2022]. This makes the proofs hard to follow. Moreover, it is not clear which are the main technical contributions of the paper, and to what extend the results follow directly from the application of known techniques.

QUESTIONS:
Why is Assumption 3.1 reasonable in the bandit with knapsack problem?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of contextual bandits with knapsacks and provides an algorithm that goes beyond worst-case (i.e., provide logarithmic regret) under the mild condition of unique optimal solution and non-degenerate solution. This also simultaneously enjoys an optimal worst-case regret bound when the conditions aren't met. The paper also shows a Omega(sqrt(T)) lower-bound when the instance has a unique optimal solution and a degenerate solution.\n\nSTRENGTHS:\n+ The paper considers a significant generalization of the results in Sankararaman and Slivkins 2021, in that it extends the logarithmic regret from two resources and best-arm optimality to arbitrary resources and unique best solution.\n\n+ The paper also proves that the new algorithm derived from resolving the program enjoys worst-case regret bounds that are optimal (in the interesting regime of B ~ O(T)).\n\n+ The results also applies to full policy-based contextual bandit problems, as opposed to prior work with logarithmic regret bounds which only applies to the linear contextual bandits setting.\n\nWEAKNESSES:\n- The algorithm needs to (a) solve a program at each round (can this be removed?) unlike UCB type algorithms where the program needs to be solved only when the ucb of any single quantity changes by a constant factor and (b) even in a single round the run-time of solving the program is unclear (I might be wrong, please correct me). Although the proposed results are interesting, this seems like a major downside from the algorithm front.\n\n- [Some rewording of comparison with prior work] The proposed algorithm is very similar to that of [Flajolet and Jaillet, 2015] (the paper mentions this, although the wording could be more generous to the prior work). Likewise, the paper mentions that the setting considered in Sankararaman and Slivkins '2021 is \"and\nalmost surely excludes all problem instances\" which is not technically true. Note that the prior work paper uses the same argument of perturbing LPs in the case of d=2. Thus, the key difference is extending to d > 2 and not about the number of instances. \n\n- Although the lower-bound presented in this paper is interesting, I think some of the lower bounds are implied from the instances presented in Sankararaman and Slivkins '2021.\n\n---\n\nEDIT: \n\nI still don't know how to implment this algorithm apart from the trivial way of having one arm per (context, arm) pair. This is important, since then the algorithm is trivial, and the results then likely follow from prior works, in particular [1] which implements this for the k-armed bandit setting. Thus, the paper needs to explain clearly why this is better than that trivial reduction, and as a result, the bounds are significantly better. I believe multiple reviewers get at this same point, and i am not able to see a  convincing argument. Please make sure to incorporate this detailed comparision in the next version of the paper.\n\n\n[1] Logarithmic regret bounds for Bandits with Knapsacks - Arthur Flajolet, Patrick Jaillet\n\nQUESTIONS:\ncan you elaborate a bit more on the exact running time of the algorithm, per step? Also can similar tricks of not having to solve in each step, as in UCB, be used for the resolving heuristic?\n\nLIMITATIONS:\nThis is a mathematical paper and no societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}