Title: Piecewise-Stationary Bandits with Knapsacks
Venue: NeurIPS 2024
Accepted: yes
Keywords: non-stationary bandits

bandits with knapsacks

competitive ratio
Abstract: We study Bandits with Knapsacks (Bwk) in a piecewise-stationary environment. We propose a novel inventory reserving algorithm which draws new insights into the problem. Suppose parameters $\eta_{\min}, \eta_{\max} \in (0,1]$ respectively lower and upper bound the reward earned and the resources consumed in a time round. Our algorithm achieves a provably near-optimal competitive ratio of $O(\log(\eta_{\max}/\eta_{\min}))$, with a matching lower bound provided. Our performance guarantee is based on a dynamic benchmark, distinguishing our work from existing works on adversarial Bwk who compare with the static benchmark. Furthermore, different from existing non-stationary Bwk work, we do not require a bounded global variation.

Piecewise-Stationary Bandits with Knapsacks

Abstract

We propose a novel inventory reserving algorithm which draws new insights into Bandits with Knapsacks (Bwk) problems in piecewise-stationary environments. Suppose parameters η min , η max P p 0 , 1 s respectively lower and upper bound the ratio between the reward earned and the resources consumed in a round. Our algorithm achieves a provably near-optimal competitive ratio of O p log p η max { η min qq , with a matching lower bound provided. Our performance guarantee is based on a dynamic benchmark that upper bounds the optimum, different from existing works on adversarial Bwk Immorlica et al. (2019); Kesselheim and Singla (2020) who compare with the stationary benchmark. Different from existing non-stationary Bwk work Liu et al. (2022), we do not require a bounded global variation.

1 Introduction

In a bandits with knapsack (Bwk) problem, each action a in the action set K is associated with a latent and random amount of reward earned, R t p a q , and resource consumed, C t p a q , in each round t ' 1 , . . . , T . A decision maker (DM) selects an action a t P K in round t , and observes bandit feedback p R t p a t q , C t p a t qq . The DM targets at maximizing the total reward ř T t ' 1 R t p a t q , while satisfying the hard capacity constraint ř T t ' 1 C t p a t q ď B . Bwk has many real-life applications such as dynamic pricing Babaioff et al. (2015), resource allocation Zhalechian et al. (2022), online auction Balseiro and Gur (2019) and assortment planning Agrawal et al. (2019). Stochastic Bwk is first introduced by Badanidiyuru et al. (2018), followed by generalizations to concave reward with convex constraints Agrawal and Devanur (2014), combinatorial bandits Sankararaman and Slivkins (2018) and contextual bandits Badanidiyuru et al. (2014); Agrawal and Devanur (2016). In stochastic Bwk problems, the expected feedback E rp R t p a q , C t p a qqs ' p r p a q , c p a qq is stationary for all a P K , t P t 1 , . . . , T u , and a sublinear-inT regret is achievable. Nevertheless, the stationary model could be too ideal in many applications.

Adversarial Bwk is firstly considered in Immorlica et al. (2019) where p r t , c t q ' tp r t p a q , c t p a qqu a P K can change arbitrarily over the horizon. They achieve a competitive ratio (CR) of O p d log p T qq with respect to a static benchmark when there are d budget constraints. A static benchmark picks a fixed optimal action (or a fixed optimal distribution over arms), and applies the same action (or distribution) in all T rounds. Kesselheim and Singla (2020) further improve the CR to O p log p d q log p T qq . Other papers consider different regimes such as unlimited rounds (Rangi et al. (2018)), large budget B ' Ω p T q (Castiglioni et al. (2022a)), strict feasibility (Castiglioni et al. (2022b)) and approximate stationarity (Fikioris and Tardos (2023)). All these works compare with static benchmarks (see Appendix A.1). Moreover, adversarial Bwk could be too conservative in certain real-life scenarios. For instance, sales patterns could be stationary for a duration of time, but only change during periods of hot seasons/promotions/new trends, which fits into our piecewise-stationary Bwk regime.

An abundance of existing works explore adversarial online knapsack problems with full feedback, where p r t , c t q can change arbitrarily but their realized value of tp R t p a q , C t p a qqu a P K is observed before choosing a t (Karp et al. (1990); Mehta et al. (2007); Zhou et al. (2008)). Many of these works compare their accrued rewards with dynamic benchmarks (stronger than the static benchmarks used in adversarial Bwk), where the DM picks different optimal actions a ˚ t in different rounds. However, the dynamic benchmarks considered in the above papers are best single arm benchmark, which only allows pulling a single arm in each round; while our benchmark is a best distribution over arms benchmark (see Appendix A.2 for a more detailed elaboration). Further, the ability of observing tp R t p a q , C t p a qqu a P K before selecting a t is crucial in the algorithm designs in Karp et al. (1990); Mehta et al. (2007); Zhou et al. (2008). Their algorithm design cannot be readily generalized to the bandit setting, where the DM only observes p R t p a t q , C t p a t qq after selecting a t .

Another line of recent research investigates non-stationary online knapsack problems with either full feedback Jiang et al. (2020); Balseiro et al. (2022) or bandit feedback Liu et al. (2022). These works quantify the scale of non-stationarity in terms of both the local variation loc ' ř T ´ 1 t ' 1 dist pp r t ` 1 , c t ` 1 q , p r t , c t qq and the global variation glo ' ř T t ' 1 dist p ř T t ' 1 p r t , c t q{ T, p r t , c t qq , where dist is a certain metric. Assuming max t loc , glo u grows sublinearly in T , they achieve sublinearinT regret bounds compared to the dynamic benchmark. However, glo growing sublinearly in T is rather strong assumption. We could have glo linear in T , even with one change point (see Remark A.3 for more detail). In this work, we consider piece-wise stationary models that allow a higher degree of non-stationarity, which is yet to be studied in all aforementioned works.

Our contributions. Firstly, on modeling (see Section 2), the DM does not know the number of change points and when changes happen. We formulate our model as a single-resource problem, and extend to d -resource problems in Appendix B.5 with an extra multiplicative factor of d on the competitve ratio. Secondly, on algorithm design (see Sections 3.1 and 4.1), we propose novel algorithms which are natural, intuitive and easy to implement. Our idea of reserving inventory based on the rewardconsumption ratio provides new insights into the problem. Thirdly, on performance guarantee (see Sections 3.2 and 4.2), we achieve a provably near-optimal competitive ratio with respect to a best distribution over arms benchmark without requiring a bounded glo , which distinguishes our work from the existing literature. Specifically, suppose there exists parameters η min , η max P p 0 , 1 s such that η min ď r t p a q , c t p a q ď η max for all t, a . Our algorithms achieve a competitive ratio of O p log p η max { η min qq , which requires a novel analysis. We prove the tightness of our competitive ratio by providing a matching lower bound (see Section 4.4). We also run some illustrative numerical experiments (see Section 5) to compare our algorithm performance with Immorlica et al. (2019) and Zhou et al. (2008) under the piecewise-stationary settings.

2 Model

2.1 Problem formulation

Problem dynamics. The online model involves T rounds, indexed as t P T ' t 1 , 2 , . . . , T u . We index an arm as a P K . Additionally, we define the null arm a null, where no allocation is made when a null is chosen. In round t P T , the DM chooses an arm a t P K Ť t a null u , and observes a noisy outcome vector p R t p a t q , C t p a t qq P r 0 , 1 s 2 as the bandit feedback, where R t p a t q and C t p a t q are the reward and the resource consumption in round t respectively. We set R t p a null q ' C t p a null q ' 0 with certainty for all t P T . The DM is endowed with B ď T units of the resource. The DM's goal is to maximize the total reward, with the constraint that the total resource consumption is at most B with certainty. Denote r t ' t r t p a qu a P K ' t E r R t p a qsu a P K and c t ' t c t p a qu a P K ' t E r C t p a qsu a P K . We consider a piece-wise stationary setting, where the planning horizon T is partitioned into L stationary pieces t t 0 ' 1 , . . . , t 1 u , t t 1 ` 1 , . . . , t 2 u , . . . , t t L ´ 1 ` 1 , . . . , t L ' T u . On each stationary piece l , we have p r t , c t q ' p r p l q , c p l q q for all t P t t l ´ 1 ` 1 , . . . , t l u .

The DM does not know the number of rounds T , the number of stationary pieces L , the rounds t 1 , . . . , t L ´ 1 where changes happen, the values of tp r p l q , c p l q qu l Pt 1 ,...,L u and their realized outcomes.

Goal and benchmark. Our goal is to develop an online algorithm that maximize the expected total reward E r ř T t ' 1 R t p a t qs while satisfying the inventory constraint ř T t ' 1 C t p a t q ď B , which can be formulated as the following dynamic program DP. In DP, X t ' t X t p a qu a P K is a binary decision vari-

able indicating whether to pick arm a in round t (i.e., X t p a q ' 1 ) or not (i.e., X t p a q ' 0 ). An online algorithm is non-anticipatory in the sense that X t depends only on B and tp R s p a s q , C s p a s q , X s qu t ´ 1 s ' 1 .

In non-stationary bandits without resource constraints, the performance bound of an online algorithm is in the form of ř T t ' 1 R t p a t q ě ř T t ' 1 r t p a ˚ t q ´ Reg, where ř T t ' 1 r t p a ˚ t q is the optimal expected reward obtained by choosing the best arm in each round, and Reg is a sublinear-inT regret characterizing the reward loss. In our Bwk setting, due to the inventory constraint, achieving a sublinear-inT regret is impossible without assuming a bounded global variation (see Appendix A.3). We denote opt p P q as the optimum of an optimization problem P. We aim for a performance guarantee of the form ř T t ' 1 R t p a t q ě 1 CR ¨ opt(DP) ´ Reg, where opt(DP) is the optimum of DP, CR is a competitive ratio, and Reg is a sublinear-inT regret. Unfortunately, DP is hard to solve. Therefore, we define a fluid approximation FA of DP, where R t and C t are replaced by their respective expectations, and the decision variables are fractional. In the following Lemma 2.1 (proved in Appendix C.1), we justify that opt(FA) can serve as a benchmark for our algorithms' performance since

and we aim to derive performance guarantees of the form ř T t ' 1 R t p a t q ě 1 CR ¨ opt p FA q ´ Reg. Lemma 2.1. opt(FA) ě opt(DP).

2.2 Assumptions, limitations and discussions

Compared with the adversarial Bwk literature, our piecewise-stationary setting has two limitations. The first is on L , the number of change points. When L is not known, our result is meaningful only when L ' o p ? T ¨ η min q . When L is known, our result is meaningful when L ' o p T ¨ η min q (see Theorem 4.2). In contrast, existing works on adversarial Bwk generally allow L ' T . The second is on the value range of non-null actions:

Assumption 2.2. For all a P K , l P t 1 , . . . , L u , there exists known constants η min , η max P p 0 , 1 s such that η min ď r p l q p a q , c p l q p a q ď η max .

While η min can be as small as 0 generally, we argue that this assumption is mild. Assumption 2.2 holds in many real-life scenarios. For instance, in portfolio management, an investor allocates a limited budget among different investment options (arms) to maximize the overall return. The investor has assessments on lower and upper ranges of the expected returns for each investment option. The lower range is usually strictly positive, since the investor would not consider investment options with 0 or negative expected return. In applications such as dynamic pricing, assortment planning, network resource allocation and energy management, expected profits and consumer demands are usually within a known positive value range. We further justify that Assumption 2.2 theoretically in the following Lemma 2.3 (proved in Appendix C.5).

Lemma 2.3. For any online algorithm, there exist an instance for which 0 ď r t p a q , c t p a q ď 1 for all a P K , t P T , and that CR ą Ω p log p η max { η min qq .

Additionally, in Appendix C.5 we demonstrate that knowing the values of η min , η max is necessary for achieving CR ' O p log p η max { η min qq .

2.3 High-level idea of our algorithm

Decomposing opt(FA) in terms of reward-consumption ratios. Throughout our paper, we fix an optimal solution t x ˚ l u L l ' 1 to FA. We define the set L ' t l P t 1 , . . . , L u : ř a P K x ˚ l p a q ą 0 u , which indexes stationary pieces where non-null allocations are made under the optimal solution t x ˚ l u L l ' 1 to FA . For each l P L , we define

Ratio p l q˚ , whose value range follows from Assumption 2.2, is the optimal expected reward earned per unit of consumed resource under t x ˚ l u L l ' 1 . We call Ratio p l q˚ the optimal expected rewardconsumption ratio of stationary piece l . B ˚ l represents the optimal expected amount of resources assigned for stationary piece l . To aid our algorithm design, we define the following linear program:

Then, we can express opt p FA q in terms of LP p ˜ r, ˜ c, ˜ B q and Ratio p l q˚ , B ˚ l as follows:

The first equation in (2) can be verified by noting that x ˚ l is feasible to LP p r p l q , c p l q , B ˚ l {p t l ´ t l ´ 1 qq for each l P t 1 , . . . , L u , and the concatenation of the optimal solutions of t LP p r p l q , c p l q , B ˚ l {p t l ´ t l ´ 1 qu l P L forms a feasible solution to FA. The second equation in (2) holds, by the definitions of Ratio p l q˚ , B ˚ l and the fact that opt p LP p r p l q , c p l q , B ˚ l {p t l ´ t l ´ 1 qqq ' ř a P K r p l q p a q x ˚ l p a q .

Algorithm design. Fix an arbitrary constant α ą 1 (we set α ' e by default, but our results hold for any constant α ą 1 ). We define M ' r log α p η max { η min q s and partition r η min { η max , η max { η min s into 2 M intervals r α ´ M , α ´ M ` 1 s Ť tp α m , α m ` 1 su M ´ 1 m '´ M ` 1 . For each stationary piece l P L , we denote m ˚ l P t´ M,...,M ´ 1 u as the interval such that Ratio p l q˚ P p α m ˚ l , α m ˚ l ` 1 s . In the forthcoming discussion, with some abuse of notation, we sometimes write interval r α ´ M , α ´ M ` 1 s as p α ´ M , α ´ M ` 1 s , and we refer to interval p α m , α m ` 1 s as reward-consumption ratio interval m , or 'interval m ' in short. Then we can decompose opt(FA) regarding reward-consumption ratio intervals:

The key intuition of our algorithm is to achieve a reward guarantee for each interval m regarding the reward-consumption ratio 1 p m ˚ l ' m q ¨ Ratio p l q˚ and the resource consumption ř l P L 1 p m ˚ l ' m q ¨ B ˚ l , which is done by performing two tasks: (a) for each l P L , we guess the value of m such that Ratio p l q˚ P p α m , α m ` 1 s . We guarantee that for at least 1 {p M ` 1 q fraction of requests on each l , our guessed ratio interval are close to the correct interval m ˚ l ; (b) for each interval m , we 'reserve' B { 2 M resource units. That is, we reserve an inventory of B { 2 M resource units to satisfy requests with a guessed reward-consumption ratio interval m . When the inventory reserved for interval m is depleted, the DM rejects (by choosing a null) all future requests with a guessed interval m .

By accomplishing task (a), we ensure that for each l P L , at least 1 p m ˚ l ' m q¨ B ˚ l {p M ` 1 q requested resources are served by resources reserved for interval m , generating reward at a ratio of at least

α m . Then, by accomplishing task (b), if the reserved inventory for interval m are not depleted by round T , our algorithm earns a reward of at least α m ¨ 1 p m ˚ l ' m q ¨ B ˚ l {p M ` 1 q during stationary piece l . Else, if the reserved B { 2 M resource units for interval m are depleted by round T , then the DMearns a reward of at least α m ¨ B {p 2 M q ě α m ¨ ř l P L 1 p m ˚ l ' m q ¨ B ˚ l {p 2 M q from resources reserved for interval m , since ř l P L 1 p m ˚ l ' m q ¨ B ˚ l ď B . By judiciously analyzing the relationship between stationary pieces and reward-consumption ratio intervals, for each interval m , we ensure that a reward of

is accrued, which is 1 { O p M q of the benchmark resources consumed on all stationary pieces l P L whose Ratio p l q˚ P p α m , α m ` 1 s . By summing over m P t´ M,...,M ´ 1 u , we achieve 1 { O p M q fraction of reward (3).

3 Warm-up: Full-feedback deterministic outcome setting

In this section, we introduce the main idea of our algorithm on the bandit model by relaxing the model uncertainty assumptions and specializing to the full-feedback deterministic setting: p R t , C t q ' p r p l q , c p l q q with certainty for each t P t t l ´ 1 ` 1 , . . . , t l u . Thus, p r p l q , c p l q q is observed at the start of the stationary piece l . The DM does not know L and t t 1 , . . . , t L ´ 1 u before the online process begins. The DM's decision can be fractional, which means on each stationary piece l , a decision can take the form of x l P t x P r 0 , 1 s | K | : ř a P K x p a q ď 1 , x p a q ě 0 @ a P K u , resulting in a reward of ř a P K r p l q p a q x l p a q and resource consumption of ř a P K c p l q p a q x l p a q in a round.

3.1 Inventory REServing (IRES) algorithm

Upon observing p R t , C t q ' p r p l q , c p l q q , guessing Ratio p l q˚ (see Section 2.3) is equivalent to guessing x ˚ l , which is equivalent to guessing B ˚ l {p t l ´ t l ´ 1 q , since t x ˚ l u L l ' 1 is an optimal solution to LP p r p l q , c p l q , B ˚ l {p t l ´ t l ´ 1 qq . In this section, when we say 'Line xx', we refer to a line of Algorithm 1, which displays IRES. At the start, the IRES reserves B { 2 M units of resource to each interval m P t´ M,...,M ´ 1 u , and each resource unit is reserved by exactly one interval. In Line 5, for each stationary piece l , we firstly solve LP p r p l q , c p l q , η min ¨ α q q for each q P t 0 , . . . , M u , and get an optimal solution x p q q˚ l ' t x p q q˚ l p a qu a P K . Each η min ¨ α q is a guess of B ˚ l {p t l ´ t l ´ 1 q . For each q P t 0 , . . . , M u , we define

as a guess of Ratio p l q˚ . Claim 3 in Appendix B.3 shows that, by guessing a q such that η min ¨ α q is within a factor of α from B ˚ l {p t l ´ t l ´ 1 q , we also have Ratio p q q l to be at most a factor of α from Ratio p l q˚ . As time progresses on l , we go round-robin on the choices of q t P t 0 , . . . , M u for each round t (Lines 7, 15). In round t P t t l ´ 1 ` 1 , . . . , t l u , we identify m t P t´ M,...,M ´ 1 u such that Ratio p q t q l P p α m t , α m t ` 1 s (Line 8). If there remains enough reserved resource units for interval m t (Line 10), the DM fulfils the t -th request by selecting fractional action x t ' x p q t q˚ l (Line 11), which consumes resources reserved for m t (Lines 9, 11). Otherwise, the DM selects a null and rejects the request (Line 13). By Line 9, we have T p m q t ' t s P t 1 , . . . , t u : m s ' m u , which consists of rounds in t 1 , . . . , t u when the DM attempts to fulfil a request with resources reserved for interval m .

3.2 Performance guarantee of IRES

We provide a performance guarantee to IRES in Theorem 3.1.

Theorem 3.1. For any given α ą 1 , IRES achieves a reward of at least

under mild requirements that t l ´ t l ´ 1 ě M ` 1 @ l P L . In particular, IRES achieves a competitive ratio of O p log α p η max { η min qq if B ě Ω p log α p η max { η min qq .

Algorithm 1 Inventory REServing with deterministic input (IRES)

3.3 Analysis

That is, ˜ T p m q consist up to the last round assigned to interval m such that the reserved inventory is not fully consumed. It is evident that if ř s P T p m q T c s p a s q ď B {p 2 M q ´ 1 , then ˜ T p m q ' T p m q T .

Define J l ' t t l ´ 1 ` 1 , . . . , t l u , the time interval of the l -th piece. The reward achieved by IRES is REW ' ř t P T ř a P K r t p a q x t p a q , which can be decomposed as REW ' ř M ´ 1 m '´ M REW p m q where

The set p Ť M ´ 1 n '´ M ˜ T p n q q Ş J l consists of rounds in stationary piece l , which requests are not rejected due to shortage in reserved resource units. The summation ř l P L 1 p m ˚ l ' m q yields the reward accrued on pieces where m ˚ l ' m . By the summation ř M ´ 1 m '´ M REW p m q , we obtain the total reward accrued with resources reserved for 2 M intervals.

Similarly, we decompose the benchmark opt(FA) ' ř M ´ 1 m '´ M opt(FA) p m q where

To prove Theorem 3.1, it suffices to show REW p m q ě 1 ´p 2 M ` 1 q{ B 6 α 2 M ¨ opt(FA) p m q for each interval m , as in the following Claim 1 and Claim 2. Then Theorem 3.1 can be established by summing over m P t´ M,...,M ´ 1 u .

Claim 1. For any interval m P t´ M,...,M ´ 1 u , if for all n P t max t m ´ 1 , ´ M u , m u we have ˜ T p n q ' T p n q T , then REW p m q ě 1 2 αM ¨ opt(FA) p m q .

Claim 2. For any interval m P t´ M,...,M ´ 1 u , if for at least one element n P t max t m ´ 1 , ´ M u , m u we have ˜ T p n q Ĺ T p n q T , then REW p m q ě 1 ´p 2 M ` 1 q{ B 6 α 2 M ¨ opt(FA) p m q .

Sketch proofs of Claims 1, 2. Claims 1, 2 are proved in Appendices C.2, C.3 respectively. We first show in Claim 3 in Appendix B.3 that on a stationary piece l P L , there exists a 'correct' q ˚ l P t 0 , . . . , M u , such that when selecting decision x t ' x p q ˚ l q˚ l (the optimal solution to the LP p r p l q , c p l q , η min ¨ α q ˚ l q ), our guess Ratio p q ˚ l q l on the ground-truth Ratio p l q˚ P p α m ˚ l , α m ˚ l ` 1 s satisfies

When taking fractional action x p q ˚ l q˚ l , we consume resources reserved for reward-consumption ratio intervals m ˚ l ´ 1 or m ˚ l . Therefore by our round-robin design, on each stationary piece l such that m ˚ l ' m , at least p t l ´ t l ´ 1 q{p M ` 1 q requests are assigned to intervals m ´ 1 or m , under decision x p q ˚ l q˚ l . It remains to analyze how many requests are fulfilled by resources reserved for the correct interval m ˚ l ' m at the correct reward-consumption ratio Ratio p q ˚ l q l , as discussed in Section 2.3.

For an interval m where ˜ T p m q ' T p m q T , ˜ T p m ´ 1 q ' T p m ´ 1 q T (Claim 1 case), there are still remaining resources reserved for intervals m ´ 1 , m at the end of the horizon. Hence, for each stationary piece l such that m ˚ l ' m , at least p t l ´ t l ´ 1 q{p M ` 1 q requests (consuming B ˚ l {p M ` 1 q resource units) are indeed fulfilled by resources reserved for interval m ´ 1 or m , accruing reward at the reward-consumption ratio of at least Ratio p l q˚ { α according to (4). Summing over all l such that m ˚ l ' m , we have REW p m q ě ř l P L p Ratio p l q˚ { α q ¨ 1 p m ˚ l ' m q ¨ B ˚ l {p M ` 1 q and Claim 1 is validated. For an interval m where there exists some n P t max t m ´ 1 , ´ M u , m u such that ˜ T p n q Ĺ T p n q T (Claim 2 case), the B { 2 M resource units reserved for interval n are depleted before the end of the horizon. In this case, some requests on stationary piece l where m ˚ l ' m may be rejected, but the B {p 2 M q resource units reserved for interval n have been consumed, generating reward at a reward-consumption ratio of at least α n ě α m ´ 1 . Since the total resources that should be consumed w.r.t. interval m under the optimal FA solution is ř l P L 1 p m ˚ l ' m q ¨ B ˚ l ď B , we have REW p m q ě α m ´ 1 ¨ B {p 2 M q ě ř l P L p Ratio p l q˚ { α 2 q ¨ 1 p m ˚ l ' m q ¨ B ˚ l , and Claim 2 is validated.

4 Bandit-feedback stochastic outcome setting

In this section, we consider the original piece-wise stationary Bwk model, where the DM receives bandit feedback on outcomes p R t , C t q , and decisions are randomized.

4.1 Inventory REServing with change monitoring (IRES-CM) Algorithm

In this section, when we say 'Line xx', we refer to a line of Algorithm 2, which displays IRESCM. In the bandit-feedback setting, guessing Ratio p l q˚ requires estimating p r p l q , c p l q q . To do so, we adaptively partition T into exploration rounds and exploitation rounds. In each round t , we conduct exploration with probability γ t ' M a | K | log p 1 { δ qp log p| K |q ` 1 q{ ? Nt (reflected in a Bernoulli random variable U p t q in Line 7), where δ P p 0 , 1 q is a confidence parameter and N is defined in (5). In an exploration round t (Lines 9-13), we uniformly sample an arm a P K and pull it for N consecutive rounds. We update an estimate p ˆ r t p a q , ˆ c t p a qq on p r t p a q , c t p a qq ' p r p l q p a q , c p l q p a qq using the tp R s p a q , C s p a qqu s P T S t p a q information, where T S t p a q denotes the set of the most recent N exploration rounds before round t when arm a is pulled. That is, we set T S t p a q ' t τ P t t ´ s t , . . . , t ´ 1 u : a τ ' a u where s t ' arg max s t ř t ´ 1 τ ' t ´ s 1 p a τ ' a q ' N u . We define

The estimates ˆ r t , ˆ c t have two sources of error: error due to random noise, which decreases with N ; and error due to non-stationarity, which increases with N . We set N according to (5) to balance these two errors. We let T R t denote the set of exploration rounds.

In an exploitation round t (Lines 17-25), we take turns to pull arms according to decision ˆ x p q t q˚ t , which is very similar to Algorithm 1 with p ˆ r t , ˆ c t q in place of p r t , c t q . We define

which can both be interpreted as a guess of Ratio p l q˚ at any round t during stationary piece l . We reserve B {p 2 M q units of resources for each interval m P t´ M,...,M ´ 1 u . In round t , we serve request t using resources reserved for interval ˆ m t such that p:q t P p α ˆ m t , α ˆ m t ` 1 s . If interval ˆ m t has remaining reserved inventory, then we pull arm a t ' a with probability ˆ x p q t q˚ t p a q , or a t ' ˆ x p q t q˚ t in short. We let T I p m q t denote the set of exploitation rounds using resources reserved for interval m . We finally highlight that the major performance difference between IRES and IRES-CM is due to estimating p r t , c t q by p ˆ r t , ˆ c t q , which is detailed in Section 4.3.

4.2 Performance guarantee of IRES-CM

We impose the following assumption on the ranges of B , opt(FA).

a

Assumption 4.1. min t B, opt(FA) u ě ˜ Ω p L | K | NT q , where ˜ Ω p¨q hides multiplicative factors in terms of log α p η max { η min q , log p 1 { δ q , p log p| K |q ` 1 q .

The performance of IRES-CM is as follows:

Theorem 4.2. For any given α ą 1 , with probability at least 1 ´ 2 | K | ¨ p log α p η max { η min q L ` T q δ , IRES-CM achieves a reward of at least

under Assumption 4.1, where o p¨q hides multiplicative factors in terms of a M { B and ˜ O p¨q hides multiplicative factors in terms of log α p η max { η min q , log p 1 { δ q , p log p| K |q ` 1 q . In particular, IRES-CM achieves a competitive ratio of O p log α p η max { η min qq as long as L ' o p ? T ¨ η min q .

The proof of Theorem 4.2 can be found in Appendix C.4. We provide a thorough comparison of our performance guarantee with existing works on adversarial and non-stationary Bwk in Appendix A. Remark 4.3 (Improved performance with known L ) . If the DM knows L , Assumption 4.1 can be relaxed to

Furthermore, in our performance guarantee in Theorem 4.2, the deductive term ˜ O p L a | K | NT q from opt(FA) can be improved to ˜ O p a L | K | NT q by setting the exploration parameter γ t ' M a L | K | log p 1 { δ qp log p| K |q ` 1 q{ ? Nt in IRES-CM. Without prior knowledge of L , the deductive term ˜ O p L a | K | NT q ' o p T q if L ' o p ? T ¨ η min q ; with prior information of L , the deductive term ˜ O p a L | K | NT q ' o p T q if L ' o p T ¨ η min q .

Remark 4.4 (Deterministic setting with bandit feedback) . In our full-feedback deterministic setting, since p r p l q , c p l q q is given at the beginning of each stationary piece, our performance guarantee is independent on | K | , L . In a bandit-feedback deterministic setting, IRES-CM can be applied by setting N ' 1 . In this case, under Assumption 4.1, IRES-CM achieves a reward of at least

4.3 Analysis

We denote σ t p a q ' min t s : s P T S t p a qu as the 1st element in T S t p a q . We partition the exploitation round set T I p m q T into two sets q T I p m q and p T I p m q , i.e., T I p m q T ' q T I p m q Ť p T I p m q , q T I p m q Ş p T I p m q ' H . A time index t P T I p m q T belongs to the set q T I p m q (referred to as 'successful exploitation rounds regarding interval m ') if and only if the following condition is satisfied for all a P K :

For t P p T I p m q (referred to as 'failed exploitation rounds regarding interval m '), inequality (6) is violated for at least one a P K . We denote p T I p m q ' t τ I p m q p 1 q , τ I p m q p 2 q , . . . u where τ I p m q p 1 q ă τ I p m q p 2 q ă . . . . We let ˜ T I p m q be a prefix of p T I p m q satisfying

which consists up to the last exploitation round satisfying (6) for interval m , such that the reserved resource is adequate. If ř s P T I p m q T C s p a s q ď B {p 2 M q ´ 1 , then ˜ T I p m q ' q T I p m q .

Sketch proof of Theorem 4.2. Recall that the performance guarantee of our algorithms is in the form of ř T t ' 1 R t p a t q ě 1 CR ¨ opt p FA q ´ Reg. The proof consists of mainly two steps: (a) we derive the CR ' O p M q by bounding two different cases of interval m in a similar manner to Claim 1 and Claim 2 (see Appendix C.4), with q T I p m q (successful exploitation rounds in IRES-CM) in place of T p m q T (all rounds in IRES); (b) we derive the Reg ' ˜ O p L a | K | NT q by bounding the number of exploration rounds | T R T | and failed exploitation rounds | Ť M m '´ M ´ 2 p T I p m q | (see Appendix B.7).

Comparing performance of IRES and IRES-CM. We highlight that the major performance difference between IRES and IRES-CM is the loss caused by estimating p r t , c t q , reflected in the following aspects: (i) reward loss caused by exploration (upper bounding | T R T | ); (ii) T S t p a q contains change points, causing failed estimation of p r t , c t q (upper bounding | Ť M m '´ M ´ 2 p T I p m q | ); (iii) T S t p a q does not contain change points, but the discrepancy between p r t , c t q and p ˆ r t , ˆ c t q results in assigning Ratio p l q˚ (estimated by z Ratio p q q t ) to the wrong interval. We remark that the losses due to (i, ii) are accounted for in Reg, while (iii) is accounted for in the CR.

4.4 A lower bound on competitive ratio

We complement our analysis by showing the tightness of our CR (see Appendix C.6 for proof).

Theorem 4.5. Consider a fixed but arbitrary α ą 1 , and set η min ' α ´ 3 ν , η max ' 1 for an arbitrary ν P Z ą 0 . For any online algorithm, there exist an instance for which η min ď r t p a q , c t p a q ď η max for all a P K , t P T , and that ř T t ' 1 R t p a t q{ opt(FA) ď Θ p 1 { log α p η max { η min qq .

5 Numerical Experiments

We run numerical experiments on a single-resource problem where L ' 2 , T ' 20000 (each stationary piece has 10000 rounds), K ' t 1 , 2 u , B ' 9360 and we set α ' e for our algorithms. The rewards and resource consumption in all rounds are uniformly distributed within a r´ 0 . 2 , ` 0 . 2 s range from their mean values. We compare the performance of IRES-CM with Immorlica et al. (2019)'s algorithm and Zhou et al. (2008)'s algorithm. Recall that Immorlica et al. (2019) focus on an adversarial Bwk problem and achieves a CR w.r.t. a static benchmark. Zhou et al. (2008) study a full-feedback adversarial setting and achieves a CR w.r.t. a single best arm benchmark. In Figure 1, each curve represents the average cumulative reward over 10 simulations, and the shaded area around each curve marks the variance over the simulations. We provide Zhou et al. (2008)'s algorithm with extra information of p r t , c t q before making decisions in each round, and compare the performance of algorithms with the linear program benchmark FA (dotted curves in Figure 1).

(a) FA optimal solution: single arm

(b) FA optimal solution: mixed arms

Performancecomparisonofalgorithms

le3

8

IRES

Cumulativereward

6

4

2

0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Time step t

1e4

Performance comparison of algorithms

1e4

1.2

1.0

IRES

reward

0.8

Cumulative

0.6

0.4

0.2

0.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Time step t

1e4

In Figure 1(a), we set r p 1 q p 1 q ' r p 1 q p 2 q ' 0 . 5 , c p 1 q p 1 q ' c p 1 q p 2 q ' 1 for stationary piece 1; and set r p 2 q p 1 q ' 1 , r p 2 q p 2 q ' 0 . 5 , c p 2 q p 1 q ' 0 . 5 , c p 2 q p 2 q ' 1 for stationary piece 2. In Figure 1(b), we switch the values of r p 2 q p 1 q and r p 2 q p 2 q . i.e., setting r p 2 q p 1 q ' 0 . 5 , r p 2 q p 2 q ' 1 . Observe that IRES-CM outperforms Immorlica et al. (2019)'s algorithm in both cases. This is mainly because Immorlica et al. (2019)'s algorithm is designed for a more general adversarial Bwk setting. In contrast, we utilize the extra information that η min ' 0 . 5 . Therefore, Immorlica et al. (2019)'s algorithm is significantly more conservative than IRES-CM in reserving inventories for future customers. Zhou et al. (2008)'s algorithm outperforms IRES-CM in Figure 1(a), but performs worse than IRES-CM in Figure 1(b). This is because that in Figure 1(a), the optimal solution of the benchmark FA chooses a single arm on each stationary piece, which aligns with Zhou et al. (2008)'s single best arm benchmark. Zhou et al. (2008)'s algorithm performs well with the extra information of p r t , c t q before making decisions. In Figure 1(b), the optimal solution of the benchmark FA chooses mixed arms on the second stationary piece, where x ˚ 2 p 1 q ' 0 . 128 , x ˚ 2 p 2 q ' 0 . 872 . The numerical results are consistent with the theoretical results that Zhou et al. (2008) achieve sub-optimal rewards compared with a best distribution over arms benchmark, while our IRES-CM performs well. Finally, our experiments are run on a Surface Pro 7 with an i5-1035G4 processor. All results can be produced within 30 minutes.

References

Agrawal, S., Avadhanula, V., Goyal, V., and Zeevi, A. (2019). Mnl-bandit: A dynamic learning approach to assortment selection. Operations Research , 67(5):1453-1485.

Agrawal, S. and Devanur, N. (2016). Linear contextual bandits with knapsacks. Advances in Neural Information Processing Systems , 29.

Agrawal, S. and Devanur, N. R. (2014). Bandits with concave rewards and convex knapsacks. In Proceedings of the fifteenth ACM conference on Economics and computation , pages 989-1006.

Babaioff, M., Dughmi, S., Kleinberg, R., and Slivkins, A. (2015). Dynamic pricing with limited supply.

Badanidiyuru, A., Kleinberg, R., and Slivkins, A. (2018). Bandits with knapsacks. Journal of the ACM (JACM) , 65(3):1-55.

Badanidiyuru, A., Langford, J., and Slivkins, A. (2014). Resourceful contextual bandits. In Conference on Learning Theory , pages 1109-1134. PMLR.

Balseiro, S. R. and Gur, Y. (2019). Learning in repeated auctions with budgets: Regret minimization and equilibrium. Management Science , 65(9):3952-3968.

Balseiro, S. R., Lu, H., and Mirrokni, V. (2022). The best of many worlds: Dual mirror descent for online allocation problems. Operations Research .

Castiglioni, M., Celli, A., and Kroer, C. (2022a). Online learning with knapsacks: the best of both worlds. In International Conference on Machine Learning , pages 2767-2783. PMLR.

Castiglioni, M., Celli, A., Marchesi, A., Romano, G., and Gatti, N. (2022b). A unifying framework for online optimization with long-term constraints. Advances in Neural Information Processing Systems , 35:33589-33602.

Fikioris, G. and Tardos, É. (2023). Approximately stationary bandits with knapsacks. arXiv preprint arXiv:2302.14686 .

Immorlica, N., Sankararaman, K. A., Schapire, R., and Slivkins, A. (2019). Adversarial bandits with knapsacks. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS) , pages 202-219. IEEE.

Jiang, J., Li, X., and Zhang, J. (2020). Online stochastic optimization with wasserstein based non-stationarity. arXiv preprint arXiv:2012.06961 .

Karp, R. M., Vazirani, U. V., and Vazirani, V. V. (1990). An optimal algorithm for on-line bipartite matching. In Proceedings of the twenty-second annual ACM symposium on Theory of computing , pages 352-358.

Kesselheim, T. and Singla, S. (2020). Online learning with vector costs and bandits with knapsacks. In Conference on Learning Theory , pages 2286-2305. PMLR.

Kim, W., Iyengar, G., and Zeevi, A. (2023). Improved algorithms for multi-period multi-class packing problems with bandit feedback. In International Conference on Machine Learning , pages 16458-16501. PMLR.

Kuszmaul, W. and Qi, Q. (2021). The multiplicative version of azuma's inequality, with an application to contention analysis. arXiv preprint arXiv:2102.05077 .

Liu, S., Jiang, J., and Li, X. (2022). Non-stationary bandits with knapsacks. Advances in Neural Information Processing Systems , 35:16522-16532.

Mehta, A., Saberi, A., Vazirani, U., and Vazirani, V. (2007). Adwords and generalized online matching. Journal of the ACM (JACM) , 54(5):22-es.

Rangi, A., Franceschetti, M., and Tran-Thanh, L. (2018). Unifying the stochastic and the adversarial bandits with knapsack. arXiv preprint arXiv:1811.12253 .

Sankararaman, K. A. and Slivkins, A. (2018). Combinatorial semi-bandits with knapsacks. In International Conference on Artificial Intelligence and Statistics , pages 1760-1770. PMLR.

Sivakumar, V., Zuo, S., and Banerjee, A. (2022). Smoothed adversarial linear contextual bandits with knapsacks. In International Conference on Machine Learning , pages 20253-20277. PMLR.

Yao, A. C.-C. (1977). Probabilistic computations: Toward a unified measure of complexity. In 18th Annual Symposium on Foundations of Computer Science (sfcs 1977) , pages 222-227. IEEE Computer Society.

Zhalechian, M., Keyvanshokooh, E., Shi, C., and Van Oyen, M. P. (2022). Online resource allocation with personalized learning. Operations Research , 70(4):2138-2161.

Zhou, Y., Chakrabarty, D., and Lukose, R. (2008). Budget constrained bidding in keyword auctions and online knapsack problems. In Proceedings of the 17th international conference on world wide web , pages 1243-1244.

A Comparing our performance guarantee with existing literature

A.1 Comparing with adversarial BwKs

Immorlica et al. (2019) (achieving O p d log p T qq competitive ratio) and Kesselheim and Singla (2020) (achieving O p log p d q ¨ log p T qq competitive ratio) study the adversarial BwKs, where the adversarial bandit feedback p r t p a t q , c t p a t qq is revealed in each time step after pulling arm a t . Their setting is more general than ours, since they require no boundedness assumption or piece-wise stationary assumption on their outcomes. Note that if our η min can be expressed as a function of T , i.e., η min ' T ´ β for a β P p 0 , 1 q , our result can extend to multiple resources and achieve a competitive ratio of O p d log p T qq (see Appendix B.5). However, we highlight that their result do not imply our result, due to the following two reasons.

Firstly, they consider a static benchmark, dubbed FD, which is FA with the additional constraint that x ˚ l ' x ˚ for all l P L ; while we compare our result with the dynamic benchmark FA. Specifically, Immorlica et al. (2019) prove that no algorithm can achieve a competitive ratio smaller than T { B 2 w.r.t. the dynamic benchmark. We further improve this lower bound to Θ p T { B q in Lemma 2.3(a) by setting L ' T { B , and provide a matching lower bound of Ω p log p T qq comparing with opt(FA) when η min ą 0 . Secondly, they have more restrictive assumptions on the value ranges of B and opt(FD). Specifically, Immorlica et al. (2019) assume opt(FD) ¨ B {| K | ą ˜ Ω p T 7 { 4 q , which is strictly stronger than our Assumption 4.1.

Some research works consider more specific regimes, including Rangi et al. (2018); Castiglioni et al. (2022a); Fikioris and Tardos (2023). We highlight that all these works still compare with static benchmarks. Castiglioni et al. (2022a) focus on the regime where B ' Ω p T q and achieves a competitive ratio of T { B . Fikioris and Tardos (2023) provides a competitive ratio depending on p max t t r t u{ min t t r t u , max t t c t u{ min t t c t uq . Rangi et al. (2018) achieves a sublinear-inT regret in a different setting with no round limit (sales stop when the inventory is depleted). Therefore, their result is incomparable with ours.

Some recent papers focus on linear contextual Bwk with adversarial contextual vectors Sivakumar et al. (2022) (achieving O p d log p T qq competitive ratio) or with multiple but stationary customer classes Kim et al. (2023) (achieving sublinear in T regret). We highlight that contextual vectors for each arm are observable before making decision in each round, while our model only observe bandit feedback after an arm is chosen. Therefore their results do not generalize to our setting.

A.2 Comparing with adversarial online knapsack problems with full feedback

In our Section 3, we discuss a warm-up setting where t R t p a q , C t p a qu a P K is observable upon the arrival of the round t customer, which is similar with adversarial online knapsack with full feedback (Karp et al. (1990); Mehta et al. (2007); Zhou et al. (2008)). Zhou et al. (2008) is the most closely related to our work, where they start off from an online matching problem and extends the results to the online knapsack problem. They define LB ' min a,t t r t p a q{ c t p a qu , UB ' max a,t t r t p a q{ c t p a qu and achieves a competitive ratio of log p UB { LB q w.r.t. a dynamic best single arm benchmark, which differs from our FA by setting x l p a q P t 0 , 1 u for each a, l . Our FA, on the other hand, is a best distribution over arms benchmark. In fact, for stationary Bwk, Badanidiyuru et al. (2018) show (see their Appendix A) that the best single arm benchmark is strictly weaker than best distribution over arms , which could noticably affect the achievable CR. It is evident that in the picewise-stationary setting, this is also the case.

Additionally, this line of works crucially require knowing tp R t p a q , C t p a qqu a P K before making decisions. By contrast, we take a different approach in algorithm design in Section 3 allows a natural generalization from full to bandit feedback shown in Section 4.

A.3 Comparing with non-stationary bandit/full-feedback online optimization with knapsacks

In Balseiro et al. (2022); Jiang et al. (2020); Liu et al. (2022), they measure the non-stationarity of a time-varying knapsack model with a quantity called the global variation

While dist can be any metric, to give a more concrete idea, we highlight an example form Liu et al. (2022) being

They all have boundedness assumption on glo . In our setting ( glo unbounded), their algorithms incur a linear-inT regret even when L ' 1 , and no non-trivial competitive ratio is established in their work. To see the case of L ' 1 but glo ' Θ p T q , consider the case of K ' t 1 u , and we have

but

In this case, we can verify that

so we have glo ' 0 . 5 T despite L ' 1 .

B Auxiliary results

B.1 Notation

For functions f p x q : R ě 0 Ñ R ě 0 and g p x q : R ě 0 Ñ R ě 0 , we say f p x q is O p g p x qq (resp. Ω p g p x qq ) if there exist positive constants C and n , such that for all x ě n , f p x q ď C ¨ g p x q (resp. f p x q ě C ¨ g p x q ). We use ˜ O p g p x qq (resp. ˜ Ω p g p x qq ) to hide logarithmic terms in x other than g p x q .

B.2 Concentration inequalities

Lemma B.1 (Multiplicative Azuma-Hoeffding Inequality (Kuszmaul and Qi (2021))) . X 1 , . . . , X n P r 0 , c s are real-valued random variables, and t F n u n i ' 0 is a filtration. Let µ ' ř n i ' 1 a i where a i are real-valued constants.

(i) Suppose E r X i | F i ´ 1 s ď a i holds for all i P t 1 , . . . , n u almost surely. Then for any δ P p 0 , 1 q ,

(ii) Suppose E r X i | F i ´ 1 s ě a i holds for all i P t 1 , . . . , n u almost surely. Then for any δ P p 0 , 1 q ,

Lemma B.2 (Lemma 2.1, Badanidiyuru et al. (2018)) . Let X 1 , . . . , X N P r 0 , 1 s be random variables. Let X ' ř N i ' 1 X i be the sample average, and let µ ' ř N i ' 1 E r X i | X 1 , . . . , X N s . Then, for any δ P p 0 , 1 q ,

B.3 Main claim on reward-consumption ratio

Recall that x p q q˚ l is an optimal solution to LP p r p l q , c p l q , η min ¨ α q qq , and x ˚ l is an optimal solution of LP p r p l q , c p l q , B ˚ l {p t l ´ t l ´ 1 qq which is an optimal solution of our benchmark. In the following Claim 3, we show that the round-robin technique in IRES ensures that, on each stationary piece l , x p q q˚ l for at least one q P t 0 , . . . , M u is close to x ˚ l in terms of both the resource consumption and the reward-consumption ratio. This leads to the important result that for all t P t t l ´ 1 ` 1 , . . . , t l u , m t P t m ˚ l ´ 1 , m ˚ l u .

Claim 3. On stationary piece l P L , there exists q ˚ l P t 0 , 1 , . . . , M u that satisfies both

and

Proof of Claim 3. The existence of q ˚ l satisfying (7) is evident, since η min ¨ α 0 ' η min and η min ¨ α M ' η max . To show that q ˚ l also satisfy (8), we define

and claim that

The first inequality in (10) holds since B ˚ l {p t l ´ t l ´ 1 q ' ř a P K c p l q p a q x ˚ l p a q ď η min ¨ α q ˚ l . Therefore, the resource constraint in LP p r p l q , c p l q , B ˚ l {p t l ´ t l ´ 1 qq is tighter than LP p r p l q , c p l q , η min ¨ α q ˚ l q . We prove the second inequality in (10) by contradiction. Suppose ř a P K r p l q p a q x p q ˚ l q˚ l p a q ą d p l q ř a P K r p l q p a q x ˚ l p a q . Then we set x l ' x p q ˚ l q˚ l { d p l q and have

In this case, x l is a feasible solution to LP p r p l q , c p l q , B ˚ l {p t l ´ t l ´ 1 qq and we have

contradicting the fact that x ˚ l is an optimal solution of LP p r p l q , c p l q , B ˚ l {p t l ´ t l ´ 1 qq . Therefore, combining (9) and (10), we establish

To establish (8), it suffices to show d p l q ď α . By (7) and (9), it is evident that d p l q ď α when q ˚ l ą 0 . If q ˚ l ' 0 , then constraints ř a P K x ˚ l p a q ď 1 and ř a P K x p q ˚ l q˚ l p a q ď 1 are not tight in both LP p r p l q , c p l q , B ˚ l {p t l ´ t l ´ 1 qq and LP p r p l q , c p l q , η min q . In this case, both LPs are knapsack problems and have closed-form solutions of

Therefore, we have

which shows that d p l q ' 1 ă α when q ˚ l ' 0 .

B.4 Decomposing ratio of REW to opt(FA) in deterministic setting

Recall

and J l ' t t l ´ 1 ` 1 , . . . , t l u . Then for the reward achieved by IRES, we have

Inequality (11) holds since by summing over w P t max t m ´ 1 , ´ M u , m, min t m ` 1 , M ´ 1 uu for each m P t´ M,...,M ´ 1 u , we repeat the sum for at most 3 times.

For the reward achieved by FA, we have

Inequality (12) follows from (2).

Therefore, the ratio of the reward achieved by IRES to opt(FA) can be decomposed as:

Then, to prove Theorem 3.1, it suffices to show

To prove (14), we focus on showing in Claims 1 and 2.

B.5 Extending results to multiple resources

Our results can be readily extend to the multiple-resource case, with | I | ' d resources indexed by i P I . An upper bound for a multi-resource allocation problem (corresponding to FA in the single-resource setting) can be formulated as:

which is evidently upper bounded by the following LP:

It is evident that the LP below achieves a reward of at least 1 { d fraction of opt(FA MUL-U q .

Therefore, FA MUL is transformed into a single-resource allocation problem FA MUL-F , with an extra multiplicative factor d in the competitive ratio.

B.6 Core lemma on reward-consumption ratio in general setting

Recall that set T S t p a q ' t τ P t t ´ s t , . . . , t ´ 1 u : a τ ' a u where s t ' arg max s t ř t ´ 1 τ ' t ´ s 1 p a τ ' a q ' N u consists of the most recent N rounds where arm a is sampled by exploration; and σ t p a q ' min t s : s P T S t p a qu is the 1 -st element in T S t p a q . Recall that

are the average resource consumption and the reward earned for the most recent N pulls of arm a . Additionally, recall that we define ˆ x p q q t ' t ˆ x p q q t p a qu a P K as a solution to LP p ˆ r t , ˆ c t , η min ¨ α q q ( ˆ x p q q˚ t being the optimal solution) for q ' 0 , 1 , . . . , M ´ 1 .

We further define

as the average mean resource consumption and the reward earned for the most recent N pulls of arm a . Note that when condition (6)

is satisfied for all a P K , then rounds s P t σ t , . . . , t u are on the same stationary piece and we have ¯ c t p a q ' c t p a q , ¯ r t p a q ' r t p a q . We let ¯ x p q q t ' t ¯ x p q q t p a qu a P K be a solution to LP p r t , c t , η min ¨ α q q ( ¯ x p q q˚ t being the optimal solution) for q ' 0 , 1 , . . . , M ´ 1 . By Claim 3, we know that there exists q ˚ l P t 0 , 1 , . . . , M u such that for all t P J l ,

We show in the following Lemma B.3 that when condition (6) is satisfied for all a P K , our decisions have several nice properties which facilitate our proofs.

Lemma B.3. Fix an arbitrary α P p 0 , 1 s . For any t ě σ t p a q , if

then with probability at least 1 ´ 2 | K | δ , then all the following inequalities are satisfied for all t P J l :

Proof of Lemma B.3. We define

Then by the multiplicative Azuma-Hoeffding inequality (Lemma B.1), for any a P K we have

The above probability bounds (19) and (20) hold since

We set p 1 ´ 3 ϵ q 2 ' 1 { α , and therefore

Our following discussion is conditioned on the good event that both (19) and (20) hold for all a P K . This good event holds with probability 1 ´ 2 | K | δ .

Validating (16). Given that (19) and (20) hold, we have

Since 1 ´ 2 ϵ ě 1 ´ 3 ϵ ' 1 { α , (21) indicates (16).

Inequality (22) follows from inequalities (19). Since ˆ x p q ˚ l q t p a q ' ¯ x p q ˚ l q˚ t p a q{p 1 ` ϵ q is a feasible solution to LP p ˆ r t , ˆ c t , η min ¨ α q ˚ l q , we have

Similarly, by letting ¯ x p q ˚ l q t p a q ' p 1 ´ ϵ q ˆ x p q ˚ l q˚ t p a q , we have

Since ¯ x p q ˚ l q t p a q ' p 1 ´ ϵ q ˆ x p q ˚ l q˚ t p a q is a feasible solution to LP p r t , c t , η min ¨ α q ˚ l q , we have

By (10) in Claim 3, we have

Given (25) and (26), we prove (17).

Validating (18). For l P L such that q ˚ l ' 0 , we have ř a P K c t p a q ¯ x p q ˚ l q˚ t p a q ' η min . Given (19), for all t P J l ,

Putting (23) and (24) together, we have

Inequality (27) and (28) gives

Recall from (8) in Claim 3, for all t P J l we have

Putting (27) and (30) together, we establish (18).

B.7 Bounding exploration rounds and failed exploitation rounds

To upper bound | T R T Ť p Ť M m '´ M ´ 2 p T I p m q q| , it suffices to upper bound | T R T | (forthcoming Claim 4) and | Ť M ` 1 m '´ M ´ 2 p T I p m q | (forthcoming Claim 5).

Claim 4. For any δ P p 0 , 1 q , | T R T | ď TNγ T ` N b 3 Tγ T log ` 2 δ ˘ with probability at least 1 ´ δ .

Proof of Claim 4. Define the event E R t ' t Conduct exploration in round t u ' E R p 1 q t Ť E R p 2 q t , where E R p 1 q t ' tD a P K s.t. t ' σ t p a qu and E R p 2 q t ' tD a P K s.t. t P T S t p a qz σ t p a qu . We define random variables Y R t ' 1 p E R t q , Y R p 1 q t ' 1 p E R p 1 q t q and Y R p 2 q t ' 1 p E R p 2 q t q . Define event E I t ' t Conduct exploitation in round t u ' E I p 1 q t Ť E I p 2 q t , where E I p 1 q t ' t q t ' 0 u and E I p 2 q t ' t q t ą 0 u . We define random variables Y I t ' 1 p E I t q , Y I p 1 q t ' 1 p E I p 1 q t q and Y I p 2 q t ' 1 p E I p 2 q t q .

We further define random variables Z t , where Z t ' Y R p 1 q t ' Bern p γ t q for all t such that Y R p 1 q t ' 1 or Y I p 1 q t ' 1 and Z t ' Bern p γ t q otherwise. It is evident that in each round t , Z t follows a Bernoulli distribution with mean γ t and Z t ě Y R p 1 q t . Therefore, the total rounds of forced exploration over the planning horizon can be upper bounded as

Since Z t are independent and E r ř T t ' 1 Z t s ' ř T t ' 1 γ t ď 2 Tγ T , we can apply the Chernoff bound (which is a subcase of Lemma B.1) on Z t ,

Hence we have | T R T | ' ř T t ' 1 Y R t ď 2 TNγ T ` N a 6 Tγ T log p 2 { δ q with probability at least 1 ´ δ .

Claim 5. Fix any δ P p 0 , 1 q , | p T I p m q | ď L | K | log p 1 { δ qp log p| K |q ` 1 q{ γ T with probability at most 1 ´| K | Lδ .

Proof of Claim 5. If round t is a change point, i.e., p r t , c t q ‰ p r t ´ 1 , c t ´ 1 q , we define set K p t q Ă K such that p r t p a q , c t p a qq ‰ p r t ´ 1 p a q , c t ´ 1 p a qq for all a P K p t q . Notice that after a change happens, some exploitation rounds could run with condition (6) violated, resulting in consuming resources from the wrong reward-consumption interval m . Hence, | p T I p m q | can be upper bounded by the total number of exploitation rounds run before all arms are updated after each change point. After all arms are updated after a change point, (6) is satisfied again. In the following Claim 6, we suppose round t is a change point, and upper bound the number of exploitation rounds run before all arms a P K p t q are updated.

Claim 6. Suppose round t is a change point. With probability 1 ´| K p t q| δ , IRES-CM run at most | K | log p 1 { δ qp log p| K p t q|q ` 1 q{ γ T exploitation rounds (Algorithm 2, Lines 17-25) before updating the change in exploration rounds (Algorithm 2, Lines 9-13).

Proof of Claim 6. For some set K 1 Ă K , let random variable Y p K 1 q denote the number of exploitation samples (i.e., line 6 of Algorithm 2 giving U p t q ' 0 ) between two nearest exploration samples (i.e., line 6 of Algorithm 2 giving U p t q ' 1 ) applied for arms a P K 1 . We denote K p t q p j q as the j -th arm explored from set K p t q . After each exploitation sample, IRES-CM runs for each q P t´ M,...,M ´ 1 u . Therefore, the number of exploitation rounds run before all a P K p t q are updated is 2 M ¨ p Y p K p t qq ` ř | K p t q| j ' 2 Y p K p t qz Ť j ´ 1 h ' 1 K p t q p h q qq . For any subset K 1 P K p t q , we further denote random variable Z p K 1 q as Y p K 1 q plus the number of times that exploration is triggered for any a P K z K 1 between two nearest exploration rounds applied for arms a P K 1 .

It is evident that Z p K 1 q is a geometric random variable with time-varying probability p t ' γ t | K 1 |{| K | of success in each round. Therefore, we have Pr p Z p K 1 q ě n q ' ś t ` n ´ 1 s ' t p 1 ´ p s q . Since for any x P r 0 , 1 s , it holds that 1 ´ x ď e ´ x , by requiring e ´ n ¨ p T ď δ , we have ś t ` n ´ 1 s ' t p 1 ´ p s q ď p 1 ´ p T q n ď e ´ n ¨ p T ď δ . In this case,

Therefore, we have

which suggests that with probability at least δ , we run at most | K | log p 1 { δ q{p γ T | K 1 |q exploitation rounds before updating p ˆ r t p a q , ˆ c t p a qq for each arm a P K 1 .

Plugging K p t qz Ť j ´ 1 h ' 1 K p t q p h q in K 1 , with probability 1 ´| K p t q| δ ,

Inequality (31) holds since

Notice that there are at most L change points over the entire planning horizon, contributing to p T I p m q for at most 2 ML | K | log p 1 { δ qp log p| K |q ` 1 q{ γ T rounds, with probability at most 1 ´| K | Lδ .

Combining Claim 4 and 5, with probability at least 1 ´ 2 M | K | Lδ ,

Recall that N ' 27 log p 2 { δ q{pp 1 ´ 1 { α q 2 ¨ η min q and γ t ' M a | K | log p 1 { δ qp log p| K |q ` 1 q{ ? Nt , we further have

Note that if the DM knows L a priori, then we can set γ t ' M a L | K | log p 1 { δ qp log p| K |q ` 1 q{ ? Nt , which results in | T R T Ť p Ť M ´ 1 m '´ M p T I p m q q| ď ˜ O p a L | K | NT q .

C Proofs

C.1 Proof of Lemma 2.1

Let π be a non-anticipatory feasible policy that achieves the expected optimum opt(DP) in DP, i.e. E r ř T t ' 1 ř a P K R t p a q X π t p a qs ' E r opt(DP) s where X π t is the decision variable under algorithm π . We let

for each l ' 1 , . . . , L in FA. We claim that t x l u L l ' 1 is feasible to FA, with objective value equal to E r ř T t ' 1 ř a P K R t p a q X π t p a qs ' E r opt(DP) s , which indicates that under t x ˚ l u L l ' 1 we have opt(FA) ě E r opt(DP) s . Thus, verifying the claims about the feasibility and the objective value proves the claim.

We first verify the feasibility to FA. Since the policy π satisfies the resource constraints, the inequality ř T t ' 1 ř a P K C t p a q X π t p a q ď B holds. Taking expectation over X π t p a q and C t p a q for t ' t l ´ 1 ` 1 , . . . , t l gives

Similarly, by taking expectation over each of the reward constraints, we have E r ř T t ' 1 ř a P K R t p a q X π t p a qs ' ř L l ' 1 p t l ´ t l ´ 1 q ř a P K r p l q p a q x l p a q ' E r opt(DP) s . Hence, the claim about the objective value is shown, and the Lemma is proved. ˝

C.2 Proof of Claim 1

Recall that in Algorithm 1, we try each q P t 0 , 1 , . . . , M u in a round-robin manner. Then we know that on each stationary piece l , for at least p t l ´ t l ´ 1 q{ M ´ 1 rounds, we choose q ' q ˚ l and take fractional decision x p q ˚ l q˚ l such that (7) and (8) hold. By Claim 3 inequality (8), resources consumed under decision x p q ˚ l q˚ l are assigned with resources reserved for intervals t max t m ˚ l ´ 1 , ´ M u , m ˚ l u . Therefore, for interval m where m ˚ l ' m ,

Recall that if we choose the optimal decision x ˚ l on stationary piece l , B ˚ l units of resources would be consumed, i.e. p t l ´ t l ´ 1 q ř a P K c p l q p a q x ˚ l p a q ' B ˚ l . Hence by Claim 3 inequality (7), we have

Putting everything together, we have

Inequality (36) holds by plugging in (35) and (8). Inequality (37) holds by plugging in (34). Inequality (38) stands since we can assume t l ´ t l ´ 1 ě 2 p M ` 1 q without loss of generality. Because otherwise we can ignore the stationary pieces where t l ´ t l ´ 1 ď 2 p M ` 1 q , causing a reward loss of at most O p M q . Following (38), we have

Therefore, for m P t´ M,...,M ´ 1 u such that ˜ T p n q ' T p n q T for n P t max t m ´ 1 , ´ M u , m u , we have

˝

C.3 Proof of Claim 2

We let ˜ m ' min n t n P t max t m ´ 1 , ´ M u , m u , ˜ T p n q Ĺ T p n q T u . Then we have

Inequality (40) stands since in rounds t P ˜ T p ˜ m q , we have ř a P K r t p a q x t p a q{ ř a P K c t p a q x t p a q P r α ˜ m , α ˜ m ` 1 s . Inequality (41) holds since ř t P ˜ T p ˜ m q ř a P K c t p a q x t p a q ě p B ´ T ¨ η min q{p 2 M q ´ 1 ' p B ´ 1 q{p 2 M q ´ 1 by the definition of ˜ T p ˜ m q . Then we have

Inequality (42) holds since m t P tt max t m ˚ l ´ 1 , ´ M u , m ˚ l u for all t P t t l ´ 1 ` 1 , . . . , t l u . Therefore, it is possible to consume resources reserved for interval ˜ m P t max t m ´ 1 , ´ M u , m u under x p q ˚ l q˚ l only when m ˚ l P t max t m ´ 1 , ´ M u , m ´ 1 ` 1 u ď t m, min t m ` 1 , M ´ 1 uu ' t max t m ´ 1 , ´ M u , m, min t m ` 1 , M ´ 1 uu .

We also have

Putting together (42) and (43), we know that for m P t´ M,...,M ´ 1 u satisfying case (ii), we have

Inequality (44) holds since we require B ě Ω p M q (see Theorem 3.1). Combining (39) and (44), we show that

C.4 Proof of Theorem 4.2

Note that for reward-consumption ratio intervals n P t´ M,...,M ´ 1 u where ˜ T I p n q ' q T I p n q , not all requests assigned to these intervals are necessarily satisfied. This is because resources could run out due to exploration before ř s P q T I p n q C s p a s q ą B {p 2 M q ´ 1 , i.e., when

It can be seen that requests in rounds p Ť M ´ 1 n '´ M ˜ T I p n q q Ş t t P T : ř t s ' 1 C s p a s q ď B ´ 1 u are satisfied. Therefore, we decompose the ratio of the IRES-CM reward to opt(FA) as follows:

To establish Theorem 4.2, it suffices to show H(1) ě p 1 ´ o p 1 qq ¨ p opt(FA) ´ ˜ O p a L | K | NT qq{p 10 α 4 M ¨ opt(FA) q (see the forthcoming Section C.4.1) and H(2) ě 1 ´ o p 1 q (see the forthcoming Section C.4.2).

C.4.1 Bounding H(1)

Recall that ř a P K ˆ r t p a q ˆ x p q t q˚ t p a q{ ř a P K ˆ c t p a q ˆ x p q t q˚ t p a q P r α ˆ m t , α ˆ m t ` 1 s . We further define ˆ m p l q˚ t such that ř a P K ˆ r t p a q ˆ x p q ˚ l q˚ t p a q{ ř a P K ˆ c t p a q ˆ x p q ˚ l q˚ t p a q P r α ˆ m p l q˚ t , α ˆ m p l q˚ t ` 1 s . Likewise, we define m p l q˚ t such that ř a P K r t p a q ˆ x p q ˚ l q˚ t p a q{ ř a P K c t p a q ˆ x p q ˚ l q˚ t p a q P r α m p l q˚ t , α m p l q˚ t ` 1 s . We define

Then H(1) can be further decomposed as:

Inequality (47) holds since by summing over w P t max t m ´ 2 , ´ M u , . . . , min t m ` 2 , M ´ 1 uu , we repeat the numerator of (46) for at most 5 times.

We partition set t´ M,...,M ´ 1 u into two disjoint sets M 1 , M 2 . An interval m P M 1 if for all n P t max t m ´ 1 , ´ M u , m, min t m ` 1 , M ´ 1 uu , we have ˜ T I p n q ' q T I p n q , i.e. ř s P q T I p n q C s p a s q ď B {p 2 M q ´ 1 . An interval m P M 2 if for some n P t max t m ´ 1 , ´ M u , m, min t m ` 1 , M ´ 1 uu , we have ˜ T I p n q Ĺ q T I p n q .

Regarding m P M 1 . In the following analysis, we focus on the good event that (16), (17), (18) in Lemma B.3 hold for all t P T . We know that with probability at least 1 ´ 2 | K | Tδ , the good event holds. On each stationary piece l P L and all t P J l , for all rounds t P p Ť M ´ 1 n '´ M ˜ T I p n q q Ş J l such that m p l q˚ t ' m , we know that ˆ m p l q˚ t P t max t m ´ 1 , ´ M u , m, min t m ` 1 , M ´ 1 uu (see (16) in Lemma B.3). Due to the round-robin technique, at least 1 {p M ` 1 q fraction of all rounds t P ´ Ť M ´ 1 n '´ M ˜ T I p n q ¯ Ş J l , l P L such that m p l q˚ t ' m are allocated by resources reserved for intervals n P t max t m ´ 1 , ´ M u , m, min t m ` 1 , M ´ 1 uu . Therefore, we have

We ignore the stationary pieces l where | J l | ď 2 M , since this cause a loss of at most O p M q .

For m P M 1 , although we have Ť min t m ` 1 ,M ´ 1 u n ' max t m ´ 1 , ´ M u ˜ T I p n q ' Ť min t m ` 1 ,M ´ 1 u n ' max t m ´ 1 , ´ M u q T I p n q (i.e., ř s P q T I p n q C s p a s q ď B {p 2 M q ´ 1 for all n P t max t m ´ 1 , ´ M u , m, min t m ` 1 , M ´ 1 uu ), it is not necessary that all requests assigned to intervals t max t m ´ 1 , ´ M u , m, min t m ` 1 , M ´ 1 uu are satisfied. The resource units reserved for these intervals can run out due to exploration, i.e., when

We define

Then we have

Inequality (49) follows from (17) in Lemma B.3, and inequality (50) follows from inequality (48). Hence, we have

Inequality (51) holds since ř M ´ 1 m '´ M p:q p m q ď | T R T | . Inequality (52) is valid since

Regarding m P M 2 . Suppose in some interval ˆ n P t max t m ´ 1 , ´ M u , . . . , min t m ` 1 , M ´ 1 uu , we have ř t P ˜ T I p ˆ n q C s p a s q ą B {p 2 M q´ 1 . Weaim validate the following two inequalities respectively:

Since given (54) and (55),for any interval m P M 2 ,

Validating (54). In our algorithm, for any t P Ť M ´ 1 m '´ M ˜ T I p m q , we have a t ' ˆ x p q t q˚ t , and hence, E r R t p a t q| F t ´ 1 s ' ř a P K r t p a q ˆ x p q t q˚ t p a q . Therefore, by Lemma B.2, for any δ P p 0 , 1 q , with probability at least 1 ´ 3 δ , we have

Inequality (57) holds since B {p 2 M q ´ 1 ă ř t P ˜ T I p ˆ n q C s p a s q ď B {p 2 M q . Then we have

Inequality (58) holds since for all t P ˜ T I p ˆ n q , we have ř a P K ˆ r t p a q ˆ x p q t q˚ t p a q{ ř a P K ˆ c t p a q ˆ x p q t q˚ t p a q ě α ˆ n . Then by inequality (16) in Lemma B.3, we have ř a P K r t p a q ˆ x p q t q˚ t p a q{ ř a P K c t p a q ˆ x p q t q˚ t p a q ě α ˆ n ´ 1 . Inequality (59) follows from inequality (57). We further have

Inequality (60) holds since the total B resource units have not run out before the reserved B {p 2 M q resource units for interval ˆ n run out, i.e.,

Inequality (61) is valid since for t P ˜ T I p ˆ n q , we have ˆ m p l q˚ t ' ˆ n . Hence, we have

Validating (55). For l P L such that q ˚ l ' 0 , by (18) in Lemma B.3, we have

For l P L such that q ˚ l ą 0 , by (15) we have

In this case,

Putting together (62) and (64)we have

Finally, let us combine the two cases where m P M 1 and m P M 2 . If ř m P M 1 ř l P L ř t P J l 1 p m p l q˚ t ' m q ¨ ř a P K r p l q p a q x ˚ l p a q ď ˜ O p a L | K | NT q , then p 53 q ' 0 and we have

If ř m P M 1 ř l P L ř t P J l 1 p m p l q˚ t ' m q ¨ ř a P K r p l q p a q x ˚ l p a q ě ˜ Ω p a L | K | NT q , then p 53 q ' p 1 ´ o p 1 qq{p α p M ` 1 qq . Then

Therefore, we conclude that

with probability at least 1 ´ 2 | K |p ML ` T q δ .

C.4.2 Bounding H(2)

H(2) is to bound the stochastic reward achieved by randomized decision and the expected reward. In IRES-CM, for t P Ť M ´ 1 m '´ M ˜ T I p m q , we have a t ' ˆ x p q t q˚ t , and hence E r R t p a t q| F t ´ 1 s ' ř a P K r t p a q ˆ x p q t q˚ t p a q . Then by Lemma B.1, for any δ P p 0 , 1 q , with probability at least 1 ´ δ , we have

Since H(1) ¨ opt(FA) ě p 1 ´ o p 1 qq ¨ p opt(FA) ´ ˜ O p a L | K | NT qq{p 10 α 4 M q ě Ω p opt(FA) q ě ˜ Ω p a L | K | NT q . Then it is evident that H(2) ě 1 ´ o p 1 q with probability at least 1 ´ δ .

C.5 Proof of Lemma 2.3

C.5.1 Proof of part (a)

The horizon T is partitioned into L ' T { B pieces with equal length B . We consider L instances with two arms K ' t 1 u and a null, and instance n happen with probability p n . All instances have deterministic outcomes, and they share the same consumption model C t p 1 q ' 1 for all t P T . Their reward functions are:

Denote FA p n q as the FA for instance n P t 1 , . . . , L u . It is clear that opt p FA p n q q ' Bα ´ n . Recall X t p 1 q ' 1 p Pull 1 in round t q . By the Yao's principle Yao (1977), the competitive ratio of any online algorithm is at most

for any p n ě 0 with ř L n ' 1 p n ' 1 . The expectation E p n q is over the randomness in X t in instance n . The instances are crafted such that during piece j P t 1 , . . . , L u , it is impossible to distinguish among instances j, . . . , L , meaning that the quantity B p n q j ' E p n q r ř t P piece j X t p 1 qs for n P t j, . . . , L u are all identical, and equal to a common value B j . Thus, for instance n P t 1 , . . . , L u we have E p n q r ř T t ' 1 R p n q t p 1 q X t p 1 qs ď ř L j ' n B j α ´ j . Consequently,

By defining p 1 ' 1 L p 1 ´ 1 { α q` 1 { α ' ` 1 ´ 1 α ˘ p n for n ' 2 , . . . , L , we have for every j ' 1 , . . . , L ,

by the inventory constraint ř L j ' 1 B j ď B on instance 1. Since L can generally be larger than log α p η max { η min q , we have shown that the CR can be significantly larger than log α p η max { η min q when η min ' 0 .

C.5.2 Proof of part (b)

While it is possible to derive a worse bound without η max by setting it to its upper bound of 1, knowing the lower bound is essential for our algorithm's functionality. To show that it is necessary to know η min , we suppose the DM be provided with a looser lower range parameter ˜ η min ă η min ď r t p a q , c t p a q @ a, t , and show that it leads to sub-optimal CR.

A general case construction. We firstly construct a case with N ` 1 instances when η min ' β ´ N for some absolute constant β ą 1 . We consider N ` 1 instances with two arms K ' t 1 u and a null , and instance n happen with probability p n . All instances have deterministic outcomes, and they share the same reward model R t p 1 q ' 1 for all t . Their consumption functions are:

Denote FA p n q as the FA for instance n P t 0 , . . . , N u . It is clear that opt p FA p n q q ' Bβ n . Recall X t p 1 q ' 1 p Pull 1 in round t q . By the Yao's principle Yao (1977), the competitive ratio of an online algorithm is at most

leading to

for any p n ě 0 with ř N n ' 0 p n ' 1 . The expectation E p n q is over the randomness in X t in instance n . The instances are crafted such that during piece j P t 0 , . . . , N u , it is impossible to distinguish among instances j, . . . , N , meaning that the quantity B p n q j ' E p n q r ř t P piece j C p n q t X t p 1 qs for n P t j, . . . , N u are all identical, and equal to a common value B j . Thus, for instance n P t 0 , . . . , N u we have E p n q r ř T t ' 1 R p n q t p 1 q X t p 1 qs ď ř N j ' n B j β j . Consequently,

By defining p 0 ' 1 2 p N ` 1 qp 1 ´ 1 { β q` 1 { β ' p 1 ´ 1 { β q p n for n ' 1 , . . . , N , we have for every j ' 0 , . . . , N ,

by the inventory constraint ř N j ' 0 B j ď B on instance N . Therefore, when the DM is provided with information η min ' β ´ N for any N P N , a CR ' Θ p N q lower bound is derived based on N ` 1 instances constructed above.

Not knowing η min ' β ´ Λ but knowing ˜ η min ' β ´ κ ¨ Λ . Wesuppose the real underlying η min ' β ´ Λ for some constant Λ , η max ' 1 , but the DM only has weaker prior information that ˜ η min ' β ´ κ ¨ Λ ( κ ą 1 can be set arbitrarily large) and η max ' 1 . The pattern of p R t , C t q follows the above case, and therefore, different η min leads to different number of instances N . The DM only knows the number of instances is no larger than κ ¨ Λ . Then from the DM's point of view, the optimal CR she/he could derive is CR ' Θ p κ ¨ Λ q ; while from the perspective of a clairvoyant who knows the real η min ' β ´ Λ , the optimal CR should be Θ p Λ q .

We first show that given ˜ η min ' β ´ κ ¨ Λ , the DM will not benefit from tightening the value ranges by blindly guessing a value of η min . We suppose the DM blindly tightens the value range to r β ´p κ ¨ Λ ´ d q , 1 s for some d ě 1 , without knowing the real η min . Then he/she derives a CR lower bound with N ` 1 ' κ ¨ Λ ´ d ` 1 instances based on the above construction. Then the DM can expect to achieve a total reward of

However, since the DM does not know the real η min , it is possible that in fact η min ' β ´ κ ¨ Λ . If this is indeed the case, the optimal reward can be as large as

based on the above constructed N ' κ ¨ Λ instances. Hence, from the DM's perspective, she/he could achieve a sub-optimal CR of

if she/he blindly assume η min ' β ´p κ ¨ Λ ´ d q . This is significantly worse than the optimal CR ' Θ p κ ¨ Λ q (if in fact η min ' β ´ κ ¨ Λ ). Thus, the DM has no motivation to assume a lower bound larger than the provided ˜ η min .

Therefore, the DM must derive a CR on the full range r β κ ¨ Λ , 1 s , which involves N ` 1 ' κ ¨ Λ ` 1 instances as constructed above. Therefore the DM expects a reward of Θ p B ¨ β κ ¨ Λ {p κ ¨ Λ qq . However, since in fact there are only Λ ` 1 instances, the DM wastes all her/his resources reserved for instance Λ ` 2 , . . . , κ ¨ Λ ` 1 and she/he can only achieve a reward of O p B ¨ β Λ {p κ ¨ Λ qq . Compared with the actual optimal reward Ω p B ¨ β Λ q with Λ ` 1 instances, the DM achieves a sub-optimal CR of Ω p κ ¨ Λ q . Since κ can be arbitrarily large, the CR derived without correct knowledge of η min is significantly worse than the optimal CR ' Θ p Λ q .

leading to

C.6 Proof of Theorem 4.5

We prove Theorem 4.5 by considering 2 ν ` 1 instances, which share the same K ' t 1 u , B P Z ą 0 and T ' B p 2 ν ` 1 q (All instances have the null arm, as stipulated by our model definition). All instances have deterministic outcomes, and they share the same consumption model C t p 1 q ' 1 for all t P T . By contrast, they differ in the reward model. The horizon T is partitioned into 2 ν ` 1 pieces with equal length B . Their reward functions are:

where ϵ ' α ´ 3 ν . Denote FA p n q as the FA for instance n P t´ ν, . . . , ν u . It is clear that opt p FA p n q q ' Bα ´ ν ´ n . Recall X t p 1 q ' 1 p Pull 1 in round t q . By the Yao's principle Yao (1977), the competitive ratio of an online algorithm is at most

for any p n ě 0 with ř ν n '´ ν p n ' 1 . The expectation E p n q is over the randomness in X t in instance n . The instances are crafted such that during piece j P t´ ν, . . . , ν u , it is impossible to distinguish among instances ´ ν, . . . , ´ j , meaning that the quantity B p n q j ' E p n q r ř t P piece j X t p 1 qs for n P t´ ν, . . . , ´ j u are all identical, and equal to a common value B j . Thus, for instance n P t´ ν, . . . , ν u we have E p n q r ř T t ' 1 R p n q t p 1 q X t p 1 qs ď Bϵ ` ř ´ n j '´ ν B j α j ´ ν ď B ¨ α ´ 3 ν ` ř ´ n j '´ ν B j α j ´ ν . Consequently,

By defining p ´ ν ' 1 2 ν p 1 ´ 1 { α q` 1 { α ' ` 1 ´ 1 α ˘ p n for n ' ´ ν ` 1 , . . . , ν , we have for every j ' ´ ν, . . . , ν ,

by the inventory constraint ř ν j '´ ν B j ď B . Since ν ' log α p η max { η min q{ 3 , the Theorem is proved.

leading to

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly stated the contributions made in the paper and important assumptions and limitations in our abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the important assumption and limitation of our work, which is η min ą 0 , in Section 2.2 'Assumption, limitation and discussion'. When we establish theorems regarding performance guarantees of our algorithms, we also state clearly the preliminary requirements for them to hold. We promise that we are honest on the limitations of our algorithm.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We promise that our paper provides the full set of assumptions and a complete (and correct) proof. Due to the page limit, our proofs are mostly in appendix. In the main paper, we provide high-level ideas and sketch proofs of our claims, while pointing to the locations of formal proofs in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We firstly make it clear that our paper is a theoretical paper. We only run simple experiments for sanity check and drawing insights. We are affirmative that our paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our data are numerically generated and codes can be provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Our experiment does not involve training data, but we have specified the test details and settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Though the experiment is not as important as theory in our paper, we run the experiment repeatedly and plot variations of all tests.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our experiment is very simple and can be run within a few minutes on any laptop.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical paper. There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does no involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies Bandits with Knapsack (BwK) under a piecewise stationary environment. For the online matching problem where the true reward is fully known in each time period, the paper obtains a $\\Omega(1/\\ln(\\eta_{\\max}/\\eta_{\\min}))$ where $\\eta_{\\min}$ and $\\eta_{\\max}$ are such that all rewards and resources are in $[\\eta_{\\min}, \\eta_{\\min}]$. The paper also studies the online learning problem and gives theoretical results.\n\nSTRENGTHS:\n1. The writing is good, and mostly easy-to-follow.\n2. The work gives theoretical guarantees to both the online matching and the online learning problem.\n\nWEAKNESSES:\nI appreciate all the technical proofs in the paper. However, it seems the paper lacks enough technical novelty as well as strong theoretical results. For details, please see Questions.\n\nQUESTIONS:\n1. The ratio of $\\ln(\\eta_{\\max}/\\eta_{\\min})$ seems to be weak. Imagine if $r_t=r$ and $c_t=c$ for all $t$, but $r$ and $c$ are dramatically different. It is reasonable to claim that the ratio approaches $1$ when the budget is not too low. Is it possible to refine the definition of $\\eta$ so that it only concerns $r/c$ (similar to that in Zhou et al. (2008))?\n2. In your contribution, you claim your performance guarantee is w.r.t. a dynamic benchmark, while existing adversarial BwK literature focus on the stationary benchmark. I think in many online matching papers the benchmark is also a dynamic one (see. e.g., Zhou et al. (2008)). Can you elaborate on this?\n3. In the online matching part, the algorithm design is not entirely novel. From my perspective, the inventory reserving idea is a deterministic version of the initial randomized policy in Immorlica et al. (2019). It is also similar to the booking limit or nested booking limit in the online matching/booking problem (see, e.g, [1]). Another issue with regard to the inventory reserving design is that it is not adaptive enough and may waste some resources because the reservation policy is fixed at the beginning. Can you elaborate on the technical novelty in your policy design?\n4. In the online learning problem, it seems you are essentially decreasing the competitive ratio in exchange for $\\sqrt{T}$ regret. Is this necessary? And also how do you set $\\alpha$?\n5. There are very limited numerical experiments for the planning problem and there is no experiment for the learning problem. I think only comparing to Immorlica et al. (2019) is not convincing. The authors may also consider comparing their algorithms with that in Zhou et al. (2008),\n\n[1] Ball, Michael O., and Maurice Queyranne. \"Toward robust revenue management: Competitive analysis of online booking.\" Operations Research 57.4 (2009): 950-963.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper studies the bandits with knapsacks problem in a piecewise-stationary environment. The paper proposes an algorithm guaranteering a near-optimal competitive ratio for the problem. The guarantees hold wrt a dynamic benchmark which is stronger than the standard stationary benchmark employed in adversarial BwK settings.

STRENGTHS:
The setup is interesting and well motivated, and it fits nicely within the line of works trying to bridge between fully stochastic and fully adversarial environments. Moreover, the proposed algorithm provides some interesting insights on the problem and a new perspective with respect to the standard LagrangeBwK approach of Immorlica et al.

WEAKNESSES:
See questions.

QUESTIONS:
Is the overhead of solving the LP at each iteration significant in your experiments? For example, how does it compare to "lighter" per-iteration updates, such as the primal-dual approach by Immorlica et al.?

LIMITATIONS:
n.a.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
This paper studies the bandits with knapsacks problem in a piecewise-stationary environment and designs an algorithm that achieves a provably near-optimal competitive ratio. Instead of using a static benchmark, the performance guarantee is present based on a dynamic benchmark.

STRENGTHS:
- The problem setup, piecewise-stationary BwK, complements the BwK literature.
- This paper is well-structured. The warm-up section 3 on deterministic outcome setting really helps in understanding the algorithm for stochastic outcome setting.
- For each algorithm/theoretical result, there are detailed explanations trying to give the intuition behind it.
- The theoretical results are built on mild assumptions. They are solid and technical.

WEAKNESSES:
- It is hard for readers not having expertise in BwK to get the intuition of why the algorithm is near-optimal. Specifically, why IRES is good by reserving an equal budget for each ratio?

QUESTIONS:
- In line 5 of algorithm 1, why using $\eta_{\min}\cdot\alpha^q$ with restricting $q$ in ${\0,1,...,M-1\}$ to guess $B^*_l/(t_l-t_{l-1})$?
- In Theorem 4.2, is it possible that $opt(FA)$ is close to the term $\tilde{O}(L\sqrt{|\mathcal{K}NT|})$? In this case, would this bound be close to zero regardless of the preceding product term?

LIMITATIONS:
The authors have discussed the limitations as they claimed in the paper checklist.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of piecewise non-stationary stochastic bandits with knapsacks. In bandit with knapsacks, at each round a learner is asked to choose an action and receives both a reward and a budget cost. The goal of the learner is to maximize its cumulative reward while satisfying some cumulative budget constraints. The authors prove a competitive ratio with respect to a non-stationary oracle with $L$ changes of order $O(1/\\log(\\eta_{\\max}/\\eta_{\\min}))$ under the assumption that budget costs smaller than $\\eta_\\max$ and rewards larger than $\\eta_\\min$ and if L is sufficiently small. Earlier results on non-stationary bandit with knapsacks are recent and depend on a global variation measure (glo) that is hardly satisfied in practice.\n\nSTRENGTHS:\n- The considered setting is relevant and of interest for practical applications. \n- The authors design an algorithm that achieves an optimal competitive ratio with a matching lower bound. \n- The non-stationary assumption is much more realistic than previous work. \n- Experiments included in the appendix demonstrate that the performance improvements with respect to the only existing baseline. The latter is however designed for the adversarial setting and is thus too conservative in their setting.\n\nWEAKNESSES:\n- The main part of the paper is too technical with many different notations, inline mathematical formulas, and not easy too follow. \n- The algorithm needs to know the loss and budget bounds in advances.\n- The dependence of the regret in $L$ seems suboptimal since the results only hold for $L \\leq o(\\sqrt{T\\eta_{\\min}})$ while we expect $L$ to be possibly as large as $o(T)$. \n- I am not convinced by the lower bound of Lemma 2.3 which uses $L = T/B$ and is thus in a setting where the upper-bound does not hold. The result would be stronger for a fixed value of $L$.\n\nQUESTIONS:\n- I suggest the authors to address my limitations.\n- The paper only considers piecewise stationarity, would it be possible to generalize it to smooth variations of losses and budget costs?\n\nTypos: \n- p3, l117: $B^2/T$\n- p5, l160: \"The DM does not *know*\"\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies Bandits with Knapsacks in a piecewise-stationary environment, where the underlying reward can change over time. \nThe authors provide provably near-optimal competitive ratio for this setting, which achieves a dynamic benchmark and obtains stronger results compared to existing adversarial Bwk works. Specifically, the algorithm proposed in the paper does not rely on the prior information of the number of stationary pieces and the time indexes when changes happen.\n\nSTRENGTHS:\n1. The contribution of this work is solid. Compared to previous works that use the global variation glo to quantify performance, the algorithm in this paper uses the number of reward changes as a measure. This is a better choice in non-stationary bandits in general.\n2. The paper is well written and provides adequate details to understand the flow of the material.\n\nWEAKNESSES:\nAs I understand, the main idea of this paper is to categorize the rewards into $M$ levels based on the maximum per unit reward (Ratio*). Each level's maximum per unit reward is a multiple of the previous one, allowing to focus solely on the highest level's maximum per unit reward. In this setting, the algorithm allocates $\\frac{B}{2M}$ resources to each level. In this regard, the algorithm's reward is at least the maximum per unit reward multiplied by $ \\frac{B}{2M} $, while the optimal algorithm's profit is at most the maximum unit profit multiplied by $ B $. This leads to a competitive ratio $O(M) = O(\\log(\\eta_{max}/\\eta_{min} ))$. \nIn this regard, such a design seems somewhat trivial. I can understand that for this problem, perhaps this method is optimal. However, given such an algorithm, I am not sure if the problem itself is sufficiently meaningful.\n\n\nTypos: \n1. Line 160: The DM does not L...\n\nQUESTIONS:\nFor the non-stationary bandit research I known before, algorithms typically need to detect whether the reward has changed every round. When a change is detected, the algorithm usually employs a restart mechanism to forget the past information. However, in Algorithm 2 of this paper, there is no such detection-restart mechanism. In this regard, I am curious about how the non-stationary rewards impact the algorithm in this paper.\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}