SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {r} _ {\delta , t} ^ {\mathrm {u c b}} (\boldsymbol {x}, a) \stackrel {\text {d e f}} {=} \operatorname {c l i p} \left[ \widehat {r} _ {t} (\boldsymbol {x}, a) + \varepsilon_ {t} (\boldsymbol {x}, a, \delta) \right] _ {0} ^ {1} \quad \text {a n d} \quad \widehat {\boldsymbol {c}} _ {\delta , t} ^ {\mathrm {l c b}} (\boldsymbol {x}, a) \stackrel {\text {d e f}} {=} \operatorname {c l i p} \left[ \widehat {\boldsymbol {c}} _ {t} (\boldsymbol {x}, a) - \varepsilon_ {t} (\boldsymbol {x}, a, \delta) \boldsymbol {1} \right] _ {- 1} ^ {1}, \tag {2}
$$
where we define clipping by  $\mathrm{clip}[x]_{\ell}^{u} = \min \left\{\max \{x,\ell \} ,u\right\}$  for a scalar value  $x\in \mathbb{R}$  and lower and upper bounds  $\ell$  and  $u$ , and apply clipping component-wise to vectors. Under Assumption 2, we have that for all  $\delta \in (0,1]$ , with probability at least  $1 - \delta$ ,
$$
r (\boldsymbol {x}, a) \leqslant \widehat {r} _ {\delta , t} ^ {\mathrm {u c b}} (\boldsymbol {x}, a) \leqslant r (\boldsymbol {x}, a) + 2 \varepsilon_ {t} (\boldsymbol {x}, a, \delta)
$$
$$
\text {a n d} \quad \boldsymbol {c} (\boldsymbol {x}, a) - 2 \varepsilon_ {t} (\boldsymbol {x}, a, \delta) \mathbf {1} \leqslant \widehat {\boldsymbol {c}} _ {\delta , t} ^ {\mathrm {l c b}} (\boldsymbol {x}) \leqslant \boldsymbol {c} (\boldsymbol {x}, a). \tag {3}
$$
Doing so, we have optimistic estimates of rewards (they are larger than the actual expected rewards) and of costs (they are smaller than the actual expected costs) for all actions  $a$ , while for actions  $a_{t+1}$  played, and only these, we will also use the control from the other side, which will lead to manageable sums of  $\varepsilon_t(\pmb{x}_{t+1}, a_{t+1}, \delta)$ , given the control on  $\beta_{T,\delta}$  by Assumption 2.
Modeling 1: Linear modeling. Agrawal and Devanur [2016] (see also Li and Stoltz, 2022, Appendix E) consider the case of a linear modeling where  $\eta_r = \eta_1 = \ldots = \eta_d \equiv 1$  and  $\Phi$  is the identity (together with assumptions on the range of  $\varphi$  and on the norms of  $\mu_{\star}$  and the  $\theta_{\star,i}$ ). The LinUCB strategy by Abbasi-Yadkori et al. [2011] may be slightly adapted (to take care of the existence of a transfer function  $\varphi$  depending on the actions taken) to show that Assumption 2 holds; see Li and Stoltz [2022, Appendix E, Lemma 3].
Modeling 2: Conversion model based on logistic bandits. Li and Stoltz [2022, Sections 1-5] discuss the case where  $\Phi(x) = 1/(1 + \mathrm{e}^{-x})$  for rewards and costs. The Logistic-UCB1 algorithm of Faury et al. [2020] (see also an earlier version by Filippi et al., 2010) may be slightly adapted (again, because of  $\varphi$ ) to show that Assumption 2 holds; see Li and Stoltz [2022, Lemma 1 in Appendix B, as well as Appendix C].
3 New, direct, dual strategy for CBwK
Our strategy heavily relies on the following equalities between the primal and the dual values of the problem considered. The underlying justifications are typically omitted in the literature (see, e.g., Slivkins et al., 2022, end of Section 2); we detail them carefully in Appendix A. We introduce dual variables  $\pmb{\lambda} \in [0, +\infty)^{d}$ , denote by  $\langle \cdot, \cdot \rangle$  the inner product in  $\mathbb{R}^d$ , and we have, provided that the problem  $(r, c, B', \nu)$  is feasible for some  $B' < B$ :
$$
\begin{array}{l} \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) = \sup  _ {\boldsymbol {\pi}: \mathcal {X} \rightarrow \mathcal {P} (\mathcal {A})} \inf  _ {\boldsymbol {\lambda} \geqslant \boldsymbol {0}} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} r (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) + \left\langle \boldsymbol {\lambda}, \boldsymbol {B} - \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right\rangle\right] \\ = \min  _ {\boldsymbol {\lambda} \geqslant \mathbf {0}} \sup  _ {\boldsymbol {\pi}: \mathcal {X} \rightarrow \mathcal {P} (\mathcal {A})} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \pi_ {a} (\boldsymbol {X}) \left(r (\boldsymbol {X}, a) - \left\langle \boldsymbol {c} (\boldsymbol {X}, a) - \boldsymbol {B}, \boldsymbol {\lambda} \right\rangle\right)\right] \\ = \min  _ {\boldsymbol {\lambda} \geqslant \mathbf {0}} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \max  _ {a \in \mathcal {A}} \left\{r (\boldsymbol {X}, a) - \left\langle \boldsymbol {c} (\boldsymbol {X}, a) - \boldsymbol {B}, \boldsymbol {\lambda} \right\rangle \right\} \right]. \tag {4} \\ \end{array}
$$
We denote by  $\lambda_B^\star$  a vector  $\lambda \geqslant 0$  achieving the minimum in the final equality. Virtually all previous contributions to (contextual) bandits with knapsacks learned the dual optimal  $\lambda^\star$  in some way; see the discussion in Main contribution #3 in Section 1. Our strategy is stated in Box B. A high-level description is that it replaces (Step 2) the ideal dual problem above by an empirical version, with estimates of  $r$  and  $C$ , with the expectation over  $X \sim \nu$  replaced by a point evaluation at  $x_t$ , and that it sequentially chooses some  $\lambda_t$  based on projected-gradient-descent steps (Step 4 uses the subgradient of the function in  $\lambda$  defined in Step 2). The gradient-descent steps are performed based on fixed step sizes  $\gamma$  in Section 3.1, and Section 3.2 will explain how to sequentially set the step sizes based on data. We also take some margin  $b1$  on  $B$ .
BOX B: PROJECTED GRADIENT DESCENT FOR CBWK WITH FIXED STEP SIZE
Inputs: step size  $\gamma > 0$ ; number of rounds  $T$ ; confidence level  $1 - \delta$ ; margin  $b$  on the average constraints; estimation procedure and error functions  $\varepsilon_{t}$  of Assumption 2; optimistic estimates (2)
Initialization:  $\lambda_0 = 0$ ; initial estimates  $\widehat{r}_{\delta,0}^{\mathrm{ucb}}$  and  $\widehat{c}_{\delta,0}^{\mathrm{bc}}$
1. Observe the context  $\boldsymbol{x}_t$
2. Pick an action  $a_{t} \in \mathcal{A}$  achieving
$$
\max  _ {a \in \mathcal {A}} \left\{\widehat {r} _ {\delta , t - 1} ^ {\mathrm {u c b}} \left(\boldsymbol {x} _ {t}, a\right) - \left\langle \widehat {\boldsymbol {c}} _ {\delta , t - 1} ^ {\mathrm {l c b}} \left(\boldsymbol {x} _ {t}, a\right) - (\boldsymbol {B} - b \boldsymbol {1}), \boldsymbol {\lambda} _ {t - 1} \right\rangle \right\};
$$
3. Observe the payoff  $r_t$  and the costs  $c_t$ ;
4. Compute  $\pmb{\lambda}_{t} = \left(\pmb{\lambda}_{t - 1} + \gamma \left(\widehat{\pmb{c}}_{\delta ,t - 1}^{\mathrm{lcb}}(\pmb{x}_{t},a_{t}) - (\pmb {B} - b\mathbf{1})\right)\right)_{+}$
5. Compute the estimates  $\widehat{r}_{\delta ,t}^{\mathrm{ucb}}$  and  $\widehat{c}_{\delta ,t}^{\mathrm{lcb}}$
3.1 Analysis for a fixed step size  $\gamma$
The strategy of Box B is analyzed through two lemmas. We let  $\min v = \min \{v_1, \dots, v_d\} > 0$  denote the smallest component of a vector  $v \in (0, +\infty)^d$  and take a margin  $b$  in  $(0, \min B)$ . We assume that the problem is feasible for some  $B' < B - b1$ , and denote by  $\lambda_{B - b1}^{\star}$  the vector achieving the minimum in (4) when the per-round cost constraint is  $B - b1$ . We use  $\|\cdot\|$  for the Euclidean norm. The bounds of Lemmas 1 and 2 critically rely on  $\|\lambda_{B - b1}^{\star}\|$ , which we bound and interpret below in Section 3.3. We denote
$$
\Upsilon_ {T, \delta} = \max \left\{\beta_ {T, \delta / 4}, 2 \sqrt {d T \ln \frac {T ^ {2}}{\delta / 4}}, \sqrt {2 T \ln \frac {2 (d + 1) T}{\delta / 4}} \right\}.
$$
Lemma 1. Fix  $\delta \in (0,1)$  and  $0 < b < \min B$ . If Assumption 2 holds and if the CBwK problem is feasible for some  $B' < B - b1$ , the Box B strategy run with  $\delta / 4$  ensures that the costs satisfy, with probability at least  $1 - \delta$ ,
$$
\forall 1 \leqslant t \leqslant T, \quad \left\| \left(\sum_ {\tau = 1} ^ {t} \boldsymbol {c} _ {\tau} - t (\boldsymbol {B} - b \mathbf {1})\right) _ {+} \right\| \leqslant \frac {1 + 3 \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \|}{\gamma} + 2 0 \sqrt {d} \Upsilon_ {T, \delta}.
$$
Lemma 2. Fix  $\delta \in (0,1)$  and  $0 < b < \min B$ . Under the same assumptions as in Lemma 1 and on the same event with probability at least  $1 - \delta$  as therein, the regret of the Box B strategy run with  $\delta /4$  satisfies
$$
\forall 1 \leqslant t \leqslant T, \quad R _ {t} \leqslant \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| (t b \sqrt {d} + 6 \Upsilon_ {T, \delta}) + 3 6 \gamma \sqrt {d} (\Upsilon_ {T, \delta}) ^ {2} + 8 \ln \frac {T ^ {2}}{\delta / 4}.
$$
Ideal choice of  $\gamma$ . The margin  $b$  will be taken proportional to the high-probability deviation provided by Lemma 1, divided by  $T$ . Now, the bounds of Lemmas 1 and 2 are respectively of the order
$$
T b \sim \left(1 + \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \|\right) / \gamma + \sqrt {T} \quad \text {a n d} \quad \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| (T b + \sqrt {T}) + \gamma T,
$$
again, up to logarithmic factors. To optimize each of these bounds, we therefore wish that  $b$  be of order  $1 / \sqrt{T}$  and that  $\gamma$  be of the order of  $\| \lambda_{B - b1}^{\star} \| / \sqrt{T}$ . Of course, this wish stumbles across the same issues of ignoring a beforehand bound on  $\| \lambda_{B - b1}^{\star} \|$ , exactly like, e.g., in Agrawal and Devanur [2016], Han et al. [2022] (where this bound is denoted by  $Z$ ). We were however able to overcome this issue in a satisfactory way, i.e., by obtaining the ideal bounds implied above through adaptive choices of  $\gamma$ , discussed in Section 3.2.
Suboptimal choices of  $\gamma$ . When  $\gamma$  is badly set, e.g.,  $\gamma = 1 / \sqrt{T}$ , the order of magnitude of the regret remains larger than  $\| \lambda_{B - b1}^{\star} \| Tb$  and the cost deviations to  $T(B - b1)$  become of the larger order  $\| \lambda_{B - b1}^{\star} \| \sqrt{T}$ , which dictates a larger choice for  $Tb$ . Namely, if in addition  $\| \lambda_{B - b1}^{\star} \|$  is bounded in some worst-case way by  $1 / \min B$  (see Section 3.3), the margin  $Tb$  is set of the order of  $\sqrt{T} / \min B$ . But of course, this margin must remain smaller than the total-cost constraints  $TB$ . This is the fundamental source of the typical  $\min B \geqslant T^{-1/4}$  constraints on the average cost constraints  $B$  observed in the literature: see, e.g., Agrawal and Devanur, 2016, Han et al., 2022, where the hyperparameter  $Z$  plays exactly the role of  $\| \lambda_{B - b1}^{\star} \|$ .
Proof sketch. Lemmas 1 and 2 are proved in detail in Appendix B; the two proofs are similar and we sketch here the one for Lemma 1. The derivations below hold on the intersection of four events of probability at least  $1 - \delta / 4$  each: one for Assumption 2, one for the application of the Hoeffding-Azuma inequality, and two for the application of a version of Bernstein's inequality. In this sketch of proof,  $\lesssim$  denotes inequalities holding up to small terms.
By the Hoeffding-Azuma inequality and by Assumption 2, the sum of the  $c_{\tau} - (B - b1)$  is not much larger than the sum of the  $\Delta \hat{c}_{\tau} = \hat{c}_{\delta /4,\tau -1}^{\mathrm{lc}}(x_{\tau},a_{\tau}) - (B - b1)$ . We bound the latter by noting that by definition,  $\lambda_{\tau}\geqslant \lambda_{\tau -1} + \gamma \Delta \hat{c}_{\tau}$ , so that after telescoping,
$$
\left\| \left(\sum_ {\tau = 1} ^ {t} \boldsymbol {c} _ {\tau} - t (\boldsymbol {B} - b \mathbf {1})\right) _ {+} \right\| \lesssim \left\| \left(\sum_ {\tau = 1} ^ {t} \Delta \widehat {\boldsymbol {c}} _ {\tau}\right) _ {+} \right\| \leqslant \frac {\left\| \boldsymbol {\lambda} _ {t} \right\|}{\gamma} \leqslant \frac {\left\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \right\|}{\gamma} + \frac {\left\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} - \boldsymbol {\lambda} _ {t} \right\|}{\gamma}. \tag {5}
$$
We are thus left with bounding the  $\| \pmb{\lambda}_{B - b1}^{\star} - \pmb{\lambda}_t\|$ . The classical bound in (projected) gradient-descent analyses (see, e.g., Zinkevich, 2003) reads:
$$
\left\| \boldsymbol {\lambda} _ {\tau} - \boldsymbol {\lambda} _ {B - b \mathbf {1}} ^ {\star} \right\| ^ {2} - \left\| \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} _ {B - b \mathbf {1}} ^ {\star} \right\| ^ {2} \leqslant 2 \gamma \left\langle \Delta \widehat {c c c c} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} _ {B - b \mathbf {1}} ^ {\star} \right\rangle + \gamma^ {2} \overbrace {\left\| \Delta \widehat {c c c c} _ {\tau} \right\| ^ {2}} ^ {\leqslant 4 d}.
$$
The estimates of Assumption 2, and the definition of  $a_{t}$  as the argument of some maximum, entail
$$
\left\langle \Delta \widehat {\boldsymbol {c}} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} _ {\boldsymbol {B} - \boldsymbol {b 1}} ^ {\star} \right\rangle = \left\langle \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\text {l c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - (\boldsymbol {B} - \boldsymbol {b 1}), \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} _ {\boldsymbol {B} - \boldsymbol {b 1}} ^ {\star} \right\rangle \lesssim g _ {\tau} \left(\boldsymbol {\lambda} _ {\boldsymbol {B} - \boldsymbol {b 1}} ^ {\star}\right) - g _ {\tau} \left(\boldsymbol {\lambda} _ {\tau - 1}\right),
$$
$$
\text {w h e r e} \quad g _ {\tau} (\boldsymbol {\lambda}) = \max  _ {a \in \mathcal {A}} \left\{r (\boldsymbol {x} _ {\tau}, a) - \left\langle \boldsymbol {c} (\boldsymbol {x} _ {\tau}, a) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} \right\rangle \right\}.
$$
We also introduce  $\Lambda_{t} = \sqrt{d}\max_{1\leqslant \tau \leqslant t}\| \pmb{\lambda}_{B - b1}^{\star} - \pmb{\lambda}_{\tau -1}\|$ . We sum up the bounds above, use telescoping, and get
$$
\left\| \boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} _ {\boldsymbol {B} - b 1} ^ {\star} \right\| ^ {2} \lesssim \left\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b 1} ^ {\star} \right\| ^ {2} + 4 d \gamma^ {2} t + 2 \gamma \underbrace {\sum_ {\tau \leqslant t} g _ {\tau} \left(\boldsymbol {\lambda} _ {\boldsymbol {B} - b 1} ^ {\star}\right) - g _ {\tau} \left(\boldsymbol {\lambda} _ {\tau - 1}\right)} _ {\lesssim 0 + 3 \Lambda_ {t} \sqrt {t \ln (T / \delta)}}, \tag {6}
$$
where the  $\lesssim$  of the sum in the right-hand side comes from a version of Bernstein's inequality, the fact that  $\mathbb{E}\big[g_{\tau}(\pmb{\lambda}_{B - b1}^{\star})\big] = \mathrm{OPT}(r,c,\pmb{B} - b1)$  is larger than the conditional expectation of  $g_{\tau}(\pmb{\lambda}_{\tau - 1})$ .
Since (6) holds for all  $t$ , we may then show by induction that for some constants  $\square$  and  $\triangle$ ,
$$
\left\| \boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} _ {\boldsymbol {B} - b 1} ^ {\star} \right\| \lesssim \square \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b 1} ^ {\star} \| + \triangle \gamma \Upsilon_ {T},
$$
and we conclude by substituting this inequality into (5).
3.2 Meta-strategy with adaptive step size  $\gamma$
As indicated after the statements of Lemmas 1 and 2, we want a (meta-)strategy adaptively setting  $\gamma$  to learn  $\|\lambda_{B - b1}^{\star}\|$ . We do so via a doubling trick using the Box B strategy as a routine and step sizes of the form  $\gamma_k = 2^k / \sqrt{T}$  in regimes  $k \geqslant 0$ ; the resulting meta-strategy is stated in Box C (a more detailed pseudo-code is provided in Appendix C). As the theorem below will reveal, the margin  $b$  therein can be picked based solely on  $T$ ,  $\delta$ , and  $d$ .
BOX C: PROJECTED GRADIENT DESCENT FOR CBWK WITH ADAPTIVE STEP SIZE
Inputs: number of rounds  $T$ ; confidence level  $1 - \delta$ ; margin  $b$  on the average constraints; estimation procedure and error functions  $\varepsilon_{t}$  of Assumption 2; optimistic estimates (2)
Initialization:  $T_0 = 1$ ; sequence  $\gamma_k = 2^k / \sqrt{T}$  of step sizes;
sequence  $M_{T,\delta ,k} = 4\sqrt{T} +20\sqrt{d}\Upsilon_{T,\delta /(k + 2)^2}$  of cost deviations
For regimes  $k = 0,1,2,\ldots$
- Take a fresh start of the Box B strategy with step size  $\gamma_{k}$ , risk level  $\delta / (4(k + 2)^2)$  and margin  $b$ ;
- Run it for  $t \geqslant T_k$  until  $\left\| \left( \sum_{\tau = T_k}^t \boldsymbol{c}_{\tau} - (t - T_k + 1) (\boldsymbol{B} - b\mathbf{1}) \right)_+ \right\|$  >  $M_{T,\delta,k}$ ;
- Stop regime  $k$ , and move to regime  $k + 1$  for the next round, i.e.,  $T_{k + 1} = t + 1$ .
Bandit problems containing a null-cost action (which is a typical assumption in the CBwK literature) are feasible for all cost constraints  $B' \geqslant 0$ ; the assumption of  $(B - 2b_{T}1)$ -feasibility in Theorem 1 below is therefore a mild assumption. Apart from this, the following theorem only imposes that the average cost constraints are larger than  $1 / \sqrt{T}$ , up to poly-logarithmic terms. We let  $\widetilde{\mathcal{O}}$  denote orders of magnitude in  $T$  up to poly-logarithmic terms.
For  $x > 0$ , we denote by  $\mathrm{ilog}x = \lceil \log_2x\rceil$  the upper integer part of the base-2 logarithm of  $x$ .
Theorem 1. Fix  $\delta \in (0,1)$  and let Assumption 2 hold. Consider the Box  $C$  meta-strategy, run with confidence level  $1 - \delta$  and with margin  $b_{T} = (1 + \mathrm{i}\log T)(M_{T,\delta ,\mathrm{i}\log T} + 2\sqrt{d}) / T = \tilde{\mathcal{O}}\big(1 / \sqrt{T}\big)$ .
If the average cost constraints  $\mathbf{B}$  are such that  $\mathbf{B} \geqslant 2b_{T}\mathbf{1}$  and if the problem  $(r, c, B - 2b_{T}\mathbf{1}, \nu)$  is feasible, then this meta-strategy satisfies, with probability at least  $1 - \delta$ :
$$
R _ {T} \leqslant \widetilde {\mathcal {O}} \left(\left(1 + \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \boldsymbol {1}} ^ {*} \|\right) \sqrt {T}\right) \quad a n d \quad \sum_ {t \leqslant T} \boldsymbol {c} _ {t} \leqslant T \boldsymbol {B},
$$
where a fully closed-form, non-asymptotic, regret bound is stated in (19) in Appendix C.
A complete proof may be found in Appendix C. Its structure is the following; all statements hold with high probability. We denote by  $\ell_{k}$  the realized lengths of the regime. First, the number of regimes is bounded by noting that if regime  $K = \mathrm{i}\log \|\pmb{\lambda}_{B - b_T\mathbf{1}}^*\|$  is achieved, then by Lemma 1 and the choice of  $M_{T,\delta,K}$ , the stopping condition of regime  $K$  will never be met; thus, at most  $K + 1$  regimes take place. We also prove that  $\mathrm{i}\log \|\pmb{\lambda}_{B - b_T\mathbf{1}}^*\| \leqslant K_T = \mathrm{i}\log T$ . Second, on each regime  $k$ , the difference of cumulated costs to  $\ell_{k}(\pmb{B} - b_{T}\mathbf{1})$  is smaller than  $M_{T,\delta,k}$  by design, so that the total cumulative costs are smaller than  $T(\pmb{B} - b_{T}\mathbf{1})$  plus something of the order of  $(K_T + 1)M_{T,\delta,K_T}$ , which is a quantity that only depends on  $T$ ,  $\delta$ ,  $d$ , and not on the unknown  $\|\pmb{\lambda}_{B - b_T\mathbf{1}}^*\|$ , and that we take for  $Tb_{T}$ . Third, we similarly sum the regret bounds of Lemma 2 over the regimes: keeping in mind that  $b_{T} = \widetilde{\mathcal{O}}(\sqrt{T})$ , we have sums of the form
$$
\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \| \sum_ {k \leqslant K} \left(b _ {T} \ell_ {k} + \sqrt {T}\right) \leqslant \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \| \widetilde {\mathcal {O}} (\sqrt {T}) \quad \text {a n d} \quad \sum_ {k \leqslant K} \gamma_ {k} T \leqslant \frac {2 ^ {K + 1}}{\sqrt {T}} \leqslant \frac {4 \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \|}{\sqrt {T}}.
$$
3.3 Discussion of the obtained regret bounds
The regret bound of Theorem 1 features a multiplicative factor  $\| \lambda_{B - b_T\mathbf{1}}^{\star}\|$  instead of the typical factor  $\mathrm{OPT}(r,c,B) / \min B$  proposed by the literature (see, e.g., Agrawal and Devanur, 2016, Li and Stoltz, 2022, Han et al., 2022). In view of the lower bound proof scheme discussed in Section 4, this bound  $\| \lambda_{B - b_T\mathbf{1}}^{\star}\| \sqrt{T}$  seems the optimal bound. We thus now provide bounds on  $\| \lambda_{B - b_T\mathbf{1}}^{\star}\|$  that have some natural interpretation. We recall that feasibility was defined in Assumption 1. The elementary proofs of Lemma 3 and Corollary 1 are based on (4) and may be found in Appendix D.
Lemma 3. Let  $b \in [0, \min B)$  and let  $\mathbf{0} \leqslant \widetilde{\mathbf{B}} < \mathbf{B} - b\mathbf{1}$ . If the contextual bandit problem with knapsacks  $(r, c, B', \nu)$  is feasible for some  $B' < \widetilde{B}$ , then
$$
\left\| \boldsymbol {\lambda} _ {B - b \mathbf {1}} ^ {\star} \right\| \leqslant \frac {\operatorname {O P T} (r , \boldsymbol {c} , \boldsymbol {B} - b \mathbf {1}) - \operatorname {O P T} (r , \boldsymbol {c} , \widetilde {\boldsymbol {B}})}{\min  (\boldsymbol {B} - b \mathbf {1} - \widetilde {\boldsymbol {B}})}.
$$
Corollary 1. When there exists a null-cost action,  $\| \pmb{\lambda}_{B - b\mathbf{1}}^{\star}\| \leqslant 2\frac{\mathrm{OPT}(r,\pmb{c},\pmb{B}) - \mathrm{OPT}(r,\pmb{c},\pmb{0})}{\min\pmb{B}}$  for all  $b\in [0,\min B / 2]$
The bounds provided by Lemma 3 and Corollary 1 must be discussed in each specific case. They are null (as expected) when the average cost constraints  $\pmb{B}$  are large enough so that costs do not constrain the choice of actions; this was not the case with the typical  $\mathrm{OPT}(r, c, B) / \min B$  bounds of the literature. Another typical example is the following.
Example 1. Consider a problem featuring a baseline action  $a_{\mathrm{null}}$  with some positive expected rewards and no cost, and additional actions with larger rewards and scalar expected costs  $c(\pmb{x}, a) \geqslant \alpha > 0$ . Denote by  $B$  the average cost constraint. Then actions  $a \neq a_{\mathrm{null}}$  may only be played at most  $B / \alpha$  times on average, and therefore,  $\mathrm{OPT}(r, c, B) - \mathrm{OPT}(r, c, 0) \leqslant B / \alpha$ . In particular, the bound stated in Corollary 1 may be further upper bounded by  $1 / (2\alpha)$ , which is fully independent of  $T$  and  $B$ ; i.e., a  $\sqrt{T}$ -regret only is suffered in Theorem 1.
4 Optimality of the upper bounds, and limitations
The main limitations to our work are relative, in our eyes, to the optimality of the bounds achieved—up to one limitation, already discussed in Section 2.1: our fairness example is intrinsically limited to the awareness set-up, where the learner has access to the group index before making the predictions.
A proof scheme for problem-dependent lower bounds. Results offering lower bounds on the regret for CBwK are scarce in the literature. They typically refer to some minimax regret, and either do so by indicating that lower bounds for CBwK must be larger in a minimax sense than the ones obtained for non-contextual bandits with knapsacks (see, e.g., Agrawal and Devanur, 2016, Li and Stoltz, 2022), or by leveraging a minimax regret lower bound for a subroutine of the strategy used (Slivkins et al., 2022). We rather focus on problem-dependent regret bounds but only offer a proof scheme, to be made more formal—see Appendix E.1. Interestingly, this proof scheme follows closely the
analysis of a primal strategy performed in Appendix F. First, an argument based on the law of the iterated logarithm shows the necessity of a margin  $b_{T}$  of order  $1 / \sqrt{T}$  to satisfy the total  $TB$  cost constraints. This imposes that only  $T\mathrm{OPT}(r,c,B - b_T\mathbf{1})$  is targeted, thus the regret is at least of order
$$
T \left(\operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B} - b _ {T} \mathbf {1})\right) + \sqrt {T}, \quad \text {i . e . ,} \quad \left(1 + \| \lambda_ {\boldsymbol {B}} ^ {\star} \|\right) \sqrt {T}.
$$
This matches the bound of Theorem 1, but with  $\lambda_{\pmb{B} - b_T\mathbf{1}}^{\star}$  replaced by  $\lambda_B^\star$
We now turn to the list of limitations pertaining to the optimality of our bounds.