SlowGuess/NeurIPS_2023_OCR huggingface dataset:
can be larger than  $T b_{T}$ , which yields the desired control  $\sum_{t=1}^{T} c_{t} \leqslant T B$ .
C.3 Step 3: Computing the associated regret bound
The total regret is the sum of the regrets suffered over each regime:
$$
R _ {T} = \sum_ {k = 0} ^ {K} \left((T _ {k + 1} - T _ {k}) \mathrm {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \sum_ {t = T _ {k}} ^ {T _ {k + 1} - 1} r _ {t}\right).
$$
On the favorable event  $\mathcal{E}_{\mathrm{meta}}$ , the bound of Lemma 2 holds in particular at the end of each regime; i.e., given the parameters  $\gamma_{k} = 2^{k} / \sqrt{T}$  and  $\delta / (4(k + 2)^2)$  to run the Box B strategy in regime  $k \in \{0, \dots, K\}$ , it holds on  $\mathcal{E}_{\mathrm{meta}}$  that
$$
\begin{array}{l} \left(T _ {k + 1} - T _ {k}\right) \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \sum_ {t = T _ {k}} ^ {T _ {k + 1} - 1} r _ {t} \leqslant \left\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \boldsymbol {1}} ^ {\star} \right\| \left(\left(T _ {k + 1} - T _ {k}\right) b _ {T} \sqrt {d} + 6 \Upsilon_ {T, \delta / (k + 2) ^ {2}}\right) \\ + 3 6 \sqrt {d} \left(\Upsilon_ {T, \delta / (k + 2) ^ {2}}\right) ^ {2} \frac {2 ^ {k}}{\sqrt {T}} + 8 \ln \frac {T ^ {2}}{\delta / \left(4 (k + 2) ^ {2}\right)}. \\ \end{array}
$$
We now sum the above bounds and use the (in)equalities  $\Upsilon_{T,\delta /(k + 2)^2}\leqslant \Upsilon_{T,\delta /(K + 2)^2}$ ,
$$
\sum_ {k = 0} ^ {K} \left(T _ {k + 1} - T _ {k}\right) b _ {T} = T b _ {T}, \quad \text {a n d} \quad \sum_ {k = 0} ^ {K} 2 ^ {k} \leqslant 2 ^ {K + 1} \leqslant 2 ^ {1 + \mathrm {i l o g} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \|} \leqslant 4 \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \|
$$
$$
\begin{array}{l} R _ {T} \leqslant \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \| (T b _ {T} \sqrt {d} + 6 K \Upsilon_ {T, \delta / (K + 2) ^ {2}}) + 1 4 4 \sqrt {d} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \| \frac {\left(\Upsilon_ {T , \delta / (K + 2) ^ {2}}\right) ^ {2}}{\sqrt {T}} \\ + 8 K \ln \frac {T ^ {2}}{\delta / (4 (K + 2) ^ {2})}. \\ \end{array}
$$
The final regret bound is achieved by substituting the inequality  $K \leqslant \mathrm{i}\log T$  proved in Step 1:
$$
\begin{array}{l} R _ {T} \leqslant \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \| \left(1 4 4 \sqrt {d} \frac {\left(\Upsilon_ {T , \delta / (2 + \mathrm {i} \log T) ^ {2}}\right) ^ {2}}{\sqrt {T}} + T b _ {T} \sqrt {d} + 6 \Upsilon_ {T, \delta / (2 + \mathrm {i} \log T) ^ {2}} \mathrm {i} \log T\right) \\ + 8 \ln \frac {T ^ {2}}{\delta / \left(4 (2 + \mathrm {i} \log T) ^ {2}\right)} \mathrm {i} \log T. \tag {19} \\ \end{array}
$$
The order of magnitude is  $\sqrt{T}$ , up to poly-logarithmic terms, for the quantity  $\Upsilon_{T,\delta/(2+\mathrm{i}\log T)^2}$ , thus for  $M_{T,\delta,\mathrm{i}\log T}$ , thus for  $Tb_{T}$ , therefore, the order of magnitude in  $\sqrt{T}$  of the above bound is, up to poly-logarithmic terms,
$$
\left(1 + \left\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \right\|\right) \sqrt {T},
$$
D Proofs of Lemma 3 and Corollary 1
Proof of Lemma 3. For  $\lambda \geqslant 0$  and  $C\in [0,1]^d$ , we denote
$$
L (\boldsymbol {\lambda}, \boldsymbol {C}) = \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \max  _ {a \in \mathcal {A}} \left\{r (\boldsymbol {X}, a) - \langle \boldsymbol {c} (\boldsymbol {X}, a) - \boldsymbol {C}, \boldsymbol {\lambda} \rangle \right\} \right]
$$
so that by (4) and the feasibility assumption, we have, at least for  $C = \widetilde{B}$  and  $C = B - b1$ :
$$
\mathrm {O P T} (r, \boldsymbol {c}, \boldsymbol {C}) = \min  _ {\boldsymbol {\lambda} \geqslant \mathbf {0}} L (\boldsymbol {\lambda}, \boldsymbol {C}) = L (\boldsymbol {\lambda} _ {\boldsymbol {C}} ^ {\star}, \boldsymbol {C}).
$$
The function  $L$  is linear in  $\pmb{C}$ , so that
$$
\begin{array}{l} \operatorname {O P T} (r, \boldsymbol {c}, \widetilde {\boldsymbol {B}}) = L (\lambda_ {\widetilde {\boldsymbol {B}}} ^ {\star}, \widetilde {\boldsymbol {B}}) \leqslant L (\lambda_ {\boldsymbol {B} - b 1} ^ {\star}, \widetilde {\boldsymbol {B}}) = L (\lambda_ {\boldsymbol {B} - b 1} ^ {\star}, \boldsymbol {B} - b 1) - \left\langle \lambda_ {\boldsymbol {B} - b 1} ^ {\star}, \boldsymbol {B} - b 1 - \widetilde {\boldsymbol {B}} \right\rangle \\ = \operatorname {O P T} (r, c, B - b 1) - \left\langle \lambda_ {B - b 1} ^ {\star}, B - b 1 - \widetilde {B} \right\rangle . \\ \end{array}
$$
The result follows from substituting
$$
\left\langle \lambda_ {B - b 1} ^ {\star}, B - b 1 - \widetilde {B} \right\rangle \geqslant \left\| \lambda_ {B - b 1} ^ {\star} \right\| _ {1} \min  (B - b 1 - \widetilde {B}) \geqslant \left\| \lambda_ {B - b 1} ^ {\star} \right\| \min  (B - b 1 - \widetilde {B})
$$
and from rearranging the inequality thus obtained.
Proof of Corollary 1. We apply Lemma 3 with  $\widetilde{B} = \varepsilon \mathbf{1}$  for some  $\varepsilon >0$  sufficiently small and obtain
$$
\left\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \right\| \leqslant \frac {\operatorname {O P T} (r , \boldsymbol {c} , \boldsymbol {B} - b \mathbf {1}) - \operatorname {O P T} (r , \boldsymbol {c} , \varepsilon \mathbf {1})}{\min  (\boldsymbol {B} - (b + \varepsilon) \mathbf {1})}.
$$
We conclude by substituting  $\mathrm{OPT}(r, c, B - b1) \leqslant \mathrm{OPT}(r, c, B)$  and  $\mathrm{OPT}\big(r, c, \varepsilon \mathbf{1}\big) \geqslant \mathrm{OPT}\big(r, c, \mathbf{0}\big)$  as well as  $\min(B - b1) \geqslant \min B/2$ , and by letting  $\varepsilon \to 0$ .
E Additional (sketch of) results concerning optimality
We detail here two series of claims made in Section 4.
E.1 A proof scheme for problem-dependent lower bounds
We provide the proof scheme for proving the problem-dependent lower bound of order  $\left(1 + \|\lambda_B^\star\|\right)\sqrt{T}$  announced in Section 4.
Step 0: Considering strict cost constraints. Our aim, as described in Box A, is to make sure that with high probability the cumulative costs are smaller than  $T B$ . If we considered softer constraints, of the form  $T B + \widetilde{\mathcal{O}}\big(\sqrt{T}\big)$ , then  $\widetilde{\mathcal{O}}\big(\sqrt{T}\big)$  regret bounds would be possible (see Appendix F); i.e., the factor  $\| \lambda_{B - b_T1}^* \|$  of Theorem 1 could be replaced by a constant. Thus, lower bounds are only interesting in the case of hard constraints stated in Box A.
Step 1: Necessity of a margin  $b_{T}$  of order  $1 / \sqrt{T}$ . First, a classical lemma in CBwK (see, e.g., Agrawal and Devanur, 2016, Lemma 1) indicates that a sequence of adaptive cannot perform better than an optimal static policy. Denote by  $\pi_{B'}^*$  a (quasi-)optimal policy for the average cost constraints  $B'$ . Provided that costs are truly random (i.e., do not stem from Dirac distributions, which in particular, does not cover the cases where there is a null-cost action, see Limitation 2 in Section 4), then the law of iterated logarithm shows that when playing  $\pi_{B'}^*$  at each round, the cumulative costs must (almost-surely, as  $T \to +\infty$ ) be larger than  $TB'$  plus a positive term of the order of  $\sqrt{T \ln \ln T}$ . Therefore, to meet the hard constraints, one should pick  $B'$  of the form  $B - b_{T}1$ , where  $b_{T}$  is of order  $1 / \sqrt{T}$  up to logarithmic terms.
Step 2: Consequences in terms of regret. Therefore, the largest average reward a strategy may target is  $\mathrm{OPT}(r, c, B - b_T \mathbf{1})$ . Deviations of the order  $\sqrt{T}$  are also bound to happen. Therefore, up to logarithmic terms, the regret lower bound is approximatively larger than something of the order
$$
T \left(\operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B} - b _ {T} \mathbf {1})\right) + \sqrt {T}.
$$
Now, an argument similar to the one used in the proof of Lemma 3 shows that
$$
\operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B} - b _ {T} \mathbf {1}) \geqslant L \left(\lambda_ {\boldsymbol {B}} ^ {\star}, \boldsymbol {B}\right) - L \left(\lambda_ {\boldsymbol {B}} ^ {\star}, \boldsymbol {B} - b _ {T} \mathbf {1}\right) = b _ {T} \| \lambda_ {\boldsymbol {B}} ^ {\star} \|.
$$
All in all, the regret lower bound is thus approximatively larger than something of the order of
$$
\left(1 + \left\| \lambda_ {B} ^ {\star} \right\|\right) \sqrt {T}.
$$
This matches the form of the bound of Theorem 1, but the dual vector  $\lambda_{B - b_T1}^{\star}$  present in the upper bound is replaced by  $\lambda_B^\star$  in our lower bound.
E.2 Faster rates may be achievable in some specific cases
We explain here why, in some specific cases, faster rates would be achievable—with  $\sqrt{T}$  in the regret bound of Theorem 1 replaced by  $\sqrt{TB}$  for a problem with scalar costs and total-cost constraints  $B$ .
Indeed, consider a problem similar to Example 1: with scalar costs, total-cost constraint  $B > 0$ , featuring a baseline action  $a_{\mathrm{null}}$  with null cost but also null reward, and additional actions with larger rewards and expected costs  $c(\pmb{x},a)\geqslant \alpha >0$ . Let  $N_{T}$  denotes the number of times non-null cost actions are played within the first  $T$  rounds. Deviation inequalities have it that
$$
\sum_ {t = 1} ^ {T} c _ {t} \geqslant \alpha N _ {T} - \tilde {O} \left(\sqrt {N _ {T}}\right),
$$
where we recall that  $\tilde{O}$  is up to poly-logarithmic factors; as a consequence, the total-cost constraint enforces that
$$
N _ {T} \leqslant \frac {T B}{\alpha} + \tilde {O} \left(\sqrt {\frac {T B}{\alpha}}\right).
$$
In particular, the margin  $b_{T}$  only needs to be of order  $\sqrt{N_T} / T$ , i.e.,  $\sqrt{B / (\alpha T)}$ , instead of  $1 / \sqrt{T}$ . Similarly, since at most  $N_{T}$  non-null actions are played, the regret should be lower bounded by something of the order of
$$
T \left(\operatorname {O P T} (r, c, B) - \operatorname {O P T} (r, c, B - b _ {T})\right) + \sqrt {N _ {T}},
$$
where, as in Step 2 above,
$$
\left(\mathrm {O P T} (r, c, B) - \mathrm {O P T} (r, c, B - b _ {T})\right) \geqslant b _ {T} | \lambda_ {B} ^ {\star} | = \tilde {O} \left(\sqrt {\frac {B}{\alpha T}}\right) | \lambda_ {B} ^ {\star} |.
$$
This suggests a lower bound on the regret of the order (up to poly-logarithmic factors) of
$$
\left(| \lambda_ {B} ^ {\star} | + 1\right) \sqrt {T B / \alpha} \quad \text {i n s t e a d o f} \quad \left(1 + | \lambda_ {B} ^ {\star} |\right) \sqrt {T}.
$$
The difference between the two bounds is significant when  $B$  is small, i.e.,  $B \ll 1$ . While proving a matching upper bound with the Box B and Box C strategies looks a bit tricky, we feel that it must be possible to do so with the primal approach of Appendix F, at least when the context space  $\mathcal{X}$  is finite. If our intuition holds, then it would be possible to get an upper bound
$$
\begin{array}{l} R _ {T} \leqslant \tilde {O} \left(T b _ {T} \left(1 + \left| \lambda_ {B - b _ {T}} ^ {*} \right|\right)\right) \\ = \tilde {O} \left(\sqrt {\frac {T B}{\alpha}} \left(1 + \frac {\operatorname {O P T} (r , c , B) - \operatorname {O P T} (r , c , 0)}{B}\right)\right) = \tilde {O} \left(\sqrt {T B / \alpha^ {3}}\right), \\ \end{array}
$$
where the last bound follows from  $\mathrm{OPT}(r,c,B) - \mathrm{OPT}(r,c,0)\leqslant B / \alpha$ , as explained in Example 1.
F Primal strategy
This section studies the primal strategy stated in Box D, which, at every round, solves an approximation of the primal optimization problem (1). The key issue in running such a primal approach is estimating  $\nu$ , see comments after Notation 1; this primal approach is essentially worth for the case of finite context sets  $\mathcal{X}$ . The aim of this section is threefold:
1. In Appendix F.1, we provide a theory of "soft" constraints, when total-cost deviations from  $TB$  of order  $\sqrt{T}$  up to logarithmic terms are allowed; at least when  $\mathcal{X}$  is a finite set, the regret bound then becomes proportional to  $\sqrt{T}$  up to logarithmic terms.
2. In Appendix F.2, we revisit and extend the results by Li and Stoltz [2022]. The extension consists of dealing with possibly signed constraints, and the revisited analysis (of the same strategy as in Li and Stoltz, 2022) consists in not directly dealing with KKT constraints (which, in addition, imposed the finiteness of  $\mathcal{X}$ ) but in only relating optimization problems—defined with the true  $r$  and  $c$  or estimates thereof. We also offer a modular approach and separate the error terms coming from estimating  $\nu$  and from estimating  $r$  and  $c$ .
3. In Appendix F.3, we generalize the results of Appendix F.2, which rely on the existence of a null-cost action, and get guarantees that correspond quite exactly to the combination of Theorem 1 and the interpretation thereof offered by Lemma 3, at least in the case of a finite  $\mathcal{X}$ . Actually, in our research path, we had first obtained these primal results, before trying to obtain them in a more general case of a continuous  $\mathcal{X}$ , by resorting to a dual strategy. The proof technique in Appendix F.3 also inspired our approach to proving problem-dependent lower bounds presented in Appendix E.1.
Throughout this appendix, we will assume that the context distribution  $\nu$  can be estimated in some way. We provide examples and pointers below.
Notation 1. Fix  $\delta \in (0,1)$ . We denote by  $\widehat{\nu}_{\delta,t}$  a sequence of estimators of  $\nu$ , each constructed on the contexts  $x_1,\ldots,x_t$ , and by  $\xi_{t,\delta}$  a sequence of estimation errors such that, with probability at least  $1 - \delta$ , for all bounded functions  $f:\mathcal{X}\to [-1,1]$ ,
$$
\left| \mathbb {E} _ {\boldsymbol {X} \sim \nu} [ f (\boldsymbol {X}) ] - \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta , t}} [ f (\boldsymbol {X}) ] \right| \leqslant \xi_ {t, \delta}.
$$
We also denote  $\Xi_{T,\delta} = \sum_{t = 1}^{T}\xi_{t - 1,\delta}$
We consider the strategy described in Box D, whether  $\mathcal{X}$  is finite (as in Li and Stoltz, 2022, where it was first considered) or not. Our simplified analyses do not rely on explicit KKT inequalities and therefore do not require anymore that  $\mathcal{X}$  is finite. In Box D, by "when the empirical cost constraints are feasible", we mean that there exists some policy  $\pi$  such that
$$
\mathbb {E} _ {\boldsymbol {X} \sim \hat {\nu} _ {\delta , t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {\boldsymbol {c}} _ {\delta , t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right] \leqslant \boldsymbol {B} + b _ {t} \mathbf {1}.
$$
We need to guarantee the existence of a policy achieving the constrained maximum of Step 2 in Box D. This is immediate when  $\mathcal{X}$  is finite, as the problem then corresponds to a finite-dimensional linear program. In general, we may note that when (4) holds, we read therein the optimal policy, as a pointwise maximum involving the optimal dual variables. The proofs reveal that up to slightly augmenting the  $\xi_{t-1,\delta}$  of Notation 1, we may assume that the strict feasibility sufficient for (4) indeed holds.
We also note that even if the argument of the maximum  $\pi_t$  is guaranteed to exist, it may be difficult to compute: the linear program of Box D cannot be solved exactly if the  $\widehat{\nu}_{\delta,t}$  are not finitely supported (which happens in general when  $\mathcal{X}$  is not finite). We do not see this as a severe issue as the Box D strategy is an ideal strategy anyway, that we study for the sake of shedding lights on our results for the dual approach in the main body of the article.
A final remark on the Box D strategy is that the margins on the average cost constraints  $B$  can now be signed, which is why they will be referred to as signed slacks.
BOX D: PRIMAL CBWK STRATEGY, GENERALIZED FROM LI AND STOLTZ [2022]
Inputs: confidence level  $1 - \delta$ ; estimation procedure and error functions  $\varepsilon_{t}$  of Assumption 2; optimistic estimates (2); estimation procedure for  $\nu$  as in Notation 1
Hyperparameter: signed slacks  $b_{1}, b_{2}, \ldots, b_{T} \in \mathbb{R}$
Initialization: initial estimates  $\widehat{r}_{\delta,0}^{\mathrm{ucb}}(\pmb{x},a)$  and  $\widehat{c}_{\delta,0}^{\mathrm{cb}}(\pmb{x},a)$ , as well as  $\widehat{\nu}_{\delta,0}$
For rounds  $t = 1,2,3,\ldots ,T$  ..
1. Compute a policy  $\pi_t$  achieving
$$
\max _ {\boldsymbol {\pi}: \mathcal {X} \to \mathcal {P} (\mathcal {A})} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta , t}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta , t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right]
$$
$$
\text {u n d e r} \quad \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu_ {\delta , t - 1}}} \left[ \sum_ {a \in \mathcal {A}} \widehat {\boldsymbol {c}} _ {\delta , t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right] \leqslant \boldsymbol {B} + b _ {t} \mathbf {1}
$$
when the empirical cost constraints are feasible, and pick an arbitrary policy  $\pi_t$  otherwise;
2. Observe  $\pmb{x}_t$  and draw an action  $a_t \sim \pi_t(\pmb{x}_t)$ ;
3. Compute the estimate  $\widehat{r}_{\delta ,t}^{\mathrm{ucb}}(\pmb {x},a)$  and  $\widehat{c}_{\delta ,t}^{\mathrm{bc}}(\pmb {x},a)$  as well as  $\widehat{\nu}_{\delta ,t}$
Discussion of the estimation of  $\nu$ . The simplest case is when  $\mathcal{X}$  is a finite set; in this case, we may take
$$
\xi_ {t, \delta} \qquad \text {o f o r d e r} \qquad \sqrt {\frac {| \mathcal {X} | \ln (1 / \delta)}{t}},
$$
where  $\mathcal{X}$  denotes the cardinality of  $\mathcal{X}$ ; see [Ai et al., 2022, Section 4.1], see also a less sharp bound based on Hoeffding's inequality in [Li and Stoltz, 2022, Section 5]. This leads to a total error term  $\Xi_{T,\delta}$  of order  $\sqrt{T}$  up to poly-logarithmic term.
When  $\mathcal{X}$  is a continuous subset of  $\mathbb{R}^n$ , some regularity conditions are put on  $\nu$ , which is typically assumed to have some smooth density with respect to the Lebesgue measure  $\mathfrak{m}$ . Estimates  $\widehat{\nu}_{\delta,t}$  are obtained by estimating the density  $\mathrm{d}\nu/\mathrm{d}\mathfrak{m}$ : the criterion in Notation 1, which is proportional to the total-variation distance between  $\nu$  and  $\widehat{\nu}_{\delta,t}$ , is then given by the  $\mathbb{L}^1(\mathfrak{m})$  distance between the two
densities. To control the latter with uniform convergence rates, so as to obtain deviation terms  $\xi_{t,\delta}$  only depending on  $t$  and  $\delta$ , heavy assumptions on the model to which  $\nu$  belongs are in order, e.g., some Hölderian regularity, and uniform estimation rates obtained degrade with the ambient dimension  $n$ ; they are generally much slower than  $1/\sqrt{t}$ . The total error terms  $\Xi_{T,\delta}$  then prevent the bounds stated below in Proposition 1 and Theorems 3 and 4 from sharing the same orders of magnitude than the bounds proved with our dual approach in Theorem 1 and interpreted in Section 3.3. On this topic, see also [Ai et al., 2022, Section 4.1] for the estimation rates as well as a similar description in Han et al. [2022, end of Section 1] of the limitation of the primal approach in CBwK due to the estimation of densities. General references on density estimations are the monographs by Devroye and Györfi [1985] in  $\mathbb{L}^1$  and Tsybakov [2008] in  $\mathbb{L}^2$ .
Note that in the dual approach, the knowledge of (the possibly complex)  $\nu$  is replaced by the knowledge of the (finite-dimensional) optimal dual variables  $\lambda_B^\star \in \mathbb{R}^d$ , which is easier to learn. This explains the fundamental efficiency of the dual approach compared to the primal approach.
F.1 Analysis with "soft" constraints
We first provide an analysis for a version of the Box D strategy that may possibly breach the total-cost constraints  $TB$ . More precisely, we allow deviations to  $TB$  of the order of  $\sqrt{T}$  times poly-log factors: this is what we refer to as "soft" constraints. Our result is that the regret may then be bounded by a quantity of order  $\sqrt{T}$  times poly-log factors, at least when  $\mathcal{X}$  is finite.
We do so for two reasons: first, because we do not think that this is a well-known result, and second, for pedagogic reasons, as the proof for "hard" constraints follows from adapting the proof scheme for soft constraints (see Appendix F.2).
Proposition 1 (soft constraints). Fix  $\delta \in (0,1)$ . Under Assumption 2 and with Notation (1), the strategy of Box  $D$ , run with  $\delta /4$  and positive slacks  $b_{t} = \xi_{t - 1,\delta /4}$ , ensures that with probability at least  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} \leqslant T \boldsymbol {B} + \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + 2 \Xi_ {T, \delta / 4}\right) \mathbf {1} \quad a n d \quad R _ {T} \leqslant 2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + 2 \Xi_ {T, \delta / 4},
$$
where  $\alpha_{T,\delta /4} = \sqrt{2T\ln\bigl((d + 1) / (\delta / 4)\bigr)}$
In particular, when  $\mathcal{X}$  is finite, the deviation terms  $2\alpha_{T,\delta /4} + \beta_{T,\delta /4} + 2\Xi_{T,\delta /4}$  are of order  $\sqrt{T}$  up to poly-logarithmic terms, so that the bound of Proposition 1 reads: with high-probability,
$$
\sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} \leqslant T \boldsymbol {B} + \widetilde {\mathcal {O}} (\sqrt {T}) \quad \text {a n d} \quad R _ {T} \leqslant \widetilde {\mathcal {O}} (\sqrt {T}).
$$
Put differently, soft-constraint satisfaction allows for  $\widetilde{\mathcal{O}}\big(\sqrt{T}\big)$  regret bounds when  $\mathcal{X}$  is finite.
Proof. As in the proofs of Lemmas 1 and 2 in Appendix B, we consider four events, each of probability at least  $1 - \delta /4$ : two events  $\mathcal{E}_{\mathrm{H - Az1}}$  and  $\mathcal{E}_{\mathrm{H - Az2}}$ , defined below, following from applications of the Hoeffding-Azuma inequality, the favorable event  $\mathcal{E}_{\mathrm{TVD}}$  of Notation 1 with  $\delta /4$ , and the favorable event  $\mathcal{E}_{\beta}$  of Assumption 2 with  $\delta /4$ . Namely, given the value for  $\alpha_{T,\delta /4}$  proposed in the statement (note this value is slightly different from the one considered in Appendix B), we have, on the one hand, on  $\mathcal{E}_{\mathrm{H - Az1}}$ ,
$$
\sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} \leqslant \alpha_ {T, \delta / 4} \mathbf {1} + \sum_ {t = 1} ^ {T} \boldsymbol {c} \left(\boldsymbol {x} _ {t}, a _ {t}\right) \quad \text {a n d} \quad \sum_ {t = 1} ^ {T} r _ {t} \geqslant - \alpha_ {T, \delta / 4} + \sum_ {t = 1} ^ {T} r \left(\boldsymbol {x} _ {t}, a _ {t}\right), \tag {20}
$$
and on the other hand, on  $\mathcal{E}_{\mathrm{H - A}2}$
$$
\sum_ {t = 1} ^ {T} \widehat {\mathbf {c}} _ {\delta / 4, t - 1} ^ {\mathrm {l c b}} \left(\boldsymbol {x} _ {t}, a _ {t}\right) \leqslant \alpha_ {T, \delta / 4} \mathbf {1} + \sum_ {t = 1} ^ {T} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \widehat {\mathbf {c}} _ {\delta / 4, t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X}, a) \pi_ {t, a} (\boldsymbol {X}) \right]
$$
$$
\text {a n d} \quad \sum_ {t = 1} ^ {T} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} \left(\boldsymbol {x} _ {t}, a _ {t}\right) \geqslant - \alpha_ {T, \delta / 4} + \sum_ {t = 1} ^ {T} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {t, a} (\boldsymbol {X}) \right]. \tag {21}
$$
The first two inequalities are obtained by considering conditional expectations with respect to the past,  $\boldsymbol{x}_t$  and  $a_t$ , while the second two inequalities follow from taking conditional expectations with respect to the past and  $\boldsymbol{x}_t$ ; we crucially use that, by definition,  $\boldsymbol{x}_t \sim \nu$  is independent from  $\pi_t$  and the estimates  $\widehat{r}_{\delta/4,t-1}^{\mathrm{ucb}}$  and  $\widehat{c}_{\delta/4,t-1}^{\mathrm{cb}}$ .