SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first note that by  $B$ -feasibility of the problem, by (3), and by Notation (1), a policy  $\pi_t$  satisfying the constraints stated in Box D exists at each round  $t \geqslant 1$  on the event  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{TVD}}$ . Indeed, denoting by  $\pi^{\mathrm{feas}}$  such a  $B$ -feasible policy, this existence follows from the inequalities
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {\boldsymbol {c}} _ {\delta / 4, t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X}, a) \pi_ {a} ^ {\text {f e a s}} (\boldsymbol {X}) \right] \leqslant \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \pi_ {a} ^ {\text {f e a s}} (\boldsymbol {X}) \right] \\ \leqslant \xi_ {t - 1, \delta / 4} + \underbrace {\mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X} , a) \pi_ {a} ^ {\text {f e a s}} (\boldsymbol {X}) \right]} _ {\leqslant B} \tag {22} \\ \end{array}
$$
and from the choice  $b_{t} = \xi_{t - 1,\delta /4}$
Cost-wise, we then successively have, by (20), then (3) and Assumption 2, and finally (21) and Notation (1), on the event  $\mathcal{E}_{\mathrm{H - Az1}}\cap \mathcal{E}_{\beta}\cap \mathcal{E}_{\mathrm{H - Az2}}\cap \mathcal{E}_{\mathrm{TVD}}$  of probability at least  $1 - \delta$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} \leqslant \alpha_ {T, \delta / 4} \mathbf {1} + \sum_ {t = 1} ^ {T} \boldsymbol {c} (\boldsymbol {x} _ {t}, a _ {t}) \\ \leqslant \left(\alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4}\right) \mathbf {1} + \sum_ {t = 1} ^ {T} \widehat {\mathbf {c}} _ {\delta / 4, t - 1} ^ {\mathrm {l c b}} \left(\boldsymbol {x} _ {t}, a _ {t}\right) \\ \leqslant \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4}\right) \mathbf {1} + \sum_ {t = 1} ^ {T} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \widehat {\boldsymbol {c}} _ {\delta / 4, t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X}, a) \pi_ {t, a} (\boldsymbol {X}) \right] \\ \leqslant \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + \Xi_ {T, \delta / 4}\right) \mathbf {1} + \sum_ {t = 1} ^ {T} \underbrace {\mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4 , t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {\boldsymbol {c}} _ {\delta / 4 , t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X} , a) \pi_ {t , a} (\boldsymbol {X}) \right]} _ {\leqslant B + b _ {t} \mathbf {1}}, \\ \end{array}
$$
where the inequalities  $\leqslant B + b_{t}\mathbf{1}$  follow from the definition of  $\pi_t$  in Box D. Substituting the choice  $b_{t} = \xi_{t - 1,\delta /4}$ , we thus proved that on  $\mathcal{E}_{\mathrm{H - Az1}}\cap \mathcal{E}_{\beta}\cap \mathcal{E}_{\mathrm{H - Az2}}\cap \mathcal{E}_{\mathrm{TVD}}$
$$
\sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} \leqslant T \boldsymbol {B} + \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + 2 \Xi_ {T, \delta / 4}\right) \mathbf {1},
$$
The control for rewards mimics the steps above for costs (and then resorts to an additional argument). We obtain first that on  $\mathcal{E}_{\mathrm{H - Az1}}\cap \mathcal{E}_{\beta}\cap \mathcal{E}_{\mathrm{H - Az2}}\cap \mathcal{E}_{\mathrm{TVD}}$
$$
\sum_ {t = 1} ^ {T} r _ {t} \geqslant - \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + \Xi_ {T, \delta / 4}\right) + \sum_ {t = 1} ^ {T} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {t, a} (\boldsymbol {X}) \right].
$$
We denote by  $\pi_B^\star$  an optimal static policy for the average cost constraints  $B$  — when it exists, e.g., by (4), as soon as the problem is feasible for some  $B' < B$ ; otherwise, we take a static policy achieving  $\mathrm{OPT}(r, c, B)$  up to some small  $e > 0$ , which we let vanish in a final step of the proof. As in (22), we have, on  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{TVD}}$ ,
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {\boldsymbol {c}} _ {\delta / 4, t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X}, a) \pi_ {\boldsymbol {B}, a} ^ {\star} (\boldsymbol {X}) \right] \leqslant \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \pi_ {\boldsymbol {B}, a} ^ {\star} (\boldsymbol {X}) \right] \\ \leqslant \xi_ {t - 1, \delta / 4} \mathbf {1} + \underbrace {\mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X} , a) \pi_ {\boldsymbol {B} , a} ^ {\star} (\boldsymbol {X}) \right]} _ {\leqslant B}, \\ \end{array}
$$
where the  $\leqslant B$  inequality follows from the definition of  $\pi_B^\star$ . Thanks to the choice  $b_{t} = \xi_{t - 1,\delta /4}$ , we have, by definition of  $\pi_t$  as some optimal policy in Box D and on  $\mathcal{E}_{\beta}\cap \mathcal{E}_{\mathrm{TVD}}$
$$
\mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {t, a} (\boldsymbol {X}) \right] \geqslant \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {\boldsymbol {B}, a} ^ {\star} (\boldsymbol {X}) \right].
$$
Again by (3) and Notation (1), we have, on  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{TVD}}$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {\boldsymbol {B}, a} ^ {\star} (\boldsymbol {X}) \right] \geqslant \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} r (\boldsymbol {X}, a) \pi_ {\boldsymbol {B}, a} ^ {\star} (\boldsymbol {X}) \right] \\ \geqslant - \xi_ {t - 1, \delta / 4} + \underbrace {\mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} r (\boldsymbol {X} , a) \pi_ {\boldsymbol {B} , a} ^ {\star} (\boldsymbol {X}) \right]} _ {= \mathrm {O P T} (r, c, B)}. \\ \end{array}
$$
Collecting all bounds, we proved that on  $\mathcal{E}_{\mathrm{H - Az1}}\cap \mathcal{E}_{\beta}\cap \mathcal{E}_{\mathrm{H - Az2}}\cap \mathcal{E}_{\mathrm{TVD}}$
$$
\sum_ {t = 1} ^ {T} r _ {t} \geqslant T \mathrm {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + 2 \Xi_ {T, \delta / 4}\right),
$$
which corresponds to the claimed regret bound.
F.2 Analysis with "hard" constraints and a null-cost action
We now turn our attention to the main kind of result that we want to achieve: when constraints must be strictly satisfied—which we refer to as "hard" constraints. For the sake of simplicity, we do so for now in the presence of a null-cost action; Appendix F.3 explains how the analysis may be generalized to cases without such null-cost actions.
The following result corresponds to the combination of Theorem 1 with Corollary 1, and also, to Li and Stoltz [2022, main result: Theorem 2].
Theorem 3 (hard constraints). Fix  $\delta \in (0,1)$ . We consider the strategy of Box  $D$ , run with  $\delta /4$  and negative slacks all equal to
$$
b _ {t} \equiv - \overline {{\Delta}} _ {T, \delta / 4}, \quad \text {w h e r e} \quad \overline {{\Delta}} _ {T, \delta / 4} \stackrel {\text {d e f}} {=} \frac {2 \alpha_ {T , \delta / 4} + \beta_ {T , \delta / 4} + \Xi_ {T , \delta / 4}}{T}
$$
and  $\alpha_{T,\delta /4} = \sqrt{2T\ln\big((d + 1) / (\delta / 4)\big)}$ . Assume that a null-cost action exists, that  $\overline{\Delta}_{T,\delta /4} < \min B$ , that Assumption 2 holds, and use Notation (1). Then, with probability at least  $1 - \delta$
$$
\sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} \leqslant T \boldsymbol {B} \quad a n d \quad R _ {T} \leqslant \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + 2 \Xi_ {T, \delta / 4}\right) \left(1 + \frac {\mathrm {O P T} (r , \boldsymbol {c} , \boldsymbol {B}) - \mathrm {O P T} (r , \boldsymbol {c} , \boldsymbol {0})}{\min  \boldsymbol {B}}\right).
$$
Proof. We explain how the proof of Proposition 1 may be adapted. We first justify the existence at each round  $t \geqslant 1$  of a policy  $\pi_t$  satisfying the cost constraints stated in Box D: for the null-cost action  $a_{\mathrm{null}}$ , we may impose that, for all  $x \in \mathcal{X}$ , all  $t \geqslant 0$ , and all  $\delta \in (0,1)$ ,
$$
\widehat {\boldsymbol {c}} _ {t} (\boldsymbol {x}, a _ {\text {n u l l}}) = \mathbf {0} \quad \text {a n d} \quad \varepsilon_ {t} (\boldsymbol {x}, a _ {\text {n u l l}}, \delta) = 0, \quad \text {s o t h a t} \quad \widehat {\boldsymbol {c}} _ {\delta , t} ^ {\text {l c b}} (\boldsymbol {x}, a _ {\text {n u l l}}) = \mathbf {0} \quad \text {a . s .};
$$
this shows that at least the static policy  $\pi^{\mathrm{null}}$  always playing  $a_{\mathrm{null}}$  satisfying the defining cost-constraints for  $\pi_t$ . (Alternatively, we may note that the policy  $\overline{\pi}_t$  defined below also satisfies the cost constraints stated in Box D, on the high-probability event considered.)
We then handle total-cost constraints similarly as in the proof of Proposition 1 and obtain that on the event  $\mathcal{E}_{\mathrm{H - Az1}}\cap \mathcal{E}_{\beta}\cap \mathcal{E}_{\mathrm{H - Az2}}\cap \mathcal{E}_{\mathrm{TVD}}$  of probability at least  $1 - \delta$
$$
\sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} \leqslant \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + \Xi_ {T, \delta / 4}\right) \mathbf {1} + \sum_ {t = 1} ^ {T} \underbrace {\mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4 , t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {\boldsymbol {c}} _ {\delta / 4 , t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X} , a) \pi_ {t , a} (\boldsymbol {X}) \right]} _ {\leqslant \boldsymbol {B} - \overline {{\Delta}} _ {T, \delta / 4} \mathbf {1}} \leqslant T \boldsymbol {B},
$$
where this time, the slacks  $b_{t} = -\overline{\Delta}_{T,\delta /4}$  are negative and were set to cancel out the positive deviation terms. Similarly, on  $\mathcal{E}_{\mathrm{H - Az1}}\cap \mathcal{E}_{\beta}\cap \mathcal{E}_{\mathrm{H - Az2}}\cap \mathcal{E}_{\mathrm{TVD}}$ ,
$$
\sum_ {t = 1} ^ {T} r _ {t} \geqslant - T \overline {{\Delta}} _ {T, \delta / 4} + \sum_ {t = 1} ^ {T} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {t, a} (\boldsymbol {X}) \right]. \tag {23}
$$
Now, the main modification to the proof of Proposition 1 is that (with its notation) we rather consider the policy
$$
\overline {{\boldsymbol {\pi}}} _ {t} = (1 - w _ {t}) \boldsymbol {\pi} _ {\boldsymbol {B}} ^ {\star} + w _ {t} \boldsymbol {\pi} ^ {\text {n u l l}}, \qquad \text {w h e r e} \quad w _ {t} = \min  \left\{\frac {\overline {{\Delta}} _ {T , \delta / 4} + \xi_ {t - 1 , \delta / 4}}{\min  \boldsymbol {B}}, 1 \right\}.
$$
As in (22), we see that this policy satisfies, on  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{TVD}}$ :
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {\boldsymbol {c}} _ {\delta / 4, t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X}, a) \overline {{\pi}} _ {t, a} (\boldsymbol {X}) \right] \leqslant \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \overline {{\pi}} _ {t, a} (\boldsymbol {X}) \right] \\ \leqslant \xi_ {t - 1, \delta / 4} \mathbf {1} + \underbrace {\mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} c (\boldsymbol {X} , a) \bar {\pi} _ {t , a} (\boldsymbol {X}) \right]} _ {\leqslant (1 - w _ {t}) \boldsymbol {B}}. \\ \end{array}
$$
By using  $B \geqslant (\min B)1$  and since we assumed that  $\overline{\Delta}_{T,\delta /4} < \min B$ , we may continue the series of inequalities as follows:
$$
\begin{array}{l} \xi_ {t - 1, \delta / 4} \mathbf {1} + (1 - w _ {t}) \mathbf {B} \leqslant \mathbf {B} + \xi_ {t - 1, \delta / 4} \mathbf {1} - w _ {t} (\min  \mathbf {B}) \mathbf {1} \\ = \boldsymbol {B} - \min  \left\{\overline {{\Delta}} _ {T, \delta / 4}, \min  \boldsymbol {B} - \xi_ {t - 1, \delta / 4} \right\} \mathbf {1} \\ \leqslant \boldsymbol {B} - \min  \left\{\overline {{\Delta}} _ {T, \delta / 4}, \overline {{\Delta}} _ {T, \delta / 4} - \xi_ {t - 1, \delta / 4} \right\} \mathbf {1} = \boldsymbol {B} + b _ {t} \mathbf {1}, \\ \end{array}
$$
meaning that  $\overline{\pi}_t$  is a policy satisfying the constraints stated in Step 2 of Box D on round  $t\geqslant 1$ . The consequence is that, first by definition of  $\pi_t$  as a maximizer and then by the optimistic estimates (3) and Notation (1), on  $\mathcal{E}_{\beta}\cap \mathcal{E}_{\mathrm{TVD}}$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {t, a} (\boldsymbol {X}) \right] \\ \geqslant \mathbb {E} _ {\boldsymbol {X}} \sim \widehat {\nu} _ {\delta / 4, t - 1} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \bar {\pi} _ {t, a} (\boldsymbol {X}) \right] \\ \geqslant - \xi_ {t - 1, \delta / 4} + \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in A} r (\boldsymbol {X}, a) \bar {\pi} _ {t, a} (\boldsymbol {X}) \right] \\ = - \xi_ {t - 1, \delta / 4} + (1 - w _ {t}) \mathrm {O P T} (r, c, B) + w _ {t} \mathrm {O P T} (r, c, \mathbf {0}) \\ = \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \xi_ {t - 1, \delta / 4} - \min  \left\{\frac {\overline {{\Delta}} _ {T , \delta / 4} + \xi_ {t - 1 , \delta / 4}}{\min  \boldsymbol {B}}, 1 \right\} \left(\operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {0})\right), \\ \end{array}
$$
where  $\mathrm{OPT}(r, c, 0)$  refers to the expected reward achieved by the null-cost action  $a_{\mathrm{null}}$ . After combination with (23) and summation, we get that on  $\mathcal{E}_{\mathrm{H - Az1}} \cap \mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{H - Az2}} \cap \mathcal{E}_{\mathrm{TVD}}$ :
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} r _ {t} - T \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) \\ \geqslant - T \bar {\Delta} _ {T, \delta / 4} - \Xi_ {T, \delta / 4} - \sum_ {t = 1} ^ {T} \min  \left\{\frac {\bar {\Delta} _ {T , \delta / 4} + \xi_ {t - 1 , \delta / 4}}{\min  \boldsymbol {B}}, 1 \right\} (\mathrm {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \mathrm {O P T} (r, \boldsymbol {c}, \boldsymbol {0})) \\ \geqslant - \left(T \overline {{\Delta}} _ {T, \delta / 4} + \Xi_ {T, \delta / 4}\right) \left(1 + \frac {\mathrm {O P T} (r , \boldsymbol {c} , \boldsymbol {B}) - \mathrm {O P T} (r , \boldsymbol {c} , \boldsymbol {0})}{\min  \boldsymbol {B}}\right), \\ \end{array}
$$
which corresponds to the stated regret bound.
F.3 General analysis with "hard" constraints
We finally generalize Theorem 3 and get a result corresponding to Theorem 1 combined with Lemma 3.
Here, we "mix" the slacks  $+\xi_{t - 1,\delta /4}$  and  $-\overline{\Delta}_{T,\delta /4}$  of Appendices F.1 and F.2, with a slight modification of  $\overline{\Delta}_{T,\delta /4}$  to compensate for the  $\xi_{t - 1,\delta /4}$ : we rather have a  $2\Xi_{T,\delta /4}$  term in the numerator of  $\overline{\Delta}_{T,\delta /4}^{\prime}$ , compared to the  $\Xi_{T,\delta /4}$  term in the numerator of  $\overline{\Delta}_{T,\delta /4}$ .
Theorem 4 (hard constraints). Fix  $\delta \in (0,1)$ . We consider the strategy of Box  $D$ , run with  $\delta / 4$  and signed slacks (depending on  $t \geqslant 1$ )
$$
b _ {t} = - \overline {{\Delta}} _ {T, \delta / 4} ^ {\prime} + \xi_ {t - 1, \delta / 4} \quad w h e r e \quad \overline {{\Delta}} _ {T, \delta / 4} ^ {\prime} = \frac {2 \alpha_ {T , \delta / 4} + \beta_ {T , \delta / 4} + 2 \Xi_ {T , \delta / 4}}{T}
$$
and  $\alpha_{T,\delta /4} = \sqrt{2T\ln((d + 1) / (\delta / 4))}$ . Assume that the problem is  $(B - m)$ -feasible for some  $m$  that does not need to be known by the strategy, with  $B - m \leqslant B - \overline{\Delta}_{T,\delta /4}^{\prime}\mathbf{1}$ . Then, under Assumption 2 and with Notation (1), with probability at least  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} \leqslant T \boldsymbol {B}
$$
$$
a n d \qquad R _ {T} \leqslant \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + 2 \Xi_ {T, \delta / 4}\right) \left(1 + \frac {\operatorname {O P T} (r , \boldsymbol {c} , \boldsymbol {B}) - \operatorname {O P T} (r , \boldsymbol {c} , \boldsymbol {B} - \boldsymbol {m})}{\min  \boldsymbol {m}}\right).
$$
Proof. We rather sketch the differences to the proofs of Theorem 3 and Proposition 1. We denote by  $\mathcal{E}_{\mathrm{all}}$  the event of probability at least  $1 - \delta$  obtained as the intersection of four convenient events of probability each at least  $1 - \delta /4$ . All inequalities below hold on  $\mathcal{E}_{\mathrm{all}}$  but for the sake of brevity, we will not highlight this fact each time.
We denote by  $\pi^{\mathrm{feas}}$  a (quasi-)optimal static policy among the ones achieving expected costs smaller than  $B - m$ ; it therefore ensures that its expected costs are smaller than  $B - m$  and achieves an expected reward of  $\mathrm{OPT}(r, c, B - m) - e$ , possibly up to a small factor  $e \geq 0$  which we let vanish. We note that for each round  $t \geq 1$ ,
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {X} \sim \hat {\nu} _ {\delta , t - 1}} \left[ \sum_ {a \in \mathcal {A}} \hat {\boldsymbol {c}} _ {\delta , t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X}, a) \pi_ {a} ^ {\mathrm {f e a s}} (\boldsymbol {X}) \right] \leqslant \xi_ {t - 1, \delta / T} \mathbf {1} + \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \pi_ {a} ^ {\mathrm {f e a s}} (\boldsymbol {X}) \right] \\ \leqslant \xi_ {t - 1, \delta / T} \mathbf {1} + \boldsymbol {B} - \boldsymbol {m} \mathbf {1} \leqslant \boldsymbol {B} + b _ {t} \mathbf {1}, \\ \end{array}
$$
given the definition  $b_{t} = -\overline{\Delta}_{T,\delta /4}^{\prime} + \xi_{t - 1,\delta /4}$  and the assumption  $B - m \leqslant B - \overline{\Delta}_{T,\delta /4}^{\prime}\mathbf{1}$ . Thus, on  $\mathcal{E}_{\mathrm{all}}$ , the strategy  $\pi_t$  is indeed defined at each round  $t \geqslant 1$  by the optimization problem stated in Step 2 of Box D (and not in an arbitrary manner).
Cost-wise, we thus have
$$
\begin{array}{l} \text {s t - w i s e , w e t h u s h a v e} \quad \sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} \leqslant \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + \Xi_ {T, \delta / 4}\right) \mathbf {1} + \sum_ {t = 1} ^ {T} \overbrace {\mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4 , t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {c} _ {\delta / 4 , t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X} , a) \pi_ {t , a} (\boldsymbol {X}) \right]} ^ {\leqslant B + b _ {t} \mathbf {1}} \\ \leqslant T \boldsymbol {B} + \left(2 \alpha_ {T, \delta / 4} + \beta_ {T, \delta / 4} + \Xi_ {T, \delta / 4}\right) \mathbf {1} - T \overline {{\Delta}} _ {T, \delta / 4} ^ {\prime} + \Xi_ {T, \delta / 4} = T \boldsymbol {B}, \\ \end{array}
$$
where the final equality follows from the definition of  $\overline{\Delta}_{T,\delta /4}^{\prime}$ , which involves a  $2\Xi_{T,\delta /4}$  term in its numerator.
Reward-wise, we introduce for each  $t \geqslant 1$ ,
$$
\overline {{\boldsymbol {\pi}}} _ {t} = (1 - w _ {t}) \boldsymbol {\pi} _ {\boldsymbol {B}} ^ {\star} + w _ {t} \boldsymbol {\pi} ^ {\text {f e a s}}, \quad \text {w h e r e} \quad w _ {t} = \min  \left\{\frac {\overline {{\Delta}} _ {T , \delta / 4} ^ {\prime}}{\min  \boldsymbol {m}}, 1 \right\},
$$
whose empirical expected cost at round  $t \geqslant 1$  is smaller than
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {X} \sim \hat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {\boldsymbol {c}} _ {\delta / 4, t - 1} ^ {\mathrm {l c b}} (\boldsymbol {X}, a) \bar {\pi} _ {t, a} (\boldsymbol {X}) \right] \leqslant \xi_ {t - 1, \delta / 4} \mathbf {1} + \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \bar {\pi} _ {t, a} (\boldsymbol {X}) \right] \\ \leqslant \xi_ {t - 1, \delta / 4} \mathbf {1} + (1 - w _ {t}) \mathbf {B} + w _ {t} (\mathbf {B} - \mathbf {m}) \\ = \boldsymbol {B} + \xi_ {t - 1, \delta / 4} \mathbf {1} - w _ {t} \boldsymbol {m}. \\ \end{array}
$$
By using  $m \geqslant (\min m)\mathbf{1}$  and thanks to the assumption  $B - m \leqslant B - \overline{\Delta}_{T,\delta /4}'\mathbf{1}$ , which can be equivalently formulated as  $\min m \geqslant \overline{\Delta}_{T,\delta /4}'$ , we may continue this series of inequalities by
$$
\begin{array}{l} \boldsymbol {B} + \xi_ {t - 1, \delta / 4} \mathbf {1} - w _ {t} \boldsymbol {m} \leqslant \boldsymbol {B} - \min  \left\{\overline {{\Delta}} _ {T, \delta / 4} ^ {\prime} - \xi_ {t - 1, \delta / 4}, \min  \boldsymbol {m} - \xi_ {t - 1, \delta / 4} \right\} \mathbf {1} \\ \leqslant \boldsymbol {B} - \min  \left\{\overline {{\Delta}} _ {T, \delta / 4} ^ {\prime} - \xi_ {t - 1, \delta / 4}, \overline {{\Delta}} _ {T, \delta / 4} ^ {\prime} - \xi_ {t - 1, \delta / 4} \right\} \mathbf {1} = \boldsymbol {B} + b _ {t} \mathbf {1}, \\ \end{array}
$$
meaning that  $\overline{\pi}_t$  is a policy satisfying the constraints stated in Step 2 of Box D on round  $t \geqslant 1$ . In particular, by definition of  $\pi_t$  as a maximizer,
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {t, a} (\boldsymbol {X}) \right] \\ \geqslant \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \bar {\pi} _ {t, a} (\boldsymbol {X}) \right] \\ \geqslant - \xi_ {t - 1, \delta / 4} + \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} r (\boldsymbol {X}, a) \bar {\pi} _ {t, a} (\boldsymbol {X}) \right] \\ \geqslant - \xi_ {t - 1, \delta / 4} + \left((1 - w _ {t}) \operatorname {O P T} (r, c B) + w _ {t} \operatorname {O P T} (r, c B - m)\right) \\ = \operatorname {O P T} (r, c B) - \min  \left\{\frac {\overline {{\Delta}} _ {T , \delta / 4} ^ {\prime}}{\min  m}, 1 \right\} \left(\operatorname {O P T} (r, c B) - \operatorname {O P T} (r, c B - m)\right) - \xi_ {t - 1, \delta / 4}. \tag {24} \\ \end{array}
$$
Finally, by combining (23) and (24),
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} r _ {t} \geqslant - \overbrace {\left(2 \alpha_ {T , \delta / 4} + \beta_ {T , \delta / 4} + \Xi_ {T , \delta / 4}\right)} ^ {= T \bar {\Delta} _ {T, \delta / 4} ^ {\prime} - \Xi_ {T, \delta / 4}} \mathbf {1} + \sum_ {t = 1} ^ {T} \mathbb {E} _ {\boldsymbol {X} \sim \widehat {\nu} _ {\delta / 4, t - 1}} \left[ \sum_ {a \in \mathcal {A}} \widehat {r} _ {\delta / 4, t - 1} ^ {\mathrm {u c b}} (\boldsymbol {X}, a) \pi_ {t, a} (\boldsymbol {X}) \right] \\ \geqslant T \operatorname {O P T} (r, \boldsymbol {c B}) + \Xi_ {T, \delta / 4} - T \overline {{\Delta}} _ {T, \delta / 4} ^ {\prime} \left(1 + \frac {\operatorname {O P T} (r , \boldsymbol {c B}) - \operatorname {O P T} (r , \boldsymbol {c B} - \boldsymbol {m})}{\min  \boldsymbol {m}}\right) - \Xi_ {T, \delta / 4}, \\ \end{array}
$$
G Numerical experiments: full description
This appendix reports numerical simulations performed on simulated data with the motivating example described in Chohlas-Wood et al. [2021] and alluded at in Section 2.1. These simulations are for the sake of illustration only.