SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and  $\left|G(\pmb{\lambda}_{\pmb{B} - b\mathbf{1}}^{\star}) - G(\pmb{\lambda}_{\tau - 1})\right| \leqslant 1 + 2\|\pmb{\lambda}_{\pmb{B} - b\mathbf{1}}^{\star} - \pmb{\lambda}_{\tau - 1}\|_1.$
Now, given the update step in Step 3 of Box B, we have the deterministic bound  $\| \pmb{\lambda}_t\| _1\leqslant 2d\gamma t$  for all  $1\leqslant t\leqslant T$ . Therefore, by a triangle inequality, the martingale increments are bounded in absolute values by  $K_{t}$ , as
$$
2 \left(1 + 2 \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| _ {1} + 1 + 2 \max  _ {\tau \leqslant t} \| \boldsymbol {\lambda} _ {\tau - 1} \| _ {1}\right) \leqslant 4 \left(1 + \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| _ {1} + 2 d \gamma t\right) = K _ {t}.
$$
The conditional variance of  $X_{\tau}$  is smaller than the squared half-width of the conditional range (Popoviciu's inequality on variances); in particular, (15) thus entails
$$
\mathfrak {S} _ {t} \leqslant \sum_ {\tau = 1} ^ {t} \left(1 + 2 \| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} - \boldsymbol {\lambda} _ {\tau - 1} \| _ {1}\right) ^ {2} \leqslant (1 + 2 \Lambda_ {t}) ^ {2} t.
$$
We now get the claimed inequalities (14) first by noting that by (4), for all  $\tau \leqslant T$ ,
$$
G \left(\boldsymbol {\lambda} _ {\boldsymbol {B} - b 1} ^ {\star}\right) - G \left(\boldsymbol {\lambda} _ {\tau - 1}\right) \leqslant 0,
$$
and second, by applying Lemma 4 for each  $1 \leqslant t \leqslant T$ , using a confidence level  $\delta/(4T)$ . By a union bound, this indeed defines an event  $\mathcal{E}_{\mathrm{Bem} - c}$  of probability at least  $1 - \delta/4$ .
Step 4: Induction to bound the  $\Lambda_{t}$ . In this step, we show by induction that, with high probability, the norms  $\| \pmb{\lambda}_{B - b1}^{\star} - \pmb{\lambda}_t\|$  satisfy the bound (18) stated below.
To do so, we combine the outcomes of Steps 1-3 and obtain that on the event  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{Bem-c}}$ , for all  $1 \leqslant t \leqslant T$ ,
$$
\begin{array}{l} \left\| \lambda_ {B - b 1} ^ {\star} - \lambda_ {t} \right\| ^ {2} \\ \leqslant \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| ^ {2} + 4 d \gamma^ {2} t + 2 \gamma \left(2 \left(1 + \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| _ {1}\right) \beta_ {t, \delta / 4} + \sum_ {\tau = 1} ^ {t} \left(g _ {\tau} \left(\boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star}\right) - g _ {\tau} \left(\boldsymbol {\lambda} _ {\tau - 1}\right)\right)\right) \\ \leqslant \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| ^ {2} + 4 d \gamma^ {2} t + 4 \gamma \left(1 + \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| _ {1}\right) \beta_ {t, \delta / 4} + 2 \gamma \left(1 + 2 \Lambda_ {t}\right) \sqrt {2 t \ln \frac {T ^ {2}}{\delta / 4}} + 4 \gamma K _ {t} \ln \frac {T ^ {2}}{\delta / 4}, \\ \end{array}
$$
where we recall that norms not indexed by a subscript are Euclidean norms, and
$$
\Lambda_ {t} = \max  _ {1 \leqslant \tau \leqslant t} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} - \boldsymbol {\lambda} _ {\tau - 1} \| _ {1} \quad \text {a n d} \quad K _ {t} = 4 \big (1 + \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| _ {1} + 2 d \gamma t \big).
$$
We upper bound  $\| \pmb{\lambda}_{B - b1}^{\star}\|_{1}$  and  $\Lambda_t$  in terms of Euclidean norms,
$$
\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| _ {1} \leqslant \sqrt {d} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| \quad \text {a n d} \quad \Lambda_ {t} \leqslant \sqrt {d} \max  _ {1 \leqslant \tau \leqslant t} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} - \boldsymbol {\lambda} _ {\tau - 1} \|,
$$
perform some crude boundings like  $\sqrt{t} \leqslant \sqrt{T}$  and  $\beta_{t,\delta/4} \leqslant \beta_{T,\delta/4}$ , and obtain the following induction relationship: on the event  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{Bern-c}}$ , for all  $1 \leqslant t \leqslant T$ ,
$$
\left\| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} - \boldsymbol {\lambda} _ {t} \right\| ^ {2} \leqslant A + B t + C \max  _ {1 \leqslant \tau \leqslant t} \left\| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} - \boldsymbol {\lambda} _ {\tau - 1} \right\|, \tag {16}
$$
$$
A = \| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} \| ^ {2} + \gamma \left(4 \big (1 + \sqrt {d} \| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} \| \big) \beta_ {T, \delta / 4} + 2 \sqrt {2 T \ln \frac {T ^ {2}}{\delta / 4}} + 1 6 \big (1 + \sqrt {d} \| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} \| \big) \ln \frac {T ^ {2}}{\delta / 4}\right),
$$
$$
B = 4 d \gamma^ {2} + 4 \times 4 \times 2 d \gamma^ {2} \ln {\frac {T ^ {2}}{\delta / 4}} = \left(4 + 3 2 \ln {\frac {T ^ {2}}{\delta / 4}}\right) d \gamma^ {2} \leqslant 3 6 d \gamma^ {2} \ln {\frac {T ^ {2}}{\delta / 4}},
$$
$$
C = 4 \gamma \sqrt {2 d T \ln \frac {T ^ {2}}{\delta / 4}}.
$$
We now show that (16) implies that on  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{Bern - c}}$
$$
\forall 0 \leqslant t \leqslant T, \quad \| \boldsymbol {\lambda} _ {B - b \mathbf {1}} ^ {\star} - \boldsymbol {\lambda} _ {t} \| \leqslant M \stackrel {\text {d e f}} {=} \frac {C}{2} + \sqrt {A + B T + \frac {C ^ {2}}{4}}. \tag {17}
$$
Indeed, for  $t = 0$ , given that  $\lambda_0 = 0$ , we have  $\| \pmb{\lambda}_{B - b1}^{\star} - \pmb{\lambda}_t\| = \| \pmb{\lambda}_{B - b1}^{\star}\| \leqslant \sqrt{A}$ . Now, if the bound (17) is satisfied for all  $0 \leqslant \tau \leqslant t$ , where  $0 \leqslant t \leqslant T - 1$ , then (16) implies that
$$
\left\| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} - \boldsymbol {\lambda} _ {t + 1} \right\| \leqslant A + B (t + 1) + C M \leqslant A + B T + C M \leqslant M ^ {2},
$$
where the final inequality follows from the fact that (by definition of  $M$ , and this explains how we picked  $M$ )
$$
M ^ {2} - C M = \left(M - \frac {C}{2}\right) ^ {2} + \frac {C ^ {2}}{4} = A + B T.
$$
Below, we will make repeated uses of  $\sqrt{x + y} \leqslant \sqrt{x} + \sqrt{y}$ , of  $xy \leqslant 2(x^2 + y^2)$ , and of  $\sqrt{x} \leqslant 1 + x$  for  $x, y \geqslant 0$ . From (17), we conclude that on the event  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{Bern - c}}$ ,
$$
\begin{array}{l} \forall 0 \leqslant t \leqslant T, \quad \| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} - \boldsymbol {\lambda} _ {t} \| \leqslant \sqrt {A} + \sqrt {B T} + C \\ = \sqrt {A} + 6 \gamma \sqrt {d T \ln \frac {T ^ {2}}{\delta / 4}} + 4 \gamma \sqrt {2 d T \ln \frac {T ^ {2}}{\delta / 4}} \leqslant \sqrt {A} + 6 \gamma \beta_ {T, \delta / 4} ^ {\prime}, \\ \end{array}
$$
where we denoted
$$
\beta_ {T, \delta / 4} ^ {\prime} = \max \left\{\beta_ {T, \delta / 4}, 2 \sqrt {d T \ln \frac {T ^ {2}}{\delta / 4}} \right\} \geqslant \ln \frac {T ^ {2}}{\delta / 4}.
$$
For the sake of readability, we may further bound  $\sqrt{A}$  as follows (in some crude way):
$$
\begin{array}{l} \sqrt {A} \leqslant \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| + \sqrt {\gamma} \sqrt {4 \left(1 + \sqrt {d} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \|\right) \beta_ {T , \delta / 4} + 2 \sqrt {2 T \ln \frac {T ^ {2}}{\delta / 4}} + 1 6 \left(1 + \sqrt {d} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \|\right) \ln \frac {T ^ {2}}{\delta / 4}} \\ \leqslant \| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} \| + \sqrt {\gamma} \sqrt {2 2 \beta_ {T , \delta / 4} ^ {\prime} + 2 0 \sqrt {d}} \| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} \| \beta_ {T, \delta / 4} ^ {\prime} \\ \leqslant \left\| \boldsymbol {\lambda} _ {B - b \mathbf {1}} ^ {\star} \right\| + 1 + 6 \gamma \beta_ {T, \delta / 4} ^ {\prime} + \left\| \boldsymbol {\lambda} _ {B - b \mathbf {1}} ^ {\star} \right\| + 5 \gamma \sqrt {d} \beta_ {T, \delta / 4} ^ {\prime}, \\ \end{array}
$$
where we used the facts that  $\sqrt{20xy} = 2\sqrt{5xy} \leqslant x + 5y$  and  $\sqrt{22x} = 2\sqrt{22x / 4} \leqslant 1 + 6x$ .
All in all, we proved that on the event  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{Bern - c}}$
$$
\forall 0 \leqslant t \leqslant T, \quad \| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} - \boldsymbol {\lambda} _ {t} \| \leqslant 2 \| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} \| + 1 7 \gamma \sqrt {d} \beta_ {T, \delta / 4} ^ {\prime} + 1. \tag {18}
$$
Step 5: Conclusion. We combine the bound (18) with the bound (9) of Step 1 and the bound (7): on the intersection of events  $\mathcal{E}_{\beta} \cap \mathcal{E}_{\mathrm{Bern-c}} \cap \mathcal{E}_{\mathrm{H-Az}}$ , which has a probability at least  $1 - 3\delta / 4$ , for all  $1 \leqslant t \leqslant T$ ,
$$
\begin{array}{l} \left\| \left(\sum_ {\tau = 1} ^ {t} \boldsymbol {c} _ {\tau} - t (\boldsymbol {B} - b \mathbf {1})\right) _ {+} \right\| \leqslant \sqrt {d} \left(\alpha_ {T, \delta / 4} + 2 \beta_ {T, \delta / 4}\right) + \frac {\| \boldsymbol {\lambda} _ {t} \|}{\gamma} \\ \leqslant \sqrt {d} \left(\alpha_ {T, \delta / 4} + 2 \beta_ {T, \delta / 4}\right) + \frac {\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \|}{\gamma} + \frac {\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} - \boldsymbol {\lambda} _ {t} \|}{\gamma} \\ \leqslant \frac {3 \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| + 1}{\gamma} + \sqrt {d} \left(\alpha_ {T, \delta / 4} + 1 9 \beta_ {T, \delta / 4} ^ {\prime}\right). \\ \end{array}
$$
This entails in particular the stated result, given the definition of  $\Upsilon_{T,\delta}$  as  $\max \left\{\beta_{T,\delta /4}^{\prime},\alpha_{T,\delta /4}\right\}$ .
The proof is similar to (but much simpler and shorter than) the one of Lemma 1 and borrows some of its arguments. We use throughout this section the notation introduced therein; we also define a new event  $\mathcal{E}_{\mathrm{Bern - r}}$  of probability at least  $1 - \delta /4$
We start from (8) and introduce the same  $\Delta \widehat{c}_t$  quantity as in (10): on the event  $\mathcal{E}_{\mathrm{H - Az}}\cap \mathcal{E}_{\beta}$ , for all  $1\leqslant t\leqslant T$
$$
\begin{array}{l} \sum_ {\tau = 1} ^ {t} r _ {\tau} \geqslant - \left(\alpha_ {t, \delta / 4} + 2 \beta_ {t, \delta / 4}\right) + \sum_ {\tau = 1} ^ {t} \widehat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) \\ \geqslant - \left(\alpha_ {t, \delta / 4} + 2 \beta_ {t, \delta / 4}\right) + \sum_ {\tau = 1} ^ {t} \left(\widehat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) - \left\langle \Delta \widehat {\boldsymbol {c}} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} \right\rangle\right) + \sum_ {\tau = 1} ^ {t} \left\langle \Delta \widehat {\boldsymbol {c}} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} \right\rangle . \\ \end{array}
$$
On the one hand, the result (11) with  $\lambda = 0$  exactly states that
$$
\sum_ {\tau = 1} ^ {t} \left\langle \Delta \hat {c} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} \right\rangle \geqslant \frac {\left\| \boldsymbol {\lambda} _ {t} \right\| ^ {2}}{2 \gamma} - 2 d \gamma t \geqslant - 2 d \gamma t.
$$
On the other hand, the result (13) states that on  $\mathcal{E}_{\beta}$ , for all  $1 \leqslant \tau \leqslant T$
$$
\begin{array}{l} \hat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - \left\langle \Delta \hat {\boldsymbol {c}} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} \right\rangle = \hat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - \left\langle \hat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} _ {\tau - 1} \right\rangle \\ \geqslant g _ {\tau} \left(\lambda_ {\tau - 1}\right). \\ \end{array}
$$
A similar application of Lemma 4 as in the proof of Lemma 1 shows that on a new event  $\mathcal{E}_{\mathrm{Bern - r}}$  of probability at least  $1 - \delta /4$ , for all  $1\leqslant t\leqslant T$
$$
\sum_ {\tau = 1} ^ {t} g _ {\tau} (\boldsymbol {\lambda} _ {\tau - 1}) \geqslant \sum_ {\tau = 1} ^ {t} G (\boldsymbol {\lambda} _ {\tau - 1}) - \sqrt {2 \sum_ {\tau = 1} ^ {t} \left(1 + 2 \| \boldsymbol {\lambda} _ {\tau - 1} \| _ {1}\right) ^ {2} \ln \frac {T ^ {2}}{\delta / 4}} - 8 (1 + 2 \gamma t) \ln \frac {T ^ {2}}{\delta / 4}.
$$
We relate  $\ell^1$ -norms to Euclidean norms, resort to a triangle inequality, and substitute (18) to get that on  $\mathcal{E}_{\mathrm{Bern - c}}$ , for all  $1 \leqslant t \leqslant T$ ,
$$
\begin{array}{l} - \sqrt {2 \sum_ {\tau = 1} ^ {t} \left(1 + 2 \| \boldsymbol {\lambda} _ {\tau - 1} \| _ {1}\right) ^ {2} \ln \frac {T ^ {2}}{\delta / 4}} \geqslant - \left(1 + 2 \sqrt {d} \max  _ {0 \leqslant \tau \leqslant t - 1} \| \boldsymbol {\lambda} _ {\tau - 1} \|\right) \sqrt {2 t \ln \frac {T ^ {2}}{\delta / 4}} \\ \geqslant - \left(1 + 2 \sqrt {d} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| + 2 \sqrt {d} \max  _ {0 \leqslant \tau \leqslant t - 1} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} - \boldsymbol {\lambda} _ {\tau - 1} \|\right) \sqrt {2 t \ln \frac {T ^ {2}}{\delta / 4}} \\ \geqslant - \left(8 \sqrt {d} \left\| \pmb {\lambda} _ {\pmb {B} - b \mathbf {1}} ^ {\star} \right\| + 3 4 \gamma d \beta_ {T, \delta / 4} ^ {\prime} + 2 \sqrt {d} + 1\right) \sqrt {2 t \ln \frac {T ^ {2}}{\delta / 4}} \\ \geqslant - \left(8 \sqrt {d} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| + 3 4 \gamma d \beta_ {T, \delta / 4} ^ {\prime} + 4 \sqrt {d}\right) \sqrt {2 t \ln \frac {T ^ {2}}{\delta / 4}} \\ \geqslant - 6 \left\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \right\| \beta_ {T, \delta / 4} ^ {\prime} - 2 5 \gamma \sqrt {d} \left(\beta_ {T, \delta / 4} ^ {\prime}\right) ^ {2} - 2 \sqrt {2} \beta_ {T, \delta / 4} ^ {\prime} \\ \geqslant - 6 \left\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \right\| \beta_ {T, \delta / 4} ^ {\prime} - 2 8 \gamma \sqrt {d} \left(\beta_ {T, \delta / 4} ^ {\prime}\right) ^ {2}, \\ \end{array}
$$
where we performed some crude boundings using the definition of  $1 \leqslant \beta_{t,\delta /4}^{\prime} \leqslant \beta_{T,\delta /4}^{\prime}$ . We also note that
$$
8 (1 + 2 \gamma t) \ln \frac {T ^ {2}}{\delta / 4} \leqslant 8 \ln \frac {T ^ {2}}{\delta / 4} + 4 \gamma \left(\beta_ {T, \delta / 4} ^ {\prime}\right) ^ {2}.
$$
By (4), we have  $\mathrm{OPT}(r, c, B - b1) = G(\lambda_{B - b1}^{\star}) \leqslant G(\lambda)$ , for all  $\lambda \geqslant 0$ . Also,
$$
\begin{array}{l} G \left(\boldsymbol {\lambda} _ {B - b 1} ^ {\star}\right) + b \left\| \boldsymbol {\lambda} _ {B - b 1} ^ {\star} \right\| _ {1} \\ = \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \max  _ {a \in \mathcal {A}} \left\{r (\boldsymbol {X}, a) - \left\langle \boldsymbol {c} (\boldsymbol {X}, a) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \right\rangle \right\} \right] + b \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| _ {1} \\ = \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \max  _ {a \in \mathcal {A}} \left\{r (\boldsymbol {X}, a) - \left\langle \boldsymbol {c} (\boldsymbol {X}, a) - \boldsymbol {B}, \lambda_ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \right\rangle \right\} \right] \geqslant \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}), \\ \end{array}
$$
where we used again (4). In particular,
$$
\sum_ {\tau = 1} ^ {t} G \left(\boldsymbol {\lambda} _ {\tau - 1}\right) \geqslant t \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - t b \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| _ {1} \geqslant t \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - t b \sqrt {d} \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| _ {1}.
$$
Collecting all bounds above and using the definition of  $\Upsilon_{T,\delta}$  as  $\max \left\{\beta_{T,\delta /4}^{\prime},\alpha_{T,\delta /4}\right\}$  and the fact that  $dt\leqslant (\Upsilon_{T,\delta})^{2}$ , we proved the following. On  $\mathcal{E}_{\mathrm{H - Az}}\cap \mathcal{E}_{\beta}\cap \mathcal{E}_{\mathrm{Bern - c}}\cap \mathcal{E}_{\mathrm{Bem - r}}$ , which is indeed an event with probability at least  $1 - \delta$ , for all  $1\leqslant t\leqslant T$
$$
\begin{array}{l} \sum_ {\tau = 1} ^ {t} r _ {\tau} \\ \geqslant - 3 \Upsilon_ {T, \delta} - 2 d \gamma t + \sum_ {\tau = 1} ^ {t} G (\boldsymbol {\lambda} _ {\tau - 1}) - \sqrt {2 \sum_ {t = 1} ^ {T} \left(1 + 2 \| \boldsymbol {\lambda} _ {t - 1} \| _ {1}\right) ^ {2} \ln \frac {T ^ {2}}{\delta / 4}} - 8 \ln \frac {T ^ {2}}{\delta / 4} - 4 \gamma \left(\beta_ {T, \delta / 4} ^ {\prime}\right) ^ {2} \\ \geqslant t \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) - \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} \| \left(t b \sqrt {d} + 6 \Upsilon_ {T, \delta}\right) - 3 6 \gamma \sqrt {d} \left(\Upsilon_ {T, \delta}\right) ^ {2} - 8 \ln \frac {T ^ {2}}{\delta / 4}. \\ \end{array}
$$
This entails in particular the stated result.
The proof is divided into three steps: on a favorable event  $\mathcal{E}_{\mathrm{meta}}$  of probability at least  $1 - \delta$ , (i) we bound by  $\operatorname{ilog} T$  the index of the last regime achieved in Box C;
Algorithm 1: Pseudo-code for the Box C strategy  
Input: number of rounds  $T$ ; confidence level  $1 - \delta$ ; margin  $b$  on the average constraints; estimation procedure and error functions  $\varepsilon_{t}$  of Assumption 2; optimistic estimates (2)  
Initialization:  $T_{0} = 1$ ; sequence  $\gamma_{k} = 2^{k} / \sqrt{T}$  of step sizes; sequence  $M_{T,\delta,k} = 4\sqrt{T} + 20\sqrt{d}\Upsilon_{T,\delta/(k+2)^{2}}$  of cost deviations  
for  $k \geqslant 0$  do // Box C part  
 $\lambda_{T_{k-1}} = 0$ ;  
for  $t \geqslant T_{k}$  do // Box B part  
if  $t = T$  then  
| Terminate algorithm;  
end  
Observe the context  $\boldsymbol{x}_{t}$ ;  
Pick an action  
 $a_{t} \in \arg \max_{a \in \mathcal{A}} \left\{ \widehat{r}_{\delta,t-1}^{\mathrm{ucb}}(\boldsymbol{x}_{t}, a) - \langle \widehat{c}_{\delta,t-1}^{\mathrm{lcb}}(\boldsymbol{x}_{t}, a) - (\boldsymbol{B} - b1), \boldsymbol{\lambda}_{t-1} \rangle \right\}$ ;  
Observe the payoff  $r_{t}$  and the costs  $c_{t}$ ;  
Compute  $\lambda_{t} = \left( \lambda_{t-1} + \gamma_{k} \left( \widehat{c}_{\delta,t-1}^{\mathrm{lcb}}(\boldsymbol{x}_{t}, a_{t}) - (\boldsymbol{B} - b1) \right) \right)_{+}$ ; // make PGD update  
Compute the estimates  $\widehat{r}_{\delta,t}^{\mathrm{ucb}}$  and  $\widehat{c}_{\delta,t}^{\mathrm{lcb}}$ ;  
if  $\left\| \left( \sum_{\tau=T_{k}}^{t} c_{\tau} - (t-T_{k}+1)(\boldsymbol{B} - b1) \right)_{+} \right\| > M_{T,\delta,k}$  then // Box C part  
Break inner for loop; // finish phase  $k$  and move to phase  $k+1$ $T_{k+1} = t+1$ ; // record beginning of phase  $k+1$   
end  
end
(ii) we bound by  $(1 + \mathrm{i}\log T)\sqrt{T}$  the excess cumulative costs with respect to  $T(\mathbf{B} - b_{T}\mathbf{1})$  and deduce that the cumulative costs are smaller than  $B$ ;
(iii) we provide a regret bound, by summing the bounds guaranteed by Lemma 1 over each regime.
The favorable event  $\mathcal{E}_{\mathrm{meta}}$  is defined as follows. By construction, and thanks to the assumption of feasibility for  $B - 2b_{T}\mathbf{1}$ , the results of Lemmas 1 and 2 hold with probability at least  $1 - \delta / (k + 2)^{2}$  at each round of each regime  $k \geqslant 0$  that is actually achieved. Given that
$$
\sum_ {k \geqslant 0} \frac {1}{(k + 2) ^ {2}} \leqslant \sum_ {k \geqslant 0} \frac {1}{(k + 1) (k + 2)} = 1,
$$
we may define  $\mathcal{E}_{\mathrm{meta}}$  as the event indicating that the (uniform-in-time) bounds of Lemmas 1 and 2 are satisfied within each of the regimes achieved.
C.1 Step 1: Bounding the number of regimes achieved
We show that, on  $\mathcal{E}_{\mathrm{meta}}$ , if regime  $K = \mathrm{ilog}\| \pmb{\lambda}_{B - b_T\mathbf{1}}^{\star}\|$  is achieved, then this regime does not stop. Hence, on  $\mathcal{E}_{\mathrm{meta}}$ , at most  $K + 1\leqslant 1 + \mathrm{ilog}\| \pmb{\lambda}_{B - b_T\mathbf{1}}^{\star}\|$  regimes take place.
Indeed, in regime  $K = \mathrm{ilog}\| \pmb{\lambda}_{B - b_T\mathbf{1}}^{\star}\|$ , if it is achieved, the meta-strategy resorts to the Box B strategy with
$$
\gamma_ {K} = \frac {2 ^ {K}}{\sqrt {T}} \geqslant \frac {\operatorname* {m a x} \left\{\left\| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \right\| , 1 \right\}}{\sqrt {T}}.
$$
Therefore, the bound of Lemma 1 entails that on  $\mathcal{E}_{\mathrm{meta}}$ , for all  $t \geqslant T_K$
$$
\begin{array}{l} \left\| \left(\sum_ {\tau = T _ {K}} ^ {t} \boldsymbol {c} _ {\tau} - (t - T _ {k} + 1) (\boldsymbol {B} - b _ {T} \mathbf {1})\right) _ {+} \right\| \leqslant \frac {1 + 3 \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b _ {T} \mathbf {1}} ^ {\star} \|}{\gamma_ {K}} + 2 0 \sqrt {d} \Upsilon_ {T, \delta / (K + 2) ^ {2}} \\ \leqslant 4 \sqrt {T} + 2 0 \sqrt {d} \Upsilon_ {T, \delta / (K + 2) ^ {2}} = M _ {T, \delta , K}. \\ \end{array}
$$
This is exactly the contrary of the stopping condition of regime  $K$ : the latter thus cannot be broken. In some bounds, we will further bound  $\mathrm{ilog}\left\| \boldsymbol{\lambda}_{B - b_T\mathbf{1}}^{\star} \right\|$  by  $\mathrm{ilog}T$ : this holds because the assumption of  $(B - 2b_{T}\mathbf{1})$ -feasibility entails, by Lemma 3 and as OPT is always smaller than 1, the crude bound
$$
\| \pmb {\lambda} _ {\pmb {B} - b _ {T} \mathbf {1}} ^ {\star} \| \leqslant \frac {\mathrm {O P T} (r , \pmb {c} , \pmb {B} - b _ {T} \mathbf {1}) - \mathrm {O P T} (r , \pmb {c} , \pmb {B} - (3 / 2) b _ {T} \mathbf {1})}{b _ {T} / 2} \leqslant \frac {2}{b _ {T}} \leqslant \frac {\sqrt {T}}{7} \leqslant T,
$$
where we used that  $b_{T} \geqslant 14 / \sqrt{T}$  given its definition. (Of course, sharper but more complex bounds could be obtained; however, they would only improve logarithmic terms in the bound, which we do not try to optimize anyway.)
C.2 Step 2: Bounding the cumulative costs
We still denote by  $K$  the index of the last regime and recall that  $K \leqslant \mathrm{i}\log\|\pmb{\lambda}_{\pmb{B} - b_T\mathbf{1}}^{\star}\| \leqslant \mathrm{i}\log(T)$ , and that for  $k \geqslant 0$ , regime  $k$  starts at  $T_k$  and stops at  $T_{k+1} - 1$ . By convention,  $T_0 = 1$  and  $T_{K+1} = T + 1$ . By the very definition of the stopping condition of regime  $k \geqslant 0$ ,
$$
\left\| \left(\sum_ {t = T _ {k}} ^ {T _ {k + 1} - 2} \boldsymbol {c} _ {t} - \left(T _ {k + 1} - T _ {k} - 1\right) (\boldsymbol {B} - b _ {T} \mathbf {1})\right) _ {+} \right\| \leqslant M _ {T, \delta , k}.
$$
For rounds of the form  $t = T_{k + 1} - 1$ , we bound the Euclidean norm of  $c_{t} - (B - b_{T}\mathbf{1})$  by  $2\sqrt{d}$ . Therefore, by a triangle inequality, satisfied both by the non-negative part  $(\cdot)_+$  and the norm  $\| \cdot \|$  functions, we have
$$
\begin{array}{l} \left\| \left(\sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} - T (\boldsymbol {B} - b _ {T} \mathbf {1})\right) _ {+} \right\| \\ \leqslant \sum_ {k = 0} ^ {K} \left\| \left(\sum_ {t = T _ {k}} ^ {T _ {k + 1} - 2} \boldsymbol {c} _ {t} - (T _ {k + 1} - T _ {k} - 1) (\boldsymbol {B} - b _ {T} \mathbf {1})\right) _ {+} \right\| + \sum_ {k = 0} ^ {K} \left\| \boldsymbol {c} _ {T _ {k + 1} - 1} - (\boldsymbol {B} - b _ {T} \mathbf {1}) \right\| \\ \leqslant (K + 1) \left(M _ {T, \delta , \operatorname {i l o g} T} + 2 \sqrt {d}\right) \leqslant T b _ {T}, \\ \end{array}
$$
where we used the fact that  $M_{T,\delta,k}$  increases with  $k$ , the bound  $K \leqslant \mathrm{i}\log T$  proved in Step 1 and holding on the event  $\mathcal{E}_{\mathrm{meta}}$ , as well as the definition of  $b_{T}$ . Therefore, on  $\mathcal{E}_{\mathrm{meta}}$ , no component of
$$
\left(\sum_ {t = 1} ^ {T} \boldsymbol {c} _ {t} - T (\boldsymbol {B} - b _ {T} \mathbf {1})\right) _ {+}
$$