SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Limitation 1: Large constants. First, we must mention that in the bounds of Lemmas 1 and 2, we did not try to optimize the constants but targeted readability. The resulting values in Theorem 1, namely, the recommended choice of  $b_{T}$  and the closed-form bound (19) on the regret, therefore involve constants that are much larger than needed.
**Limitation 2:** Not capturing possibly fast regret rates in some cases. Second, a careful inspection of our proof scheme for lower bounds shows that it only works in the absence of a null-cost action. When such a null-cost action exists and costs are non-negative, Example 1 shows that costly actions are played at most  $B / \alpha$  times on average. If the null-cost action has also a null reward, the costly actions are the only ones generating some stochasticity, thus, we expect that the  $\sqrt{T}$  factors (coming, among others, from a version of Bernstein's inequality) be replaced by  $\sqrt{B / \alpha}$  factors. As a consequence, in the setting of Example 1, bounds growing much slower than  $\sqrt{T}$  should be achievable, see the discussion in Appendix E.1.
**Limitation 3:**  $\sqrt{T}$  regret not captured in case of softer constraints. For the primal strategy discussed in Appendix F, we could prove a general  $\sqrt{T}$  regret bound in the case where total costs smaller than  $TB + \tilde{\mathcal{O}}(\sqrt{T})$  are allowed. We were unable to do so for our dual strategy.
5 Brief overview of the numerical experiments performed
In Appendix G, we implement a specific version of the motivating example of Section 2.1, as proposed by Chohlas-Wood et al. [2021] in the public repository https://github.com/ stanford-policylab/learning-to-be-fair. Fairness costs together with spendings related to rideshare assistance or transportation vouchers are considered. We report below the performance of the strategies considered only in terms of average rewards and rideshare costs, and for the fairness threshold  $\tau = 10^{-7}$ . The Box B strategies with fixed step sizes may fail to control the budget when  $\gamma$  is too large: we observe this for  $\gamma = 0.01$ . We see overall a tradeoff between larger average rewards and larger average costs. The Box C strategy navigates between three regimes,  $\gamma = 0.01$  to start (regime  $k = 0$ ), then  $\gamma = 0.02$  (regime  $k = 1$ ), and finally  $\gamma = 0.04$  (regime  $k = 2$ ), which it sticks to. It overall adaptively finds a decent tuning of  $\gamma$ .
Evgenii Chzhen, Christophe Giraud, Zhen Li, and Gilles Stoltz have no direct funding to acknowledge other than the salaries paid by their employers, CNRS, Université Paris-Saclay, BNP Paribas, and HEC Paris. They have no competing interests to declare.
A.B. Tsybakov. Introduction to Nonparametric Estimation. Springer Series in Statistics. Springer, 2008.
M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML'03), pages 928-935, 2003.
A Proof of the strong duality (4)
In this section, we explain why the equalities (4) hold when the problem  $(r,c,B^{\prime},\nu)$  is feasible for some  $B^{\prime} < B$ . We restate first these inequalities for the convenience of the reader:
$$
\begin{array}{l} \operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B}) = \sup  _ {\boldsymbol {\pi}: \mathcal {X} \rightarrow \mathcal {P} (\mathcal {A})} \inf  _ {\boldsymbol {\lambda} \geqslant \boldsymbol {0}} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} r (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) + \left\langle \boldsymbol {\lambda}, \boldsymbol {B} - \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right\rangle\right] \\ \stackrel {(\star)} {=} \min  _ {\boldsymbol {\lambda} \geqslant \mathbf {0}} \sup  _ {\boldsymbol {\pi}: \mathcal {X} \to \mathcal {P} (\mathcal {A})} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \pi_ {a} (\boldsymbol {X}) \left(r (\boldsymbol {X}, a) - \left\langle \boldsymbol {c} (\boldsymbol {X}, a) - \boldsymbol {B}, \boldsymbol {\lambda} \right\rangle\right) \right] \\ = \min  _ {\lambda \geqslant 0} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \max  _ {a \in \mathcal {A}} \left\{r (\boldsymbol {X}, a) - \left\langle \boldsymbol {c} (\boldsymbol {X}, a) - \boldsymbol {B}, \lambda \right\rangle \right\} \right]. \\ \end{array}
$$
We deal here with a linear program, with primal variables in functional space  $\mathcal{X}\to \mathbb{R}^{\mathcal{A}}$  and dual variables in the finite-dimensional space  $\mathbb{R}^d$ .
The first and third equalities are straightforward. The first equality holds because in  $\mathrm{OPT}(r, c, B)$ , we want to consider only policies with expected cost smaller than or equal to  $B$ ; the infimum over  $\lambda \geqslant 0$  equals  $-\infty$  for policies with expected costs strictly larger than  $B$  and is achieved at  $\lambda = 0$  otherwise. The third inequality follows from identifying that for a given  $\lambda$ , the best policy may be defined pointwise as the argument of the maximum written in the expectation. Thus, only the middle equality  $(\star)$  deserves a proof. We obtain it by applying a general theorem of strong duality (which requires feasibility for slightly smaller cost constraints).
Statement of a general theorem of strong duality. We restate a result extracted from the monograph by Luenberger [1969]. It relies on the dual functional  $\varphi$ , whose expression we recall below.
Theorem 2 (stated as Theorem 1 in Section 8.6, page 224 in Luenberger, 1969). Let  $f$  be a real-valued convex functional defined on a convex subset  $\Omega$  of a vector space  $X$ , and let  $G$  be a convex mapping of  $X$  into a normed space  $Z$ . Suppose there exists an  $x_{1}$  such that  $G(x_{1}) < \theta$  and that  $\mu_0 = \inf \{f(x):G(x)\leqslant \theta ,x\in \Omega \}$  is finite. Then
$$
\inf_{\substack{G(x)\leqslant \theta \\ x\in \Omega}}f(x) = \max_{z^{\star}\geqslant \theta}\varphi (z^{\star})
$$
and the maximum on the right is achieved by some  $z_0^\star \geqslant \theta$ .
The dual function  $\varphi$  is defined as follows (Luenberger, 1969, pages 215, 223, and 224, which we quote). We denote by  $P$  a positive convex cone  $P \subset Z$ , and let  $P^{\star} = \{z^{\star} \in Z^{\star} : \forall z \in P, \langle z, z^{\star} \rangle \geqslant 0\}$  be its corresponding positive cone in the dual space  $Z^{\star}$ . The dual functional is defined, for each element  $z^{\star} \in P^{\star}$ , as
$$
\varphi (z ^ {\star}) = \inf  _ {x \in \Omega} \left\{f (x) + \langle G (x), z ^ {\star} \rangle \right\}.
$$
Application. To prove the middle equality  $(\star)$ , we apply Theorem 2 with the vector space  $X$  of all functions  $\mathcal{X} \to \mathbb{R}^{\mathcal{A}}$ , its convex subset  $\Omega$  formed by functions  $\mathcal{X} \to \mathcal{P}(\mathcal{A})$ , and note that the function  $f$  is actually linear in our situation:
$$
f (\pi) = - \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} r (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right].
$$
(We take a - sign to match the form of the minimization problem over  $\pi$  in Theorem 2.) We take  $Z = \mathbb{R}^d = Z^\star$  with positive cones  $P = P^\star = [0, +\infty)^d$ . The mapping  $G$  is a linear mapping from  $X$  to  $Z = \mathbb{R}^d$ :
$$
G (\pi) = \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right] - \boldsymbol {B},
$$
and we take  $\theta = 0$ . We have that  $\mu_0 = \mathrm{OPT}(r, c, B)$  is indeed finite. We note that the condition " $(r, c, B', \nu) \text{ is feasible for some } B' < B" \text{ is required to apply the theorem. The result follows from noting that we exactly have, for } z = \lambda \in P^{\star}$ ,
$$
\varphi (\boldsymbol {\lambda}) = \inf  _ {\boldsymbol {\pi}: \mathcal {X} \rightarrow \mathcal {P} (\mathcal {A})} \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ - \sum_ {a \in \mathcal {A}} r (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) + \sum_ {a \in \mathcal {A}} \langle \boldsymbol {c} (\boldsymbol {X}, a) - \boldsymbol {B}, \boldsymbol {\lambda} \rangle \pi_ {a} (\boldsymbol {X}) \right].
$$
B Proofs of Lemmas 1 and 2
In both proofs, we will use the following deviation inequalities, holding on an event  $\mathcal{E}_{\mathrm{H - Az}}\cap \mathcal{E}_{\beta}$  of probability at least  $1 - \delta /2$  , where the event  $\mathcal{E}_{\beta}$  is defined in Assumption 2 with a confidence level  $1 - \delta /4$  and the event  $\mathcal{E}_{\mathrm{H - Az}}$  is defined below: on  $\mathcal{E}_{\mathrm{H - Az}}\cap \mathcal{E}_{\beta}$
$$
\forall 1 \leqslant t \leqslant T, \quad \sum_ {\tau = 1} ^ {t} \boldsymbol {c} _ {\tau} \leqslant \left(\alpha_ {t, \delta / 4} + 2 \beta_ {t, \delta / 4}\right) \mathbf {1} + \sum_ {\tau = 1} ^ {t} \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) \tag {7}
$$
$$
\text {a n d} \quad \sum_ {\tau = 1} ^ {t} r _ {\tau} \geqslant - \left(\alpha_ {t, \delta / 4} + 2 \beta_ {t, \delta / 4}\right) + \sum_ {\tau = 1} ^ {t} \widehat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right), \tag {8}
$$
where  $\beta_{t,\delta /4}$  is also defined in Assumption 2 and
$$
\alpha_ {t, \delta / 4} = \sqrt {2 t \ln \frac {2 (d + 1) T}{\delta / 4}}.
$$
These inequalities come, on the one hand, by the Hoeffding-Azuma inequality applied  $(d + 1)T$  times on the range  $[-1, 1]$ : it ensures that on an event  $\mathcal{E}_{\mathrm{H - Az}}$  with probability at least  $1 - \delta / 4$ , for all  $1 \leqslant t \leqslant T$ ,
$$
\left\| \sum_ {\tau = 1} ^ {t} \boldsymbol {c} _ {\tau} - \sum_ {\tau = 1} ^ {t} \boldsymbol {c} (\boldsymbol {x} _ {\tau}, a _ {\tau}) \right\| _ {\infty} \leqslant \alpha_ {t, \delta / 4} \mathbf {1} \quad \text {a n d} \quad \left| \sum_ {\tau = 1} ^ {t} r _ {\tau} - \sum_ {\tau = 1} ^ {t} r (\boldsymbol {x} _ {\tau}, a _ {\tau}) \right| \leqslant \alpha_ {t, \delta / 4}
$$
(where we denoted by  $\| \cdot \|_{\infty}$  the supremum norm). On the other hand, Assumption 2 and the clipping (3) entail, in particular, that on an event  $\mathcal{E}_{\beta}$  of probability at least  $1 - \delta /4$  , for all  $1\leqslant t\leqslant T$
$$
\sum_ {\tau = 1} ^ {t} \boldsymbol {c} (\boldsymbol {x} _ {\tau}, a _ {\tau}) \leqslant \sum_ {\tau = 1} ^ {t} \left(\widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) + 2 \varepsilon_ {\tau - 1} (\boldsymbol {x} _ {\tau}, a _ {\tau}, \delta / 4) \mathbf {1}\right) = 2 \beta_ {t, \delta / 4} \mathbf {1} + \sum_ {\tau = 1} ^ {t} \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau})
$$
$$
\text {a n d (s i m i l a r l y)} \quad \sum_ {\tau = 1} ^ {t} r \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) \geqslant - 2 \beta_ {t, \delta / 4} + \sum_ {\tau = 1} ^ {t} \widehat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right).
$$
For  $\tau \geqslant 1$ , by definition of  $\lambda_{\tau}$  in Box B,
$$
\begin{array}{l} \boldsymbol {\lambda} _ {\tau} - \boldsymbol {\lambda} _ {\tau - 1} = \left(\boldsymbol {\lambda} _ {\tau - 1} + \gamma \left(\widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - (\boldsymbol {B} - \boldsymbol {b 1})\right)\right) _ {+} - \boldsymbol {\lambda} _ {\tau - 1} \\ \geqslant \gamma \left(\widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) - (\boldsymbol {B} - b \mathbf {1})\right). \\ \end{array}
$$
For  $t \geqslant 1$ , as  $\lambda_0 = 0$  and  $\lambda_t \geqslant 0$ , we get, after telescoping and taking non-negative parts,
$$
\boldsymbol {\lambda} _ {t} \geqslant \gamma \left(\sum_ {\tau = 1} ^ {t} \left(\widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) - (\boldsymbol {B} - b \mathbf {1})\right)\right) _ {+},
$$
$$
\text {t h u s} \quad \left\| \left(\sum_ {\tau = 1} ^ {t} \left(\widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) - (\boldsymbol {B} - b \mathbf {1})\right)\right) _ {+} \right\| \leqslant \frac {\left\| \boldsymbol {\lambda} _ {t} \right\|}{\gamma}. \tag {9}
$$
Up to the deviation terms (7),  $\| \pmb{\lambda}_t\| /\gamma$  bounds how larger the cost constraints till round  $t$  are from  $t(\pmb {B} - b\mathbf{1})$ . Most of the rest of the proof, namely, the Steps 1-4 below, thus focus on upper bounding the  $\| \pmb {\lambda}_t\|$ , while Step 5 will collect all bounds together and conclude.
Step 1: Gradient-descent analysis. We introduce
$$
\Delta \widehat {\boldsymbol {c}} _ {\tau} = \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l b c}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) - (\boldsymbol {B} - b \mathbf {1}) \tag {10}
$$
and prove the following deterministic inequality: for all  $1 \leqslant t \leqslant T$ ,
$$
\forall \boldsymbol {\lambda} \geqslant \mathbf {0}, \quad \| \boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} \| ^ {2} \leqslant \| \boldsymbol {\lambda} \| ^ {2} + 4 d \gamma^ {2} t + 2 \gamma \sum_ {\tau = 1} ^ {t} \left\langle \Delta \widehat {\boldsymbol {c}} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} \right\rangle . \tag {11}
$$
To do so, we proceed as is classical in (projected) gradient-descent analyses; see, e.g., Zinkevich [2003]. Namely, for all  $1 \leqslant \tau \leqslant T$ ,
$$
\begin{array}{l} 2 \left\langle - \Delta \widehat {c} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} \right\rangle = \frac {1}{\gamma} \left(\left\| \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} \right\| ^ {2} + \left\| \gamma \Delta \widehat {c} _ {\tau} \right\| ^ {2} - \left\| \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} + \gamma \Delta \widehat {c} _ {\tau} \right\| ^ {2}\right) \\ = \gamma \| \Delta \hat {\boldsymbol {c}} _ {\tau} \| ^ {2} + \frac {1}{\gamma} \left(\| \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} \| ^ {2} - \| \boldsymbol {\lambda} _ {\tau - 1} + \gamma \Delta \hat {\boldsymbol {c}} _ {\tau} - \boldsymbol {\lambda} \| ^ {2}\right) \\ \leqslant \gamma \left\| \Delta \widehat {\boldsymbol {c}} _ {\tau} \right\| ^ {2} + \frac {1}{\gamma} \left(\| \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} \| ^ {2} - \| \boldsymbol {\lambda} _ {\tau} - \boldsymbol {\lambda} \| ^ {2}\right), \\ \end{array}
$$
where the inequality comes from the definition  $\pmb{\lambda}_{\tau} = \left(\pmb{\lambda}_{\tau -1} + \gamma \Delta \widehat{\pmb{c}}_{\tau}\right)_{+}$  and the fact that
$$
\forall x \in \mathbb {R}, \forall y \geqslant 0, \quad | (x) _ {+} - y | \leqslant | x - y |
$$
(which may be proved by distinguishing the cases  $x \leqslant 0$  and  $x \geqslant 0$ ).
$$
\boldsymbol {B} - b \mathbf {1} \in [ 0, 1 ] ^ {d} \quad \text {a n d} \quad \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) \in [ - 1, 1 ] ^ {d}, \qquad \text {s o t h a t} \qquad \left\| \Delta \widehat {\boldsymbol {c}} _ {\tau} \right\| ^ {2} \leqslant 4 d.
$$
Collecting all bounds above, we get, after summation and telescoping,
$$
\sum_ {\tau = 1} ^ {t} \left\langle - \Delta \widehat {\boldsymbol {c}} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} \right\rangle \leqslant 2 d \gamma t + \frac {1}{2 \gamma} \left(\| \boldsymbol {\lambda} _ {0} - \boldsymbol {\lambda} \| ^ {2} - \| \boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} \| ^ {2}\right).
$$
Rearranging and substituting  $\lambda_0 = 0$  yields the claimed inequality (11).
Step 2: Relating estimated costs (and rewards) to true conditional expectations. In this part, we upper bound the right-hand side of (11) by showing that on the event  $\mathcal{E}_{\beta}$ ,
$$
\forall \lambda \geqslant 0, \quad \forall 1 \leqslant t \leqslant T,
$$
$$
\sum_ {\tau = 1} ^ {t} \left\langle \Delta \widehat {\boldsymbol {c}} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} \right\rangle \leqslant 2 \left(1 + \| \boldsymbol {\lambda} \| _ {1}\right) \beta_ {t, \delta / 4} + \sum_ {\tau = 1} ^ {t} \left(g _ {\tau} (\boldsymbol {\lambda}) - g _ {\tau} \left(\boldsymbol {\lambda} _ {\tau - 1}\right)\right), \tag {12}
$$
$$
\text {w h e r e} \quad g _ {\tau} (\boldsymbol {\lambda}) = \max  _ {a \in \mathcal {A}} \left\{r \left(\boldsymbol {x} _ {\tau}, a\right) - \left\langle \boldsymbol {c} \left(\boldsymbol {x} _ {\tau}, a\right) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} \right\rangle \right\}
$$
and where we recall that  $\| \cdot \| _1$  denotes the  $\ell_1$ -norm.
Adding and subtracting  $\widehat{r}_{\delta /4,\tau -1}^{\mathrm{ucb}}(\pmb{x}_{\tau},a_{\tau})$  , here, we deal with
$$
\begin{array}{l} \sum_ {\tau = 1} ^ {t} \left\langle \Delta \widehat {\boldsymbol {c}} _ {\tau}, \boldsymbol {\lambda} _ {\tau - 1} - \boldsymbol {\lambda} \right\rangle = \sum_ {\tau = 1} ^ {t} \Big (\widehat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - \left\langle \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} \right\rangle \Big) \\ - \sum_ {\tau = 1} ^ {t} \left(\widehat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - \left\langle \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} _ {\tau - 1} \right\rangle\right). \\ \end{array}
$$
Now, for each  $\tau$ , by (3) and the fact that  $\lambda \geqslant 0$
$$
\begin{array}{l} \widehat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - \left\langle \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} (\boldsymbol {x} _ {\tau}, a _ {\tau}) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} \right\rangle \\ \leqslant r \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) + 2 \varepsilon_ {\tau - 1} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}, \delta / 4\right) - \left\langle \boldsymbol {c} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} \right\rangle + 2 \varepsilon_ {\tau - 1} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}, \delta / 4\right) \| \boldsymbol {\lambda} \| _ {1} \\ \leqslant g _ {\tau} (\boldsymbol {\lambda}) + 2 \left(1 + \| \boldsymbol {\lambda} \| _ {1}\right) \varepsilon_ {\tau - 1} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}, \delta / 4\right). \\ \end{array}
$$
On the other hand, by definition of  $a_{\tau}$  for the equality, and then by the other inequalities in (3), for each  $\tau$ , and the fact that  $\lambda_{\tau - 1} \geqslant 0$ ,
$$
\begin{array}{l} \widehat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) - \left\langle \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} \left(\boldsymbol {x} _ {\tau}, a _ {\tau}\right) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} _ {\tau - 1} \right\rangle \tag {13} \\ = \max  _ {a \in \mathcal {A}} \left\{\widehat {r} _ {\delta / 4, \tau - 1} ^ {\mathrm {u c b}} \left(\boldsymbol {x} _ {\tau}, a\right) - \left\langle \widehat {\boldsymbol {c}} _ {\delta / 4, \tau - 1} ^ {\mathrm {l c b}} \left(\boldsymbol {x} _ {\tau}, a\right) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} _ {\tau - 1} \right\rangle \right\} \\ \geqslant \max  _ {a \in \mathcal {A}} \left\{r \left(\boldsymbol {x} _ {\tau}, a\right) - \left\langle \boldsymbol {c} \left(\boldsymbol {x} _ {\tau}, a\right) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} _ {\tau - 1} \right\rangle \right\} = g _ {\tau} \left(\boldsymbol {\lambda} _ {\tau - 1}\right). \\ \end{array}
$$
Collecting the two series of bounds concludes this part.
Step 3: Application of a Bernstein-Freedman inequality. We recall that we denoted by  $\lambda_{B - b1}^{\star}$  the optimal dual variable in (4) for  $\mathrm{OPT}(r, c, B - b1)$ ; it exists because we assumed feasibility of a problem with average cost constraints  $B' < B - b1$ .
We now upper bound the sum appearing in the right hand side of (12) at  $\lambda = \lambda_{B - b1}^{\star}$  by showing that on an event  $\mathcal{E}_{\mathrm{Bern - c}}$  of probability at least  $1 - \delta /4$ , for all  $1\leqslant t\leqslant T$
$$
\sum_ {\tau = 1} ^ {t} \left(g _ {\tau} \left(\boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star}\right) - g _ {\tau} \left(\boldsymbol {\lambda} _ {\tau - 1}\right)\right) \leqslant \left(1 + 2 \Lambda_ {t}\right) \sqrt {2 t \ln \frac {T ^ {2}}{\delta / 4}} + 2 K _ {t} \ln \frac {T ^ {2}}{\delta / 4}, \tag {14}
$$
where  $\Lambda_{t} = \max_{1\leqslant \tau \leqslant t}\| \pmb{\lambda}_{B - b\mathbf{1}}^{\star} - \pmb{\lambda}_{\tau -1}\|_{1}$  and  $K_{t} = 4\big(1 + \| \pmb{\lambda}_{B - b\mathbf{1}}^{\star}\|_{1} + 2d\gamma t\big)$ .
We will do so by applying a version of the Bernstein-Freedman inequality for martingales stated in Cesa-Bianchi et al. [2005, Corollary 16] involving the sum of the conditional variances (and not only a deterministic bound thereon); it is obtained via peeling based on the classic version of Bernstein's inequality (Freedman, 1975). We restate it here for the convenience of the reader (after applying some simple boundings).
Lemma 4 (a version of the Bernstein-Freedman inequality by Cesa-Bianchi et al., 2005). Let  $X_{1}, X_{2}, \ldots$  be a martingale difference with respect to the filtration  $\mathcal{F} = (\mathcal{F}_s)_{s \geqslant 0}$  and with increments bounded in absolute values by  $K$ . For all  $t \geqslant 1$ , let
$$
\mathfrak {S} _ {t} = \sum_ {\tau = 1} ^ {t} \mathbb {E} \left[ X _ {\tau} ^ {2} \mid \mathcal {F} _ {\tau - 1} \right]
$$
denote the sum of the conditional variances of the first  $t$  increments. Then, for all  $\delta \in (0,1)$  and all  $t \geqslant 1$ , with probability at least  $1 - \delta$ ,
$$
\sum_ {\tau = 1} ^ {t} X _ {\tau} \leqslant \sqrt {2 \mathfrak {S} _ {t} \ln \frac {t}{\delta}} + 2 K \ln \frac {t}{\delta}.
$$
We introduce, for all  $\lambda \geqslant 0$ , the common expectation of the  $g_{\tau}(\lambda)$ , namely,
$$
G (\boldsymbol {\lambda}) = \mathbb {E} \left[ g _ {\tau} (\boldsymbol {\lambda}) \right] = \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \max  _ {a \in \mathcal {A}} \left\{r (\boldsymbol {X}, a) - \left\langle \boldsymbol {c} (\boldsymbol {X}, a) - (\boldsymbol {B} - b \mathbf {1}), \boldsymbol {\lambda} \right\rangle \right\} \right],
$$
and consider the martingale increments
$$
X _ {\tau} = \left(g _ {\tau} \left(\boldsymbol {\lambda} _ {\boldsymbol {B} - b 1} ^ {\star}\right) - g _ {\tau} \left(\boldsymbol {\lambda} _ {\tau - 1}\right)\right) - \left(G \left(\boldsymbol {\lambda} _ {\boldsymbol {B} - b 1} ^ {\star}\right) - G \left(\boldsymbol {\lambda} _ {\tau - 1}\right)\right).
$$
As  $B - b\mathbf{1} \in [0,1]^d$  and  $c$  takes values in  $[-1,1]^d$ , for all  $x \in \mathcal{X}$ , all  $a \in \mathcal{A}$ , and all  $v \in \mathbb{R}^d$ , the quantities  $r(\boldsymbol{x},a) - \langle c(\boldsymbol{x},a) - (B - b\mathbf{1}), v \rangle$  take absolute values smaller than  $1 + 2\|v\|_1$ . Using that a difference of maxima is smaller than the maximum of the differences, we get, in particular,
$$
\left| g _ {\tau} \left(\boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star}\right) - g _ {\tau} \left(\boldsymbol {\lambda} _ {\tau - 1}\right) \right| \leqslant 1 + 2 \| \boldsymbol {\lambda} _ {\boldsymbol {B} - b \mathbf {1}} ^ {\star} - \boldsymbol {\lambda} _ {\tau - 1} \| _ {1} \quad \text {a . s .} \tag {15}
$$