SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sup  _ {c _ {t} \in [ 0, 1 ] ^ {K}} \mathbb {E} _ {\hat {y} _ {t}, \hat {c} _ {t}} \left[ c _ {t} (\hat {y} _ {t}) + \operatorname {R E L} (I _ {1: t}) \right] \leq \gamma + \sup  _ {c _ {t} \in [ 0, 1 ] ^ {K}} \mathbb {E} _ {\hat {y} _ {t}, \hat {c} _ {t}} \left[ \langle q _ {t} ^ {*}, \hat {c} _ {t} \rangle + \operatorname {R E L} (I _ {1: t}) \right].
$$
By expansion of the second term on the right hand side, we rewrite the relation as
$$
\begin{array}{l} \sup  _ {c _ {t} \in [ 0, 1 ] ^ {K}} \mathbb {E} _ {\hat {y} _ {t}, \hat {c} _ {t}} \left[ \left\langle q _ {t} ^ {*}, \hat {c} _ {t} \right\rangle + \operatorname {R E L} \left(I _ {1: t}\right) \right] \\ = \sup  _ {c _ {t} \in [ 0, 1 ] ^ {K}} \mathbb {E} _ {\hat {y} _ {t}, \hat {c} _ {t}} \left[ \langle q _ {t} ^ {*}, \hat {c} _ {t} \rangle + \mathbb {E} _ {\rho_ {t}} \left[ R ((x, \hat {c}) _ {1: t}, \rho_ {t}) \right] \right] \\ = \sup  _ {c _ {t} \in [ 0, 1 ] ^ {K}} \mathbb {E} _ {\hat {y} _ {t}, \hat {c} _ {t}} \left[ \langle q _ {t} ^ {*}, \hat {c} _ {t} \rangle + \mathbb {E} _ {\rho_ {t}} \left[ \sup  _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right] \right] + \gamma (T - t) \\ = \sup  _ {c _ {t} \in [ 0, 1 ] ^ {K}} \mathbb {E} _ {\hat {y} _ {t}, \hat {c} _ {t}} \left[ \mathbb {E} _ {\rho_ {t}} \left[ \langle q _ {t} ^ {*} (\rho_ {t}), \hat {c} _ {t} \rangle + \sup  _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right] \right] + \gamma (T - t). \\ \end{array}
$$
Furthermore, the symmetric construction of  $\hat{c}_t$  implies that  $\hat{c}_t$  takes the value  $K e_i / \gamma$  with probability at most  $\gamma / K$ . Therefore, we know that  $\hat{c}_t$  is sampled from a  $p_t \in \Delta_D'$  where  $\Delta_D'$  is defined as in Equation (5). Taking the maximum over all possible  $p_t$ , we bound the supremum term above with
$$
\begin{array}{l} \sup _ {c _ {t} \in [ 0, 1 ] ^ {K}} \mathbb {E} _ {\hat {y} _ {t}, \hat {c} _ {t}} \left[ \mathbb {E} _ {\rho_ {t}} \left[ \langle q _ {t} ^ {*} (\rho_ {t}), \hat {c} _ {t} \rangle + \sup _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right] \right] \\ \leq \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}} \left[ \mathbb {E} _ {\rho_ {t}} \left[ \left\langle q _ {t} ^ {*} (\rho_ {t}), \hat {c} _ {t} \right\rangle + \sup  _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right] \right] \\ \leq \mathbb {E} _ {\rho_ {t}} \left[ \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}} \left[ \langle q _ {t} ^ {*} (\rho_ {t}), \hat {c} _ {t} \rangle + \sup  _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right] \right]. \\ \end{array}
$$
where the second inequality is an application of Jensen's inequality. We first replace the optimized  $q_{t}^{*}$  by the corresponding infimum operator over  $q_{t}$ , thus the term inside the expectation is equivalent to
$$
\inf _ {q \in \Delta_ {K}} \sup _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}} \left[ \langle q, \hat {c} _ {t} \rangle + \sup _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right].
$$
conditioned on  $\rho_{t}$ . By the minimax theorem, we can interchange the infimum and supremum operators without decreasing the quantity, and this is further upper bounded as
$$
\begin{array}{l} \inf _ {q \in \Delta_ {K}} \sup _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}} \left[ \langle q, \hat {c} _ {t} \rangle + \sup _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right] \\ = \sup _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \inf _ {q \in \Delta_ {K}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}} \left[ \langle q, \hat {c} _ {t} \rangle + \sup _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right]. \\ \end{array}
$$
and moreover, since the objective is linear with respect to  $q_{t}$  we can instead work with
$$
\sup _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \min _ {i \in [ K ]} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}} \left[ \hat {c} _ {t} (i) + \sup _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right].
$$
Symmetrization permits us to rewrite the above as
$$
\begin{array}{l} \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \min  _ {i \in [ K ]} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}} \left[ \hat {c} _ {t} (i) + \sup  _ {\pi \in \Pi} (- \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}) \right] \\ = \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}} \left[ \sup  _ {\pi \in \Pi} \left(\min  _ {i \in [ K ]} \mathbb {E} _ {\hat {c} _ {t} ^ {\prime} \sim p _ {t}} \left[ \hat {c} _ {t} ^ {\prime} (i) \right] - \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] \\ \leq \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}} \left[ \sup  _ {\pi \in \Pi} \left(\mathbb {E} _ {\hat {c} _ {t} ^ {\prime} \sim p _ {t}} \left[ \hat {c} _ {t} ^ {\prime} (\pi (x _ {t})) \right] - \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] \\ \leq \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t}, \hat {c} _ {t} ^ {\prime} \sim p _ {t}} \left[ \sup  _ {\pi \in \Pi} \left(\hat {c} _ {t} ^ {\prime} (\pi (x _ {t})) - \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right]. \\ \end{array}
$$
where the last inequality is an additional application of Jensen's inequality. Since  $\hat{c}_t, \hat{c}_t'$  are sampled from the same distribution, this can be rewritten as
$$
\begin{array}{l} \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t}, \hat {c} _ {t} ^ {\prime} \sim p _ {t}} \left[ \sup  _ {\pi \in \Pi} \left(\hat {c} _ {t} ^ {\prime} (\pi (x _ {t})) - \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] \\ = \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t}, \hat {c} _ {t} ^ {\prime} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(\delta \left(\hat {c} _ {t} ^ {\prime} (\pi (x _ {t})) - \hat {c} _ {t} (\pi (x _ {t}))\right) + A _ {\pi , t}\right) \right] \\ = \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t}, \hat {c} _ {t} ^ {\prime} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(\delta \hat {c} _ {t} ^ {\prime} (\pi (x _ {t})) - \delta \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right]. \\ \end{array}
$$
and further split the term within the inner supremum to obtain
$$
\begin{array}{l} \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t}, \hat {c} _ {t} ^ {\prime} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(\delta \hat {c} _ {t} ^ {\prime} (\pi (x _ {t})) - \delta \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] \\ = \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t}, \hat {c} _ {t} ^ {\prime} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(\delta \hat {c} _ {t} ^ {\prime} (\pi (x _ {t})) + \frac {A _ {\pi , t}}{2} - \delta \hat {c} _ {t} (\pi (x _ {t})) + \frac {A _ {\pi , t}}{2}\right) \right] \\ \leq \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t}, \hat {c} _ {t} ^ {\prime} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(\delta \hat {c} _ {t} ^ {\prime} (\pi (x _ {t})) + \frac {A _ {\pi , t}}{2}\right) + \sup  _ {\pi \in \Pi} \left(- \delta \hat {c} _ {t} (\pi (x _ {t})) + \frac {A _ {\pi , t}}{2}\right) \right] \\ = \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(2 \delta \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right]. \\ \end{array}
$$
D Proof of the final step condition
Lemma 10. The relaxation function (4) satisfies the final step condition (2).
Proof. By setting  $t = T$  in our relaxation  $\mathrm{REL}(I_{1:T})$  and definition of our unbiased estimator  $\hat{c}_t$  of  $c_t$ ,
$$
\begin{array}{l} \mathbb {E} _ {\hat {y} _ {1: T}} \left[ \operatorname {R E L} \left(I _ {1: T}\right) \right] = \mathbb {E} _ {\hat {y} _ {1: T}} \left[ - \sup  _ {\pi \in \Pi} \sum_ {\tau = 1} ^ {T} \hat {c} _ {\tau} \left(\pi \left(x _ {\tau}\right)\right) \right] \\ \geq \sup  _ {\pi \in \Pi} - \mathbb {E} _ {\hat {y} _ {1: T}} \left[ \sum_ {\tau = 1} ^ {T} \hat {c} _ {\tau} (\pi (x _ {\tau})) \right] \\ = \sup  _ {\pi \in \Pi} - \sum_ {\tau = 1} ^ {T} c _ {\tau} (\pi (x _ {\tau})). \\ \end{array}
$$