SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As mentioned in the introduction, our main novelty is the use of a new relaxation function, which we discussed in Section 3, that uses less variance in the hallucinated cost vectors. Our initial result verifies that the our novel relaxation is indeed admissible and, as a result, we can leverage the prior work demonstrating the expected regret of these algorithms.
Theorem 5. The relaxation function defined in (4), and the corresponding strategy (7) are admissible (Definition 2).
Theorem 5 contrasts with existing admissible relaxations in that it only uses a single Rademacher variable for each time step, while prior work – Lemma 2 in Rakhlin and Sridharan (2015), Theorem 2 in Rakhlin and Sridharan (2016) and Theorem 3 in Syrgkanis et al. (2016b) – all use  $k$  independent Rademacher variables. To our knowledge, this is the first work in which the number of Rademacher variables used in the relaxation does not grow with the number of arms. As we discuss below, this
allows us to reduce the variance of the hallucinated costs, leading to a better regret bound. The proof of Theorem 5 is provided in Section 4.1, and is based on a novel symmetrization step (Lemma 9), which may be of independent interest.
As highlighted in Section 2.2, admissible relaxations are a powerful framework for upper bounding the expected regret in online learning through Lemma 3 and the value of  $\mathrm{REL}(\emptyset)$ . Formally, Lemma 3 implies
$$
\mathbb {E} [ \operatorname {R E G} ] \leq \operatorname {R E L} (\emptyset) = \gamma T + \mathbb {E} _ {\rho_ {0}} \left[ \sup  _ {\pi \in \Pi} \left(\sum_ {\tau = 1} ^ {T} 2 Z _ {\tau} \varepsilon_ {\tau} \left(\pi \left(x _ {\tau}\right)\right)\right) \right]. \tag {8}
$$
In order to bound the regret of our algorithm, it suffices to bound the Rademacher averages term above, which we formally do in the following Theorem.
Theorem 6. For any  $\gamma >\frac{K}{T}\log (|\Pi |) / 2$  , the following holds:
$$
\mathbb {E} _ {(Z, \varepsilon) _ {1: T}} \left[ \sup  _ {\pi \in \Pi} \sum_ {i = 1} ^ {T} Z _ {t} \varepsilon_ {t} (\pi (x _ {t})) \right] \leq 2 \sqrt {\frac {K T \log | \Pi |}{\gamma}}.
$$
The above theorem can be thought of as an improved version of Lemma 2 from Syrgkanis et al. (2016b), where we improve by a factor of  $K$ . Our improvement comes from the use of the new Rademacher vectors that only contain a single non-zero coordinate, together with a more refined analysis. We refer to Section 4.2 for a formal proof of the result.
Combining Lemma 4, Equation (8), and Theorem 6, we obtain the main result of our paper which we state here.
Theorem 7. The contextual bandits algorithm implemented in Algorithm 1 has expected regret upper bounded by
$$
4 \sqrt {\frac {T K \log (| \Pi |)}{\gamma}} + \gamma T,
$$
for any  $\frac{K\log|\Pi|}{2T} < \gamma \leq 1$ , which implies the regret order of  $O((K\log |\Pi|)^{1/3}T^{2/3})$  when  $T > 4K\log (|\Pi|)$ . Furthermore, the Algorithm makes at most  $K + 1$  calls to a value-of-ERM oracle in each round.
We refer to Appendix A for the proof of this result.
4.1 Proof of Theorem 5
In order to prove Theorem 5, we need to verify the final step condition (2), and show that the  $q_{t}$  defined in Equation (7) certifies the condition (1), i.e.,
$$
\mathbb {E} _ {x _ {t}} \left[ \sup  _ {c _ {t}} \mathbb {E} _ {\hat {y} _ {t}, S _ {t}} \left[ c _ {t} \left(\hat {y} _ {t}\right) + \operatorname {R E L} \left(I _ {1: t}\right) \right] \right] \leq \operatorname {R E L} \left(I _ {1: t - 1}\right), \tag {9}
$$
where  $\hat{y}_t$  is sampled from  $q_t$  and  $I_{1:t}$  denotes  $(I_{1:t-1}, I_t(x_t, q_t, \hat{y}_t, c_t, S_t))$ . Verifying condition (2) is standard and we do this in Appendix D. It remains to prove Equation (9). Since most admissibility proofs in the literature (Rakhlin et al., 2012; Rakhlin and Sridharan, 2015, 2016; Syrgkanis et al., 2016b) follow the framework of the original derivation of Rakhlin et al. (2012), in order to emphasize our novelty, we divide the proof into two parts. The first part (Lemma 8) is based on existing techniques (in particular, the proof of Theorem 3 in Syrgkanis et al. (2016b)) and its proof is provided in Appendix C. The second part (Lemma 9) uses new techniques and we present its proof here.
Lemma 8. For any  $t \in [T]$ , define  $A_{\pi, t}$  and  $C_t$  as
$$
A _ {\pi , t} := - \sum_ {\tau = 1} ^ {t - 1} \hat {c} _ {\tau} (\pi (x _ {\tau})) - \sum_ {\tau = t + 1} ^ {T} 2 Z _ {\tau} \varepsilon_ {\tau} (\pi (x _ {\tau})), \quad C _ {t} := \gamma (T - t + 1).
$$
Letting  $\delta$  denote a Rademacher random variable independent of  $\rho_t$  and  $\hat{c}_t$ , the following holds for any value of  $x_t$ :
$$
\sup  _ {c _ {t}} \mathbb {E} _ {\hat {y} _ {t}, S _ {t}} \left[ c _ {t} (\hat {y} _ {t}) + \operatorname {R E L} (I _ {1: t}) \right] \leq \mathbb {E} _ {\rho_ {t}} \left[ \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(2 \delta \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] \right] + C _ {t}.
$$
Lemma 9. Defining  $A_{\pi ,t}$  as in Lemma 8, the following bound holds for any  $t\in [T]$ :
$$
\sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(2 \delta \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] \leq \mathbb {E} _ {\varepsilon_ {t}, Z _ {t}} \left[ \sup  _ {\pi \in \Pi} \left(2 Z _ {t} \cdot \varepsilon_ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right]. \tag {10}
$$
Combining the above lemmas we obtain Equation (9) by definition of REL:
$$
\begin{array}{l} \mathbb {E} _ {x _ {t}} \left[ \sup  _ {c _ {t}} \mathbb {E} _ {\hat {y} _ {t}, S _ {t}} \left[ c _ {t} (\hat {y} _ {t}) + \operatorname {R E L} (I _ {1: t}) \right] \right] \leq \mathbb {E} _ {x _ {t}, \rho_ {t}} \left[ \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(2 \delta \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] \right] + C _ {t} \\ \leq \mathbb {E} _ {x _ {t}, \varepsilon_ {t}, Z _ {t}, \rho_ {t}} \left[ \sup  _ {\pi \in \Pi} \left(2 Z _ {t} \cdot \varepsilon_ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] + C _ {t} \\ \leq \mathbb {E} _ {\rho_ {t - 1}} \left[ \sup  _ {\pi \in \Pi} \left(2 Z _ {t} \cdot \varepsilon_ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] + C _ {t} \\ \leq \operatorname {R E L} \left(I _ {1: t - 1}\right). \\ \end{array}
$$
Proof of Lemma 9. For any distribution  $p_t \in \Delta_D'$ , the distribution of the each coordinate of  $\hat{c}_t$  has support on  $\{0, \gamma^{-1}K\}$  and is equal to  $\gamma^{-1}K$  with probability at most  $\gamma / K$ . Using  $p_t(i)$  to denote  $\mathbb{P}\left[\hat{c}_t(i) = \gamma^{-1}K \cdot \mathbf{e}_i\right]$  we can rewrite the LHS (left hand side) of Equation (10) as
$$
\begin{array}{l} \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(2 \delta \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] \\ = \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \left((1 - \sum_ {i} p _ {t} (i)) \sup  _ {\pi} A _ {\pi , t} + \sum_ {i} p _ {t} (i) \mathbb {E} _ {\delta} \left[ \sup  _ {\pi} A _ {\pi , t} + \frac {2 K \delta}{\gamma} \mathbb {1} [ \pi (x _ {t}) = i ] \right]\right) \\ = \sup  _ {0 \leq p _ {t} (i) \leq \gamma / K} \left((1 - \sum_ {i} p _ {t} (i)) \sup  _ {\pi} A _ {\pi , t} + \sum_ {i} p _ {t} (i) \mathbb {E} _ {\delta} \left[ \sup  _ {\pi} A _ {\pi , t} + \frac {2 K \delta}{\gamma} \mathbb {1} [ \pi (x _ {t}) = i ] \right]\right), \\ \end{array}
$$
where the first equality follows from expanding the expectation with respect to  $\hat{c}_t$ , and the second equality follows from the definition of  $\Delta_D'$ . We argue that this value is maximized when each  $p_t(i)$  takes on its maximum value, i.e.,  $\gamma / K$ . It suffices to observe that
$$
\mathbb {E} _ {\delta} \left[ \sup  _ {\pi} A _ {\pi , t} + \frac {2 K \delta}{\gamma} \mathbb {1} [ \pi (x _ {t}) = i ] \right] \geq \sup  _ {\pi} \left(A _ {\pi , t} + \mathbb {E} _ {\delta} \left[ \frac {2 K \delta}{\gamma} \mathbb {1} [ \pi (x _ {t}) = i ] \right]\right) = \sup  _ {\pi} A _ {\pi , t},
$$
where the inequality follows from the fact that supremum of expectation is less than expectation of supremum, and the equality uses the fact that  $\mathbb{E}[\delta] = 0$ . Therefore, we maximize the LHS of Equation (10) via selecting  $p_t$  that satisfies  $p_t(i) = \frac{\gamma}{K}$  for  $i \geq 1$ . It follows that
$$
\begin{array}{l} \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \mathbb {E} _ {\hat {c} _ {t} \sim p _ {t}, \delta} \left[ \sup  _ {\pi \in \Pi} \left(2 \delta \hat {c} _ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right] \\ \leq \left((1 - \gamma) \sup  _ {\pi} A _ {\pi , t} + \sum_ {i} \frac {\gamma}{K} \mathbb {E} _ {\delta} \left[ \sup  _ {\pi} A _ {\pi , t} + \frac {2 K \delta}{\gamma} \mathbb {1} [ \pi (x _ {t}) = i ] \right]\right) \\ = \mathbb {E} _ {\varepsilon_ {t}, Z _ {t}} \left[ \sup  _ {\pi \in \Pi} \left(2 Z _ {t} \cdot \varepsilon_ {t} (\pi (x _ {t})) + A _ {\pi , t}\right) \right], \\ \end{array}
$$
finishing the proof.
4.2 Proof of Theorem 6
We start with the following standard inequalities for handling the supremum using the moment generating function.
$$
\begin{array}{l} \mathbb {E} _ {(Z, \varepsilon) _ {1: T}} \left[ \sup  _ {\pi \in \Pi} \sum_ {i = 1} ^ {T} Z _ {t} \varepsilon_ {t} (\pi (x _ {t})) \right] = \mathbb {E} _ {Z _ {1: T}} \left[ \frac {1}{\lambda} \cdot \mathbb {E} _ {\varepsilon_ {1: T}} \left[ \log \left(\sup  _ {\pi \in \Pi} e ^ {\lambda \sum_ {t = 1} ^ {T} Z _ {t} \varepsilon_ {t} (\pi (x _ {t}))}\right) \right] \right] \\ \leq \mathbb {E} _ {Z _ {1: T}} \left[ \frac {1}{\lambda} \cdot \mathbb {E} _ {\varepsilon_ {1: T}} \left[ \log \left(\sum_ {\pi \in \Pi} e ^ {\lambda \sum_ {t = 1} ^ {T} Z _ {t} \varepsilon_ {t} (\pi (x _ {t}))}\right) \right] \right] \\ \stackrel {(i)} {\leq} \mathbb {E} _ {Z _ {1: T}} \left[ \frac {1}{\lambda} \cdot \log \left(\mathbb {E} _ {\varepsilon_ {1: T}} \left[ \sum_ {\pi \in \Pi} e ^ {\lambda \sum_ {t = 1} ^ {T} Z _ {t} \varepsilon_ {t} (\pi (x _ {t}))}\right)\right) \right] \\ \stackrel {(i i)} {=} \mathbb {E} _ {Z _ {1: T}} \left[ \frac {1}{\lambda} \cdot \log \left(\sum_ {\pi \in \Pi} \prod_ {t = 1} ^ {T} \mathbb {E} _ {\varepsilon_ {t}} \left[ e ^ {\lambda Z _ {t} \varepsilon_ {t} (\pi (x _ {t}))} \right]\right) \right]. \\ \end{array}
$$
Inequality (i) holds due to the concavity of log and (ii) follows from the independence of  $\varepsilon_{t}$ . We will additionally assume that  $\lambda$  is upper bounded by  $\gamma \sqrt{2} / K$  in the remaining analysis.
By our construction of the random variable  $\varepsilon_{t}$ , for any fixed  $\pi$ ,  $\varepsilon_{t}(\pi(x_{t}))$  takes the value 0 with probability  $1 - \frac{1}{K}$  and the values  $-1$  and 1 each with probability  $\frac{1}{2K}$ . We therefore have that
$$
\begin{array}{l} \mathbb {E} _ {\varepsilon_ {t}} \left[ e ^ {\lambda Z _ {t} \varepsilon_ {t} (\pi (x _ {t}))} \right] = \left(1 - \frac {1}{K}\right) + \frac {1}{2 K} \cdot e ^ {\lambda Z _ {t}} + \frac {1}{2 K} \cdot e ^ {- \lambda Z _ {t}} \\ \leq 1 - \frac {1}{K} + \frac {1}{K} \cdot e ^ {\lambda^ {2} Z _ {t} ^ {2} / 2} \\ \leq e ^ {\frac {1}{K} \left(e ^ {\lambda^ {2} Z _ {t} ^ {2} / 2} - 1\right)}. \\ \end{array}
$$
The first inequality above uses  $e^x + e^{-x} \leq 2e^{x^2 / 2}$  while the second inequality uses  $e^x \geq 1 + x$  for  $x \in \mathbb{R}$ . This further yields
$$
\begin{array}{l} \mathbb {E} _ {Z _ {1: T}} \left[ \frac {1}{\lambda} \cdot \log \left(\sum_ {\pi \in \Pi} \prod_ {t = 1} ^ {T} \mathbb {E} _ {\varepsilon_ {t}} \left[ e ^ {\lambda Z _ {t} \varepsilon_ {t} (\pi (x _ {t}))} \right]\right) \right] \leq \mathbb {E} _ {Z _ {1: T}} \left[ \frac {1}{\lambda} \cdot \log \left(| \Pi | \cdot \prod_ {t = 1} ^ {T} e ^ {\frac {e ^ {\lambda^ {2} Z _ {t} ^ {2} / 2 - 1}}{K}}\right) \right] \\ = \mathbb {E} _ {Z _ {1: T}} \left[ \frac {1}{\lambda} \log (| \Pi |) + \sum_ {t = 1} ^ {T} \frac {e ^ {\lambda^ {2} Z _ {t} ^ {2} / 2} - 1}{\lambda K} \right] \\ = \frac {\log (| \Pi |)}{\lambda} + \frac {1}{\lambda K} \sum_ {t = 1} ^ {T} \mathbb {E} _ {Z _ {1: T}} \left[ e ^ {\lambda^ {2} Z _ {t} ^ {2} / 2} - 1 \right]. \\ \end{array}
$$
Recall that  $Z_{t}$  takes the values 0 and  $\frac{K}{\gamma}$  with probabilities  $1 - \gamma$  and  $\gamma$  respectively. It follows that
$$
\mathbb {E} _ {Z _ {1: T}} \left[ e ^ {\lambda^ {2} Z _ {t} ^ {2} / 2} - 1 \right] = \gamma \left(e ^ {\frac {\lambda^ {2} K ^ {2}}{2 \gamma^ {2}}} - 1\right) \leq \gamma \frac {\lambda^ {2} K ^ {2}}{\gamma^ {2}},
$$
where the inequality follows from the fact that  $e^x - 1 \leq 2x$  for  $x \in (0,1)$  and the assumption  $\lambda \leq \gamma \sqrt{2} / K$ . Therefore,
$$
\mathbb {E} _ {Z _ {1: T}} \left[ \frac {1}{\lambda} \cdot \log \left(\sum_ {\pi \in \Pi} \prod_ {t = 1} ^ {T} \mathbb {E} _ {\varepsilon_ {t}} \left[ e ^ {\lambda Z _ {t} \varepsilon_ {t} (\pi (x _ {t}))} \right]\right) \right] \leq \frac {\log (| \Pi |)}{\lambda} + \frac {T K \lambda}{\gamma}.
$$
By taking derivative with respect to  $\lambda$  we obtain
$$
\frac {- \log (| \Pi |)}{\lambda^ {2}} + T K / \gamma = 0,
$$
and compute that the equation above is minimized at  $\lambda = \sqrt{\frac{\gamma\log(|\Pi|)}{TK}}$ . We note that  $\lambda$  satisfies the assumption  $\lambda \leq \gamma \sqrt{2} /K$  because this is equivalent to  $\frac{\log(|\Pi|)}{2T} < \frac{\gamma}{K}$ , which holds by the assumption of the lemma. Plugging this again yields
$$
\log (| \Pi |) \sqrt {\frac {T K}{\gamma \log (| \Pi |)}} + \frac {T K}{\gamma} \sqrt {\frac {\gamma \log (| \Pi |)}{T K}} = 2 \sqrt {T K \log (| \Pi |) / \gamma},
$$
which is the desired bound.
In this paper, we presented a novel efficient relaxation for the adversarial contextual bandits problem and proved that its regret is upper bounded by  $O(T^{2/3}(K\log |\Pi|)^{1/3})$ . This provides a marked improvement with respect to the parameter  $K$  as compared to the prior best result and matches the original baseline of Langford and Zhang (2007) for the stochastic version of the problem. As mentioned earlier, non-efficient algorithms can obtain a regret bound of  $O(\sqrt{TK\log(|\Pi|)})$ , which is information theoretically optimal. While oracle-efficient algorithms can obtain the optimal regret bound in the stochastic setting (Dudik et al., 2011), they do not always achieve optimal regret rates (Hazan and Koren, 2016). Whether or not optimal regret can be obtained in our setting using efficient algorithms remains an open problem and improving both the upper and lower bounds are interesting directions for future work. Additionally, while our work operates in the same setting as prior work (Rakhlin and Sridharan, 2016; Syrgkanis et al., 2016b), it would be interesting to relax some of the assumptions in the setting, most notably the sampling access to the context distribution.
This work is partially supported by DARPA QuICC NSF AF:Small #2218678, and NSF AF:Small #2114269. We thank Alex Slivkins for pointing us to the problem and initial fruitful discussions.
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. 2014. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning. PMLR, 1638-1646.
Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. 2017. Corralling a band of bandit algorithms. In Conference on Learning Theory. PMLR, 12-38.
Shipra Agrawal, Nikhil R Devanur, and Lihong Li. 2016. An efficient algorithm for contextual bandits with knapsacks, and an extension to concave objectives. In Conference on Learning Theory. PMLR, 4-18.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. 2002. Finite-time analysis of the multiarmed bandit problem. *Machine learning* 47 (2002), 235-256.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. 1995. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Proceedings of IEEE 36th annual foundations of computer science. IEEE, 322-331.
Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. 2011. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 19-26.
Adam Block, Yuval Dagan, Noah Golowich, and Alexander Rakhlin. 2022. Smoothed online learning is as easy as statistical learning. arXiv preprint arXiv:2202.04690 (2022).
Nicolo Cesa-Bianchi, Yoav Freund, David Haussler, David P Helmbold, Robert E Schapire, and Manfred K Warmuth. 1997. How to use expert advice. Journal of the ACM (JACM) 44, 3 (1997), 427-485.
Vasilis Syrgkanis, Akshay Krishnamurthy, and Robert Schapire. 2016a. Efficient algorithms for adversarial contextual learning. In International Conference on Machine Learning. PMLR, 2159-2168.
Vasilis Syrgkanis, Haipeng Luo, Akshay Krishnamurthy, and Robert E Schapire. 2016b. Improved regret bounds for oracle-based adversarial contextual bandits. Advances in Neural Information Processing Systems 29 (2016).
A Proof of Theorem 7
Combining Equation 8 and Theorem 6 we obtain
$$
\operatorname {R E G} _ {T} \leq 4 \sqrt {\frac {T K \log (| \Pi |)}{\gamma}} + \gamma T.
$$
Setting the derivative with respect to  $\gamma$  of RHS to zero, we obtain
$$
T - 2 \sqrt {T K \log | \Pi |} \gamma^ {- 3 / 2} = 0,
$$
which is equivalent to  $\gamma = \left(\frac{4K\log|\Pi|}{T}\right)^{1 / 3}$ . Note however that  $\gamma$  needs to satisfy  $\gamma >KT^{-1}\log (|\Pi |) / 2$  and  $\gamma \leq 1$ . To verify the first condition, we should have  $T > K\log (|\Pi |) / (2\gamma)$ . Putting our designated  $\gamma$  implies that  $T$  should satisfy
$$
\begin{array}{l} T > K \log (| \Pi |) \cdot \left(\frac {T}{4 K \log (| \Pi |)}\right) ^ {1 / 3} \cdot 1 / 2 \iff T ^ {3} > K ^ {3} \log^ {3} (| \Pi |) \frac {T}{4 K \log (| \Pi |)} \cdot 1 / 8 \\ \Longleftrightarrow T ^ {2} > \frac {K ^ {2} \log^ {2} | \Pi |}{3 2} \Longleftrightarrow T > \frac {K \log | \Pi |}{4 \sqrt {2}}, \\ \end{array}
$$
which holds by assumption. To verify the second condition of  $\gamma \leq 1$ , we need to have  $T > 4K\log (|\Pi |)$ , which again holds by assumption.
Plugging  $\gamma$  to the regret bound, we have
$$
\mathcal {O} \left(T ^ {2 / 3} \left(K \log | \Pi |\right) ^ {1 / 3}\right),
$$
completing the proof.
B Proof of Lemma 4
The proof is based on Lemma 4 in Syrgkanis et al. (2016b) and is provided for completeness. The pseudocode of our algorithm is provided in Algorithm 2.
Algorithm 2: Compute  $q_{t}^{*}$
Input: value-of-ERM oracle,  $(x,\hat{c})_{1:t - 1},x_{t}$  and  $\rho_t$
Output:  $q_{t}^{*}(\rho_{t})$  as in Equation 6
Compute for all  $i\in [K],\psi_{i} = \inf_{\pi \in \Pi}$  of
$$
\sum_ {\tau = 1} ^ {t - 1} \hat {c} _ {\tau} (\pi (x _ {\tau})) + \frac {K}{\gamma} \mathbf {e} _ {i} (\pi (x _ {t})) + \sum_ {\tau = t + 1} ^ {T} 2 Z _ {\tau} \varepsilon_ {\tau} (\pi (x _ {\tau}))
$$
using the value-of-ERM oracle
Compute  $\eta_{i} = \frac{\gamma(\psi_{i} - \psi_{0})}{K}$  for all  $i\in [K]$
Set  $m = 1,q = 0$
for  $k = 1,2,\ldots ,K$  do
$\mid q(i)\gets \min \{(\eta_i)^+,m\} ,\quad m\gets m - q(i)$
If  $m > 0$ , distribute remaining  $m$  uniformly across coordinates of  $q$ .
Proof. We prove the result by rewriting the minimizer equation to be composed of only calls to our value-of-ERM oracle. For  $i \in [K]$ , we define  $\psi_i$  to be
$$
\inf  _ {\pi \in \Pi} \left(\sum_ {\tau = 1} ^ {t - 1} \hat {c} _ {\tau} (\pi (x _ {\tau})) + \gamma^ {- 1} K \mathbf {e} _ {i} (\pi (x _ {t})) + \sum_ {\tau = t + 1} ^ {T} 2 Z _ {\tau} \varepsilon_ {\tau} (\pi (x _ {\tau}))\right)
$$
where  $\mathbf{e}_0 = \mathbf{0}$ . We can thus write the definition of  $q_t^* (\rho_t)$  as
$$
\arg \inf  _ {q \in \Delta_ {K}} \sup  _ {p _ {t} \in \Delta_ {D} ^ {\prime}} \sum_ {i = 1} ^ {K} p _ {t} (i) \left(\frac {K q (i)}{\gamma} - \psi_ {i}\right) - p (0) \cdot \psi_ {0}
$$
and note that the values of  $\psi_{i}$  can be computed via a single call to the value-of-ERM oracle, and moreover we require  $K + 1$  calls to compute all the  $\psi_{i}$ .
Now to compute the minimizer of the above, we first let  $z_{i} = \frac{Kq(i)}{\gamma} - \psi_{i}$  and  $z_{0} = -\psi_{0}$  and rewrite the minimax value as
$$
\operatorname *{arg  inf}_{q\in \Delta_{D}}\sup_{p_{t}\in \Delta^{\prime}_{D}}\sum_{i = 1}^{K}p_{t}(i)\cdot z_{i} + p_{t}(0)\cdot z_{0}.
$$
We reiterate that each  $p_t(i) \leq \gamma / K$  for  $i > 0$  and thus we must distribute maximal probability across the  $z_i$  coordinates of largest value, i.e. we will put as much probability weight as permitted on  $\arg \max_{i \in [K]} z_i$ , and proceed to do the same for the second largest value, repeating the process until we exhaust the probability distribution or reach the terminal  $z_0$ . At this point, we can put the remainder of the probability weight on this final coordinate to ensure summation to 1.
To proceed in analyzing the above water-filling argument, sort the coordinates  $z_{i}$  such that  $z_{(1)} \geq z_{(2)} \geq \ldots \geq z_{(K)}$  for  $i > 0$  and further define index  $\mu$  to be the smallest index such that  $z_{(\mu)} \geq z_0$ . By the probability weight distribution argument above, we can reduce the supremum over  $p_t$  to be
$$
\sum_ {i = 1} ^ {\mu} \frac {\gamma}{K} z _ {(i)} + (1 - \frac {\gamma}{K} \mu) z _ {0} = \sum_ {i = 1} ^ {\mu} \frac {\gamma}{K} (z _ {(i)} - z _ {0}) + z _ {0}.
$$
since we maximize  $p(i)$  for the  $z_{(i)}$  in order and set  $p(i) = 0$  for terms less than  $z_0$ , which would otherwise yield an addition of negative terms in the above summation. Thus, by definition of  $\mu$ , we have for  $i > \mu$
$$
\sum_ {i = 1} ^ {\mu} \frac {\gamma}{K} \left(z _ {(i)} - z _ {0}\right) + z _ {0} = \sum_ {i = 1} ^ {K} \frac {\gamma}{K} \left(z _ {(i)} - z _ {0}\right) ^ {+} + z _ {0}.
$$
where the  $+$  superscript denotes the RELU operator,  $(x)^{+} = \max \{x,0\}$ . Therefore, the minimax expression is reformulated as
$$
\arg \inf  _ {q \in \Delta_ {D}} \sum_ {i = 1} ^ {K} \frac {\gamma}{K} (z _ {(i)} - z _ {0}) ^ {+} + z _ {0}.
$$
which is equivalent to minimizing the RELU term
$$
\arg \inf  _ {q \in \Delta_ {D}} \sum_ {i = 1} ^ {K} \frac {\gamma}{K} (z _ {(i)} - z _ {0}) ^ {+} = \arg \inf  _ {q \in \Delta_ {D}} \sum_ {i = 1} ^ {K} \left(q (i) - \frac {\gamma}{K} \cdot (\psi_ {i} - \psi_ {0})\right) ^ {+}.
$$
Simplify notation by setting  $\eta_{i} = \frac{\gamma(\psi_{i} - \psi_{0})}{K}$  so that the expression becomes
$$
\arg \inf  _ {q \in \Delta_ {D}} \sum_ {i = 1} ^ {K} (q (i) - \eta_ {i}) ^ {+}.
$$
We argue the minimization procedure as follows: select  $i \in [K]$  such that  $\eta_i \leq 0$ . Any positive probability weight on  $q(i)$  will yield an increase in the expression, whereas for  $\eta_i > 0$  we experience no increase in the objective until the value of  $\eta_i$  surpasses  $q(i)$ . Thus, the minimizer will weight the actions with  $\min \{\sum_{i:\eta_i > 0} \eta_i, 1\}$  on the coordinates with  $\eta_i > 0$  and distribute the remaining weight (if nonzero) arbitrarily among  $[K]$ . This is more precisely outlined in the pseudocode of Algorithm 2.
C Proof of Lemma 8
Proof of Lemma 8. Denote by  $q_{t}^{*} = \mathbb{E}_{\rho_{t}}[q_{t}^{*}(\rho_{t})]$ . We note that since we are drawing from  $\rho_{t}$ , calculating  $q_{t}^{*}(\rho_{t})$ , and then drawing from  $q_{t}^{*}(\rho_{t})$ , our algorithm is effectively sampling from  $q_{t}^{*}$ , even though we do not calculate  $q_{t}^{*}$  explicitly. Now note that
$$
\mathbb {E} _ {\hat {y} _ {t} \sim q _ {t}} \left[ c _ {t} \left(\hat {y} _ {t}\right) \right] = \left\langle q _ {t}, c _ {t} \right\rangle \leq \left\langle q _ {t} ^ {*}, c _ {t} \right\rangle + \gamma \left\langle \frac {1}{K}, c _ {t} \right\rangle \leq \mathbb {E} _ {\hat {y} _ {t}, \hat {c} _ {t}} \left[ \left\langle q _ {t} ^ {*}, \hat {c} _ {t} \right\rangle \right] + \gamma .
$$
holds by definition of  $q_{t}(\rho_{t})$  and the assumed bounds on  $c_{t}$ . Thus, it further holds that