SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Near optimal trade-off of RAPR. Note that the environments constructed in Theorem 3 satisfy  $f^{*} \in \mathcal{F}$  with  $\max(|\mathcal{F}|, |\Pi|) \leq F$  and also satisfy (7) with  $(A, \lambda, \Delta) = (1, 0, 0.24)$ . With appropriate oracles, Assumptions 1 and 2 are satisfied with  $B = 0$  (i.e.  $m^{*} = \infty$ ) and  $\xi(n, \delta') = \mathcal{O}(\log(F / \delta') / n)$ . Hence for large enough  $T$ ,  $\omega$ -RAPR achieves a simple regret bound of  $\tilde{\mathcal{O}}(\sqrt{(K / \omega) \log F / T})$  with probability at least 0.95 and thus is a member of  $\Psi_{\phi}$  for some  $\phi = \tilde{\mathcal{O}}(K / \omega)$ . Theorem 3 lower bounds the cumulative regret of such algorithms by  $\tilde{\Omega}(\sqrt{K / \phi} \sqrt{KT \log F}) = \tilde{\Omega}(\sqrt{\omega} \sqrt{KT \log F})$ . Up to logarithmic factors, this matches the cumulative regret upper bound for  $\omega$ -RAPR. Re-emphasizing that the trade-off observed in Theorems 1 and 2 is near optimal for large  $T$ .
Simulations. To demonstrate the computational tractability of our approach, we ran a simulation on setting within a  $\mathbb{R}^2$  context space, eight arms, linear models, and an exploration horizon of 5000. Our algorithms ran in less than 9 seconds on a MacBook M1 Pro. We also compare with other baselines on simple/cumulative regret. See Appendix E.4 for details.
Conclusion. We develop Risk Adjusted Proportional Response (RAPR), a computationally efficient regression-based contextual bandit algorithm. It is the first contextual bandit algorithm capable of trading-off worst-case cumulative regret guarantees with instance-dependent simple regret guarantees. The versatility of our algorithm allows for general reward models, handles misspecification, extends to finite and continuous arm settings, and allows us to choose the trade-off between simple and cumulative regret guarantees. The key ideas underlying RAPR are conformal arm sets (CASs) to quantify uncertainty, proportional response principle for cumulative regret minimization, optimal cover as a surrogate for simple regret, and risk adjustment for better bounds on the optimal cover.
Limitations. A limitation of our approach is that we do not utilize the structure of the policy class being explored. Further refining CASs with other forms of uncertainty quantification that leverage such structure can lead to significant improvements, and potentially avoid trade-offs between simple/cumulative regret when policy class structure allows for it.
$^{13}$ S.A. and S.K.K. are grateful for the support provided by Golub Capital Social Impact Lab and the ONR grant N00014-19-1-2468. E.B. is grateful for the support of NSF grant 2112926.