SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that our restriction to "slightly randomized" policies for the continuous arm case results in regret bounds with respect to a "slightly randomized" (smooth) benchmark. Hence for the continuous arm case, our cumulative regret bounds translate to smooth regret bounds from [40] with  $K = 1 / h$ . Where  $h$  is the measure of smoothness in smooth regret (a leading objective for this setting).
This ensures that  $\xi_{m}$  defined in Lemma 1 is non-increasing in  $m$  for any epoch schedule with increasing epoch lengths.
our algorithm relies on EvalOracle, which provides consistent independent policy value estimates and helps compare them with policy value estimates with respect to the estimated reward model.
Assumption 2 (Evaluation Oracle). We assume access to an oracle (EvalOracle) that takes as input an action selection kernel  $p$ ,  $n$  independently and identically drawn samples from the distribution  $D(p)$ , a set of  $m$  models  $\{g_i | i \in [m]\} \subseteq \mathcal{F}$ , and another action selection kernel  $q$ . The oracle then outputs a policy evaluation estimator  $\hat{R}$  of true policy value, and a set of  $m$  policy evaluation estimators  $\{\hat{R}_{g_i} | i \in [m]\}$  that estimate policy value with respect to the models  $g_1, g_2, \ldots, g_m$  respectively. Such that for any  $\delta' \in (0,1)$ , the following conditions simultaneously hold with probability at least  $1 - (m + 1)\delta'$ :
-  $|\hat{R}(\pi) - R(\pi)| \leq \sqrt{2V(p, \pi)\xi(n, \delta')} + 2\xi(n, \delta') / (\min_{(x,a) \in \mathcal{X} \times \mathcal{A}} p(a|x))$  for all  $\pi \in \Pi \cup \{q\}$ .
-  $|\hat{R}_f(\pi) - R_f(\pi)| \leq \sqrt{2\xi(n,\delta')}$  for all  $\pi \in \Pi \cup \{q\}$  and for all  $f \in \{g_i | i \in [m]\}$ .
When  $\mathcal{F}$  and  $\Pi$  are finite, one can construct oracles such that Assumptions 1 and 2 hold with  $\xi(n, \delta') = \mathcal{O}(\log(\max(|\mathcal{F}|, |\Pi|)/\delta'))/n$ . One example of such a construction is given by using empirical squared loss minimization for EstOracle, using inverse propensity scores (IPS) for estimating  $R(\pi)$  in EvalOracle, and using the empirical average for estimating  $R_f(\pi)$  in EvalOracle. The guarantees of these assumptions can be derived using Bernstein's inequality and union bounding. When  $\mathcal{F}$  has pseudo-dimension [20] bounded by  $d$  and  $\Pi$  has the Natarajan-dimension bounded by  $d$  [17, 16], one can construct oracles such that Assumptions 1 and 2 hold with  $\xi(n, \delta') = \mathcal{O}(d\log(nK/\delta'))/n$ .
Algorithm 1  $\omega$  Risk Adjusted Proportional Response (ω-RAPR)
input: Trade-off parameter  $\omega \in [1, K]$ , proportional response threshold  $\beta_{\mathrm{max}} = 1/2$ , and confidence parameter  $\delta$  (used in definition of  $\xi_m$ ).  
1: Let  $p_1(a|x) \equiv 1/\mu(A) = 1/K$ ,  $\hat{f}_1 \equiv 0$ ,  $\alpha_1 = 3K$ ,  $\tau_1 = 3$ , and safe = True.  
2: for epoch  $m = 1, 2, \ldots$  do  
3:  $\tau_m = 2\tau_{m-1}$ .  
4: if safe then  
5: for round  $t = \tau_{m-1} + 1, \ldots, \tau_m$  do  
6: Observe context  $x_t$ , sample  $a_t \sim p_m(\cdot|x_t)$ , and observe  $r_t(a_t)$ .  
7: end for  
8: Let  $S_m$  denote the data collected in epoch  $m$ .  
9: We split  $S_m$  into three equally sized sets  $S_{m,1}, S_{m,2}$  and  $S_{m,3}$ .  
10: Let  $\hat{f}_{m+1} \gets \text{EstOracle}(p_m, S_{m,1})$ , and let  $C_{m+1}$  be given by Definition 2.  
11: Let  $\eta_{m+1}$  be the solution to (5) and let  $\alpha_{m+1} := 3K/\eta_{m+1}$ .  
12: Now let  $p_{m+1}$  be given by (4).  
13: Let  $\hat{R}_{m+1}, \{\hat{R}_{m+1,\hat{f}_i} | i \in [m+1]\} \gets \text{EvalOracle}(p_m, S_{m,3}, \{\hat{f}_i | i \in [m+1]\}, p_{m+1})$ .  
14: if (2) does not hold, then  
15:  $\hat{m}$ , safe  $\leftarrow m$ , False.  
16: end if  
17: else  
18: for round  $t = \tau_{m-1} + 1, \ldots, \tau_m$  do  
19: Observe context  $x_t$ , sample  $a_t \sim p_{\hat{m}}(\cdot|x_t)$ , and observe  $r_t(a_t)$ .  
20: end for  
21: Let  $S_m$  denote the data collected in epoch  $m$ .  
22: Let  $\hat{R}_{m+1}, \{\hat{R}_{m+1,\hat{f}_i} | i \in [\hat{m}]\} \gets \text{EvalOracle}(p_{\hat{m}}, S_m, \{\hat{f}_i | i \in [\hat{m}]\}, p_{\hat{m}})$ .  
23: end if  
24: end for
At a high level, our algorithm operates in two modes, indicated by a Boolean variable "safe". During mode one (safe = True), where estimated reward models are sufficiently accurate at evaluating policies in the class  $\Pi$ , we use our estimated models to update our action selection kernel used
<sup>11</sup>Where the estimated reward models pass the misspecification test.
during exploration. During mode two (safe = False), where the condition for mode one no longer holds, we stop updating the action selection kernel used for exploration. Operationally our algorithm runs in epochs/batches indexed by  $m$ . Epoch  $m$  begins at round  $t = \tau_{m-1} + 1$  and ends at  $t = \tau_m$ , and we use  $m(t)$  to denote the epoch index containing round  $t$ . We let  $\hat{m}$  denote the critical epoch, at the end of which our algorithm changes mode (with "safe" being updated from "True" to "False"); we refer to  $\hat{m}$  as the algorithmic safe epoch. For all rounds in epoch  $m \leq \hat{m}$ , our algorithm samples action using the action selection kernel  $p_m$  defined later in (4). For  $m > \hat{m}$ , our algorithm samples action using  $p_{\hat{m}}$  -the action selection kernel used in the algorithmic safe epoch  $\hat{m}$ .
We now describe the critical components of our algorithm. These include (i) data splitting and using oracle sub-routines; (ii) misspecification tests, which we use to identify the safe-mode switching epoch  $\hat{m}$ ; and (iii) conformal arm sets, which presents a new form of uncertainty quantification that is critical in constructing  $p_{m+1}$  at the end of each epoch  $m \in [\hat{m}]$ . Finally, we use these components to describe our final algorithm.
Data splitting and oracle sub-routines. Consider an epoch  $m \in [\hat{m}]$ . Let  $S_{m}$  denote the set of samples collected in this epoch:  $S_{m} = \{(x_{t}, a_{t}, r_{t}(a_{t})) | t \in [\tau_{m-1}, \tau_{m}] \}$ . Our algorithm splits  $S_{m}$  into three equally-sized subsets:  $S_{m,1}, S_{m,2}$  and  $S_{m,3}$ . Algorithm 1 outlines using these subsets and the oracles (described in Section 1.2) to estimate reward models and evaluate policies. Based on Assumptions 1 and 2, we bound the errors for these estimates in terms of  $\xi_{m+1} = 2\xi((\tau_{m} - \tau_{m-1}) / 3, \delta / (16m^{3}))$ , where  $\delta$  is a specified confidence parameter. As we will see later, our algorithm relies on these bounds to test for misspecification and construct action selection kernels.
Misspecification test. We first discuss the need for our misspecification test. Note that Assumption 1 is flexible and allows our reward model class  $\mathcal{F}$  to be misspecified. In particular, the squared error of our reward model estimate may depend on an unknown bias term  $B$ . To account for this unknown  $B$ , it is useful to center our analysis around the safe epoch  $m^{*} \coloneqq \arg \max \{m \geq 1 | \xi_{m+1} \geq 2B\}$ , which denotes the last epoch where variance dominates bias. We show that for any epoch  $m \in [m^{*}]$ , the estimated reward model  $\hat{f}_{m+1}$  is "sufficiently accurate" at evaluating the expected reward of any policy in  $\Pi \cup \{p_{m+1}\}$ . This property is critical in ensuring that the constructed action selection kernel  $p_{m+1}$  has low exploration regret  $\mathrm{Reg}_{f^{*}}(p_{m+1})$  and a small optimal cover  $(V(p_{m+1}, \pi^{*}))$ . Since  $B$  and  $m^{*}$  are unknown, we need to test whether the estimated reward model is sufficiently accurate at evaluating these policies. When the test fails, the algorithm sets the variable "safe" to False and stops updating the action selection kernel used for exploration. The core idea for this test comes from [23] although its application to simple regret minimization is new, and the form of our test differs a bit. We now state our misspecification test (2). At the end of each epoch  $m$ , the test is passed if (2) holds:
$$
\begin{array}{l} \max _ {\pi \in \Pi \cup \{p _ {m + 1} \}} | \hat {R} _ {m + 1, \hat {f} _ {m + 1}} (\pi) - \hat {R} _ {m + 1} (\pi) | - \sqrt {\alpha_ {m} \xi_ {m + 1}} \sum_ {\bar {m} \in [ m ]} \frac {\hat {R} _ {m + 1 , \hat {f} _ {\bar {m}}} (\pi_ {\hat {f} _ {\bar {m}}}) - \hat {R} _ {m + 1 , \hat {f} _ {\bar {m}}} (\pi)}{4 0 \bar {m} ^ {2} \sqrt {\alpha_ {\bar {m} - 1} \xi_ {\bar {m}}}} \\ \leq 2. 0 5 \sqrt {\alpha_ {m} \xi_ {m + 1}} + 1. 1 \sqrt {\xi_ {m + 1}}, \tag {2} \\ \end{array}
$$
where  $\alpha_{\bar{m}}$  empirically bounds  $V(p_{\bar{m}},\pi^{*})$ , the optimal cover for the action selection kernel used in epoch  $\bar{m}$  (see (49)). The first term in (2) measures how well the estimated reward model  $\hat{f}_{m + 1}$  evaluates the policy  $\pi$ , and the second term accounts for under-explored policies (policies that have high regret under the reward model  $\hat{f}_m$  would be less explored in epoch  $m$ ).
Conformal arm sets. We proceed to introduce the notion of conformal arm sets (CASs), based on which we construct the action selection kernels employed by our algorithms. At the beginning of each epoch  $m$ , we construct CASs, denoted as  $\{C_m(x,\zeta)|x\in \mathcal{X},\zeta \in [0,1]\}$ ; here  $\zeta$  controls the probability with which the set  $C_m$  contains the optimal arm. The construction of these sets relies on the models  $(\hat{f}_1,\dots ,\hat{f}_m)$  estimated from data up to epoch  $m - 1$ , as defined below.
Definition 2 (Conformal Arm Sets). Consider  $\zeta \in (0,1)$ . At epoch  $m$ , for context  $x$ , the arm set  $C_m(x,\zeta)$  is given by (3).
$$
\begin{array}{l} C _ {m} (x, \zeta) := \pi_ {\hat {f} _ {m}} (x) \bigcup \bar {C} _ {m} (x, \zeta), \quad \bar {C} _ {m} (x, \zeta) := \bigcap \tilde {C} _ {\bar {m}} \left(x, \frac {\zeta}{2 x \bar {m} ^ {2}}\right), \\ \bar {m} \in [ m ] \quad \tag {3} \\ \end{array}
$$
$$
\tilde {C} _ {\bar {m}} (x, \zeta^ {\prime}) := \left\{a: \hat {f} _ {\bar {m}} (x, \pi_ {\hat {f} _ {\bar {m}}} (x)) - \hat {f} _ {\bar {m}} (x, a) \leq \frac {2 0 \sqrt {\alpha_ {\bar {m} - 1} \xi_ {\bar {m}}}}{\zeta^ {\prime}} \right\} \forall \bar {m} \in [ m ], \zeta^ {\prime} \in (0, 1).
$$
Similar to conformal prediction (CP) [36, 33], CASs have marginal coverage guarantees. We show that with high probability, we have  $\pi^{*}(x)$  lies in  $C_m(x,\zeta)$  with probability at least  $1 - \zeta$  over the context distribution. That is,  $\operatorname{Pr}_{x\sim D_{\mathcal{X}}}\left(\pi^{*}(x)\in C_{m}(x,\zeta)\right)\geq 1 - \zeta$  with high-probability (see Appendix E.1). However, there is also a key technical difference. While CP provides coverage guarantees for the conditional random outcome, CASs provide coverage guarantees for  $\pi^{*}(x)$  - which is not a random variable given the context  $x$ . Hence, intervals estimated by CP need to be wide enough to account for conditional outcome noise, whereas CASs do not. CASs also have several advantages compared to pointwise confidence intervals used in UCB algorithms. First, CASs are computationally easier to construct. Second, CAS widths have a polynomial dependency on model class complexity, whereas pointwise intervals may have an exponential dependence for some function classes [see lower bound examples in 13]. Third, pointwise intervals require realizability, whereas the guarantees of CASs hold even without realizability (as long as the misspecification test in (2) holds). However, it's important to remember that these benefits of CASs come with the risk of only covering  $\pi^{*}(x)$  marginally over the context distribution - that is, these sets may not contain  $\pi^{*}(x)$  at all  $x$ .
Risk Adjusted Proportional Response Algorithm. We now describe the design of our algorithm, which is summarized in Algorithm 1. The algorithm depends on the following input parameters:  $\omega \in [1,K]$  which controls the trade-off between simple and cumulative regret, the proportional response threshold  $\beta_{\mathrm{max}} = 1 / 2$  , and confidence parameter  $\delta$  . The algorithm also computes  $\eta_{m + 1}$  (risk adjustment parameter for  $p_{m + 1}$  ),  $\alpha_{m + 1}$  (empirical bound on optimal cover for  $p_{m + 1}$  ), and  $\lambda_{m + 1}(\cdot)$  (empirical bound on average CAS size). At the end of every epoch  $m\in [\hat{m} ]$  , we construct the action selection kernel  $p_{m + 1}$  given by (4).
$$
p _ {m + 1} (a | x) = \frac {\left(1 - \beta_ {\max }\right) I \left[ a \in C _ {m + 1} \left(x , \beta_ {\max } / \eta_ {m + 1}\right) \right]}{\mu \left(C _ {m + 1} \left(x , \beta_ {\max } / \eta_ {m + 1}\right)\right)} + \int_ {0} ^ {\beta_ {\max }} \frac {I \left[ a \in C _ {m + 1} \left(x , \beta / \eta_ {m + 1}\right) \right]}{\mu \left(C _ {m + 1} \left(x , \beta / \eta_ {m + 1}\right)\right)} d \beta . \tag {4}
$$
At any context  $x$ , sampling arm  $a$  from  $p_{m+1}(\cdot|x)$  is equivalent to the following. Sample  $\beta$  uniformly from [0, 1], then sample arm  $a$  uniformly from the set  $C_{m+1}(x, \min(\beta_{\max}, \beta)/\eta_{m+1})$ . A small  $\beta$  results in a larger CAS and a higher probability of containing the optimal arm for the sampled context. However, uniformly sampling an arm from a larger CAS also implies a lower probability on every arm in the set. Sampling  $\beta$  uniformly allows us to respond proportionately to the risk of not sampling the optimal arm while enjoying the benefits of smaller CASs. We refer to this as the Proportional Response Principle.
Similarly note that, a larger risk-adjustment parameter  $\eta_{m + 1}$  encourages reliance on less risky albeit larger CASs. We want to choose  $\eta_{m + 1}$  to tightly bound the optimal cover (surrogate for simple regret), subject to cumulative regret constrains imposed by the trade-off parameter  $\omega$ . To do this, we first let  $\lambda_{m + 1}(\eta)$  be a high-probability empirical upper bound on  $E_{x\sim D_X}[\mu (C_{m + 1}(x,\beta_{\max} / \eta))]$ . Hence, using (49), we can upper bound the optimal cover  $(V(p_{m + 1},\pi^{*}))$  by  $\frac{\lambda_{m + 1}(\eta_{m + 1})}{1 - \beta_{\max}} +\frac{K}{\eta_{m + 1}}$ . Our choice of  $\eta_{m + 1}$  approximately minimizes this upper bound on the optimal cover, by choosing the largest feasible  $\eta \in [\eta_m,\sqrt{\omega K / \alpha_m} ]$  such that  $\lambda_{m + 1}(\eta)\leq \frac{K}{\eta}$  (see (5)). Note that this choice of  $\eta_{m + 1}$  balances the risk of a small  $\eta$  (large  $\frac{K}{\eta}$ ) with the benefits of a small  $\lambda_{m + 1}(\eta)$  (small  $\frac{\lambda_{m + 1}(\eta)}{1 - \beta_{\max}}$ ).
$$
\lambda_ {m + 1} (\eta) := \min \left(1 + \frac {1}{| S _ {m , 2} |} \sum_ {t \in S _ {m, 2}} \mu (\bar {C} _ {m + 1} \Big (x, \frac {\beta_ {\max}}{\eta} \Big)) + \sqrt {\frac {K ^ {2} \ln (8 | S _ {m , 2} | (m + 1) ^ {2} / \delta)}{2 | S _ {m , 2} |}}, K\right),
$$
$$
\eta_ {m + 1} \leftarrow \max  \left\{\eta_ {m}, \max  \left\{\eta = \frac {\left| S _ {m , 2} \right|}{n} \mid n \in \left[ \left| S _ {m, 2} \right| \right], \eta \leq \sqrt {\frac {\omega K}{\alpha_ {m}}}, \lambda_ {m + 1} (\eta) \leq \frac {K}{\eta} \right\} \right\}. \tag {5}
$$
With  $\eta_{m + 1}$  chosen, the action selection kernel  $p_{m + 1}$  is decided. Now let  $\alpha_{m + 1} = 3K / \eta_{m + 1}$ , which is a high-probability empirical upper bound on  $V(p_{m + 1},\pi^{*})$ . We then use  $\alpha_{m + 1}$  at the end of epoch  $m + 1$  to construct CASs, compute the risk-adjustment parameter, and test for misspecification.
**Computation.** We have  $\mathcal{O}(\log (T))$  epochs. At the end of any epoch  $m\in [\hat{m} ]$ , we solve three optimization problems. The first is for estimating  $\hat{f}_{m + 1}$ , which often reduces to empirical squared loss minimization and is computationally tractable for several function classes  $\mathcal{F}$ . The second is for computing the risk-adjustment parameter in (5) which can be solved via binary search. The third is for the misspecification test in (2), which can be solved via two calls to a cost-sensitive classification (CSC) solver (don't need this when assuming realizability, further if we only care about cumulative
regret, sufficient to use the simpler test in [22]). Finally, to learn a policy  $\hat{\pi}$  at the end of  $T$  rounds, we need to solve (8) using a CSC solver (under realizability we can set  $\hat{\pi} = \pi_{\hat{f}_{m(T)-1}}$ ). Hence, overall,  $\omega$ -RAPR makes exponentially fewer calls to solvers compared to regression-free algorithms like [26].
Our algorithm/analysis/results hold for both the discrete and continuous arm cases. As discussed before, minimizing optimal cover helps us ensure improved simple regret guarantees. Hence  $\alpha_{m} \in [1, 3K]$  (the high-probability empirical upper bound on the optimal cover  $V(p_{m}, \pi^{*})$ ) will play a critical role throughout this results section. We start with stating our cumulative regret bounds.
Theorem 1. Suppose Assumptions 1 and 2 hold. Then with probability  $1 - \delta$ ,  $\omega$ -RAPR attains the following cumulative regret guarantee. Here  $\xi_{m + 1} = \xi((\tau_m - \tau_{m - 1}) / 3, \delta / (16m^3)$  for all  $m$ .
$$
\begin{array}{l} C R e g _ {T} \leq \tilde {\mathcal {O}} \left(\sum_ {t = \tau_ {1} + 1} ^ {T} \sqrt {\frac {K}{\alpha_ {m (t)}} \frac {\alpha_ {m (t) - 1}}{\alpha_ {m (t)}}} \left(\sqrt {K B} + \sqrt {K \xi_ {m (t)}}\right)\right) (6a) \\ \leq \tilde {\mathcal {O}} \left(\sqrt {\omega K B} T + \sum_ {t = \tau_ {1} + 1} ^ {T} \sqrt {\omega K \xi_ {m (t)}}\right). (6b) \\ \end{array}
$$
Where we use  $\tilde{\mathcal{O}}$  to hide terms logarithmic in  $T, K, \xi(T, \delta)$ .
We start with discussing (6b). The first part  $\sqrt{\omega KB}T$  comes from the bias of the regression oracle with model class  $\mathcal{F}$  and will vanish under the realizability assumption. The second part  $\sum_{t = \tau_1 + 1}^{T}\sqrt{\omega K\xi_{m(t)}}$ , when setting  $\omega = 1$ , recovers near-optimal (upto logarithmic factors) minimax cumulative regret guarantees for common model classes, as demonstrated by the following examples.
Corollary 1. We consider  $\omega$ -RAPR with appropriate oracles in the following cases and let  $B$  denote the corresponding bias terms. When  $\mathcal{F}$  and  $\Pi$  are finite,  $CReg_{T} \leq \tilde{\mathcal{O}}(\sqrt{\omega KB T} + \sqrt{\omega KT \log(\max(|\mathcal{F}|, |\Pi|)/\delta})$  with probability at least  $1 - \delta$ . When  $\mathcal{F}$  has a finite pseudo dimension  $d$ ,  $\Pi$  has a finite Natarajan dimension  $d$ , and  $\mathcal{A}$  is finite,  $CReg_{T} \leq \tilde{\mathcal{O}}(\sqrt{\omega KB T} + \sqrt{\omega KT d \log(TK/\delta)})$  with probability at least  $1 - \delta$ . Note that under realizability ( $B = 0$ ), 1-RAPR achieves near-optimal minimax cumulative guarantees.
In (6a), we observe that the multiplicative  $\sqrt{\omega}$  cost to cumulative regret is only incurred if the empirical bound on optimal cover  $(\alpha_{m} \in [1, 3K])$  can get small. That is, our cumulative regret bounds degrade only if our algorithm better bounds the optimal cover and thus ensures better simple regret guarantees. We now provide instance-dependent simple regret guarantees for our algorithm.
Theorem 2. Suppose Assumptions 1 and 2 hold. For some  $(\lambda, \Delta, A) \in [0,1] \times (0,1] \times [1, K]$ , consider instances where for  $1 - \lambda$  fraction of contexts at most  $A$  arms are  $\Delta$  optimal (i.e. (7) holds).
$$
\mathbb {P} _ {x \sim D _ {\mathcal {X}}} \left(\mu \left(\{a \in \mathcal {A}: f ^ {*} (x, \pi_ {f ^ {*}} (x)) - f ^ {*} (x, a) \leq \Delta \}\right) \leq A\right) \geq 1 - \lambda . \tag {7}
$$
Let  $m' = \min(\hat{m}, m(T)) - 1$ . Let the learned policy  $\hat{\pi}$  be given by (8) (equivalent to variance penalized policy optimization).
$$
\hat {\pi} \in \arg \max  _ {\pi \in \Pi} \hat {R} _ {m (T)} (\pi) - \frac {1}{2} \sqrt {\alpha_ {m ^ {\prime}} \xi_ {m (T)}} \sum_ {\bar {m} \in [ m ^ {\prime} ]} \frac {\hat {R} _ {m (T) , \hat {f} _ {\bar {m}}} \left(\pi_ {\hat {f} _ {\bar {m}}}\right) - \hat {R} _ {m (T) , \hat {f} _ {\bar {m}}} (\pi)}{4 0 \bar {m} ^ {2} \sqrt {\alpha_ {\bar {m} - 1} \xi_ {\bar {m}}}}. \tag {8}
$$
Then with probability  $1 - \delta$ ,  $\omega$ -RAPR has the following simple regret bound when  $T$  samples.
$$
\begin{array}{l} R e g _ {\Pi} (\hat {\pi}) \leq \mathcal {O} \left(\sqrt {\alpha_ {m ^ {\prime}} \xi_ {m (T)}}\right) \\ \leq \mathcal {O} \left(\sqrt {\xi_ {m (T)} \min  \left(K , A + K \lambda + \frac {K}{\omega} + \frac {K ^ {3 / 2} \omega^ {1 / 2}}{\Delta} \sqrt {\xi_ {\min  (m ^ {*} , m (T) - 1) - \lceil \log_ {2} \log_ {2} (K) \rceil}}\right)}\right). \\ \end{array}
$$
For large  $t$ , once our bounds on optimal cover can't be significantly improved, we have  $\alpha_{m(t)} / \alpha_{m(t) - 1} = O(1)$ . Hence for large  $T$ , our cumulative regret is a factor of  $\sqrt{K / \alpha_{m(T) + 1}}$  larger than the near optimal minimax guarantees. As we will see in Theorem 3, this multiplicative factor is unavoidable.
Under (7), we can only argue that the expected (over context distribution) measure of  $\Delta$  optimal arms is at most  $(1 - \lambda)A + K\lambda = O(A + K\lambda)$ . Hence for large  $T$ , the best we can hope for is instance-dependent simple regret guarantees that shrink/improve over minimax guarantees by a factor of  $\mathcal{O}(\sqrt{(A + K\lambda) / K})$ . We show that this is guaranteed by Theorem 2. Suppose  $\omega = K$ ,  $\mathcal{F}$  has a finite pseudo dimension bounded by  $d$ , and  $\Pi$  has a finite Natarajan dimension bounded by  $d$ . The simple regret guarantee of Theorem 2 reduces to  $\tilde{\mathcal{O}} (\min ((\sqrt{Kd / T},\sqrt{(A + K\lambda)d / T} + (K / \sqrt{\Delta})\sqrt{d / T}\sqrt[4]{B + d / T}))$ . When the reward model estimation bias  $B$  is small enough, the term  $(K / \sqrt{\Delta})\sqrt{d / T}\sqrt[4]{B + d / T})$  is dominated by the remaining terms for large  $T$ . Hence, in this case, we get a simple regret bound of  $\tilde{\mathcal{O}} (\sqrt{(A + K\lambda)d / T})$  for large  $T$ . As promised, this improves upon the minimax guarantees by a factor of  $\mathcal{O}(\sqrt{(A + K\lambda) / K})$ .
Note that Theorem 1 guarantees are better for  $\omega$  closer to 1 whereas Theorem 2 guarantees are better for  $\omega$  closer to  $K$ . Hence these theorems show a tradeoff between the cumulative and simple regret guarantees for  $\omega$ -RAPR. Theorem 3 shows that improving upon minimax simple regret guarantees for instances satisfying (7) may come at the unavoidable cost of worse than minimax optimal cumulative regret guarantees. This contrasts with non-contextual bandits, where successive elimination ensures improved (compared to minimax) gap-dependent guarantees for both simple and cumulative regret.
Theorem 3. Given parameters  $K, F, T \in \mathbb{N}$  and  $\phi \in [1,\infty)$ . There exists a context space  $\mathcal{X}$  and a function class  $\mathcal{F} \subseteq (\mathcal{X} \times \mathcal{A} \to [0,1])$  with  $K$  actions such that  $|\mathcal{F}| \leq F$  and the following lower bound on cumulative regret holds:
$$
\inf  _ {\mathbf {A} \in \Psi_ {\phi}} \sup  _ {D \in \mathcal {D}} \underset {D} {\mathbb {E}} \left[ \sum_ {t = 1} ^ {T} \left(r _ {t} \left(\pi^ {*} (x _ {t})\right) - r _ {t} (a _ {t})\right) \right] \geq \tilde {\Omega} \left(\sqrt {\frac {K}{\phi}} \sqrt {K T \log F}\right)
$$
Here  $(a_{1},\ldots a_{T})$  denotes the actions selected by an algorithm  $A$ .  $\mathcal{D}$  denotes the set of environments such that  $f^{*}\in \mathcal{F}$  and (7) hold with  $(A,\lambda ,\Delta) = (1,0,0.24)$ .  $\Pi$  denotes policies induced by  $\mathcal{F}$ .  $\Psi_{\phi}$  denotes the set of CB algorithms that run for  $T$  rounds and output a learned policy with a simple regret guarantee of  $\sqrt{\phi\log F / T}$  for any instance in  $\mathcal{D}$  with confidence at least 0.95, i.e.,  $\Psi_{\phi}:= \{A:\mathbb{P}(Reg(\hat{\pi}_{\mathbf{A}})\leq \sqrt{\phi\log F / T})\geq 0.95$  for any instance in  $\mathcal{D}\}$ . Finally,  $\tilde{\Omega} (\cdot)$  hides factors logarithmic in  $K$  and  $T$ .