SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {f} _ {t} = \underset {f \in \mathcal {F}} {\arg \min } \sum_ {\ell = 1} ^ {t - 1} \left(f \left(\tilde {x} _ {\ell}, \tilde {a} _ {\ell}\right) - \tilde {r} _ {\ell}\right) ^ {2}. \tag {3}
$$
Standard Least Squares (LS) results (see Lemma A.8) imply that  $\| \widehat{f}_t - f_\star \|_{\tilde{\mathcal{D}}_t} \leqslant \beta(t, \delta)$  with high probability for all  $t \in \mathbb{N}$ . These confidence sets are used to define a sequence of optimistic (deterministic) policies  $\{\widetilde{\pi}_t^{\mathrm{opt}}\}_{t=1}^T$ :
$$
\widetilde {\pi} _ {t} ^ {\operatorname {o p t}} (x) = \underset {a \in \mathcal {A}} {\arg \max } \underset {\tilde {f} \text {s . t .} \| \hat {f} _ {t} - f \| _ {\tilde {\mathcal {D}} _ {t}} \leqslant \beta_ {\mathcal {F}} (t, \delta)} {\max } \tilde {f} (x, a) \tag {4}
$$
The candidate optimal policy  $\widehat{\pi}_T$  is then defined as  $\widehat{\pi}_T = \mathrm{Uniform}(\widetilde{\pi}_1^{\mathrm{opt}},\dots ,\widetilde{\pi}_T^{\mathrm{opt}})$ . The main result in this section is the following Theorem.
Theorem 4.4. Let  $\varepsilon >0$ . There exists a universal constant  $c > 0$  such that if
$$
T \geqslant c \frac {\max  ^ {2} (B , \bar {B} , 1) d _ {\text {e l u d e r}} (\mathcal {F} , B / T) \log \left(| \mathcal {F} | T / \delta\right)}{\varepsilon^ {2}} \tag {5}
$$
then with probability at least  $1 - \delta$  the eluder planning algorithm's policy  $\widehat{\pi}_T$  is  $\varepsilon$ -optimal.
Remark 4.5. The results of this section hold for the structured bandits setting [23, 27]; that is, scenarios where  $\mathcal{X} = \{\varnothing\}$ .
Both sides of the inequality defining  $T$  in Theorem 4.4 depend on  $T$ . Provided  $d_{\mathrm{eluder}}(\mathcal{F}, B / T)$  is sublinear as a function of  $T$ , there exists a finite value of  $T$  satisfying Equation 5. When  $d_{\mathrm{eluder}}(\mathcal{F}, \bar{\varepsilon}) = \bar{d}_{\mathrm{eluder}} \log (1 / \bar{\varepsilon})$  for some  $\bar{d}_{\mathrm{eluder}}$  and for all  $\bar{\varepsilon} > 0$  setting  $T \geqslant c \frac{\max^2(B, B, 1) d_{\mathrm{eluder}}(\mathcal{F}, B \varepsilon) \log (|\mathcal{F}| / \varepsilon \delta)}{\varepsilon^2}$  is enough to guarantee the conditions of Theorem 4.4 are satisfied. Examples 4 and 5 from [42] show the eluder dimension of linear or generalized linear function classes can be written in such way. In these cases  $\bar{d}_{\mathrm{eluder}}$  is a constant multiple of the underlying dimension of the space.
Using standard techniques, the results of Theorem 4.4 can be adapted to the case when the function class  $\mathcal{F}$  is infinite but admits a metrization and has a covering number. In this case the sample complexity will scale not with  $\log (|\mathcal{F}|)$  but instead with the logarithm of the covering number of  $\mathcal{F}$ . For example, in the case of linear functions, the logarithm of the covering number of  $\mathcal{F}$  under the  $\ell_2$  norm will scale as  $d$ , the ambient dimension of the space up to logarithmic factors of  $T$  (see for example Lemma D.1 of [14]). Plugging this into the sample guarantees of Theorem 4.4 recovers the  $\tilde{\mathcal{O}}\left(d^2/\varepsilon^2\right)$  sample complexity for linear experiment planning from [53].
4.1 Proof Sketch of Theorem 4.4
Let  $\tilde{x}_1, \dots, \tilde{x}_T$  be the sampler's context sequence. The proof works by showing that with probability at least  $1 - \delta$  the regret of the  $\{\widetilde{\pi}_t^{\mathrm{opt}}(\cdot)\}_{t=1}^T$  sequence satisfies the regret bound
$$
\sum_ {t = 1} ^ {T} \max  _ {a \in \mathcal {A}} f _ {\star} (\widetilde {x} _ {t}, a) - f _ {\star} (\widetilde {x} _ {t}, \widetilde {\pi} _ {t} ^ {\mathrm {o p t}} (\widetilde {x} _ {t})) \leqslant \mathcal {O} \left((B + \bar {B}) \sqrt {d _ {\text {e l u d e r}} (\mathcal {F} , B / T) T \log (| \mathcal {F} | T / \delta)}\right)
$$
A key part of the proof involves relating the balls defined by the data norms of the planner datasets  $\{\mathcal{D}_t\}_{t=1}^T$  and those induced by the data norms of the sampler datasets  $\{\widetilde{\mathcal{D}}_t\}_{t=1}^T$ . The precise statement of this relationship is provided in Lemma 4.6, and its proof can be found in Appendix C.
Lemma 4.6. With probability at least  $1 - \frac{\delta}{12}$ ,
$$
\left\{\left(f, f ^ {\prime}\right) \text {s . t .} \| f - f ^ {\prime} \| _ {\tilde {\mathcal {D}} _ {t}} \leqslant 2 \beta_ {\mathcal {F}} (t, \delta) \right\} \subseteq \left\{\left(f, f ^ {\prime}\right) \text {s . t .} \| f - f ^ {\prime} \| _ {\mathcal {D} _ {t}} \leqslant 4 \beta_ {\mathcal {F}} (t, \delta) \right\} \tag {6}
$$
Simultaneously for all  $t \in [T]$ . Where  $\{D_t\}_{t=1}^T$  is the dataset sequence resulting of the execution of EluderPlanner while  $\{\widetilde{D}_t\}_{t=1}^T$  is the dataset sequence resulting of the execution of the Sampler and  $\beta_{\mathcal{F}}(\delta, t)$  is the confidence radius function defined in Equation 2
Let  $\widehat{f}_t$  as in Equation 3 and define  $\mathcal{E}$  as the event where the Standard Least Squares results (see Lemma A.8) hold i.e.  $\| \widehat{f}_t - f_\star \|_{\widetilde{\mathcal{D}}_t} \leqslant \beta_\mathcal{F}(t, \delta)$  for all  $t \in [T]$  and also Equation 6 from Lemma 4.6 holds for all  $t \in [T]$ . The results of Lemmas A.8 and 4.6 imply  $\mathbb{P}(\mathcal{E}) \geqslant 1 - \frac{\delta}{6}$ .
For context  $x$  and action  $a$  let's denote by  $\tilde{f}_t^{x,a}$  as a function achieving the inner maximum in the definition of  $\tilde{\pi}_t^{\mathrm{opt}}$  (see Equation 4). When  $\mathcal{E}$  holds the policies  $\{\tilde{\pi}_t^{\mathrm{opt}}\}_{t=1}^T$  are optimistic in the sense that  $\tilde{f}_t^{x,\tilde{\pi}_t^{\mathrm{opt}}(x)}(x, \tilde{\pi}_t^{\mathrm{opt}}(x)) \geqslant \max_{a \in \mathcal{A}} f_\star(x, a)$  and therefore,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \max  _ {a \in \mathcal {A}} f _ {\star} (\tilde {x} _ {t}, a) - f _ {\star} (\tilde {x} _ {t}, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (\tilde {x} _ {t})) \stackrel {(i)} {\leqslant} \sum_ {t = 1} ^ {T} \tilde {f} _ {t} ^ {x, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (x)} (\tilde {x} _ {t}, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (\tilde {x} _ {t})) - f _ {\star} (\tilde {x} _ {t}, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (\tilde {x} _ {t})) \\ \stackrel {(i i)} {\leqslant} \sum_ {t = 1} ^ {T} \max  _ {f, f ^ {\prime} \in \mathbb {B} _ {t} (2, \tilde {\mathcal {D}} _ {t})} f (\tilde {x} _ {t}, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (\tilde {x} _ {t})) - f ^ {\prime} (\tilde {x} _ {t}, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (\tilde {x} _ {t})) \\ \stackrel {(i i i)} {\leqslant} \sum_ {t = 1} ^ {T} \max  _ {f, f ^ {\prime} \in \mathbb {B} _ {t} (4, \mathcal {D} _ {t})} f (\tilde {x} _ {t}, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (\tilde {x} _ {t})) - f ^ {\prime} (\tilde {x} _ {t}, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (\tilde {x} _ {t})) = (\star). \\ \end{array}
$$
Where  $\mathbb{B}_t(\gamma, \mathcal{D}) = \{f, f' \in \mathcal{F} \text{ s.t. } \| f - f' \|_{\mathcal{D}} \leqslant \gamma \beta_{\mathcal{F}}(\delta, t)\}$ . Inequality (i) follows by optimism, (ii) is a consequence of  $\tilde{f}_t^{x,\tilde{\pi}_t^{\mathrm{opt}}(x)}$ ,  $f_\star \in \mathbb{B}_t(2, \tilde{\mathcal{D}}_t)$  when  $\mathcal{E}$  holds since in this case  $\| \hat{f}_t - f_\star \|_{\tilde{\mathcal{D}}_t} \leqslant \beta_{\mathcal{F}}(t, \delta)$  and  $\| \tilde{f}_t^{x,\tilde{\pi}_t^{\mathrm{opt}}(x)} - \hat{f}_t \|_{\tilde{\mathcal{D}}_t} \leqslant \beta_{\mathcal{F}}(t, \delta)$ , and (iii) follows because when  $\mathcal{E}$  holds, Equation 6 of Lemma 4.6 is satisfied and therefore  $\mathbb{B}_t(2, \tilde{\mathcal{D}}_t) \subseteq \mathbb{B}_t(4, \underline{\mathcal{D}}_t)$ . The RHS ( $\star$ ) of the equation above satisfies,
$$
(\star) = \sum_ {t = 1} \omega (\tilde {x} _ {t}, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (\tilde {x} _ {t}), \mathcal {D} _ {t}) \leqslant \sum_ {t = 1} ^ {\tau} \max  _ {a \in \mathcal {A}} \omega (\tilde {x} _ {t}, a, \mathcal {D} _ {t}) = \sum_ {t = 1} ^ {\tau} \omega (\tilde {x} _ {t}, \pi_ {t} (\tilde {x} _ {t}), \mathcal {D} _ {t}).
$$
We relate the sum of uncertainty radii  $\{\omega (\tilde{x}_t,\pi_t(\tilde{x}_t),\mathcal{D}_t)\}_{t = 1}^T$  with those of the planner  $\{\omega (x_{t},\pi_{t}(x_{t}),\mathcal{D}_{t})\}_{t = 1}^{T}$  via Hoeffding Inequality (see Lemma A.1) and conclude that w.h.p,
$$
\sum_ {t = 1} ^ {T} \max  _ {a \in \mathcal {A}} f _ {\star} (\tilde {x} _ {t}, a) - f _ {\star} (\tilde {x} _ {t}, \tilde {\pi} _ {t} ^ {\mathrm {o p t}} (\tilde {x} _ {t})) \leqslant \sum_ {t = 1} ^ {T} \omega (x _ {t}, \pi_ {t} (x _ {t}), \mathcal {D} _ {t}) + \mathcal {O} \left(B \sqrt {T \log (1 / \delta)}\right).
$$
Lemma 4.3 allows us to bound the sum of these uncertainty radii as
$$
\sum_ {t = 1} ^ {T} \omega \left(x _ {t}, \pi_ {t} \left(x _ {t}\right), \mathcal {D} _ {t}\right) \leqslant \mathcal {O} \left(\min  \left(B T, B d _ {\text {e l u d e r}} (\mathcal {F}, B / T) + \beta_ {\mathcal {F}} (T, \delta) \sqrt {d _ {\text {e l u d e r}} (\mathcal {F} , B / T) T}\right)\right),
$$
and therefore w.h.p,
$$
\sum_ {\ell = 1} ^ {T} \max  _ {a \in \mathcal {A}} f _ {\star} (\tilde {x} _ {\ell}, a) - f _ {\star} (\tilde {x} _ {\ell}, \tilde {\pi} _ {j} ^ {\mathrm {o p t}} (\tilde {x} _ {\ell})) \leqslant \mathcal {O} \left(\min  \left(B T, B d _ {\mathrm {e l u d e r}} (\mathcal {F}, B / T) + \beta_ {\mathcal {F}} (T, \delta) \sqrt {d _ {\mathrm {e l u d e r}} (\mathcal {F} , B / T) T}\right)\right).
$$
Converting this cumulative regret bound into a simple regret one (Lemma B.1) finalizes the result. A detailed version of the proof can be found in Appendix C.1.
5 Uniform Sampling Strategies
In this section we show that a uniform sampling strategy can produce an  $\varepsilon$  optimal policy with probability at least  $1 - \delta$  after collecting  $\mathcal{O}\left(\frac{\max^2(B,\bar{B})|\mathcal{A}|\log(|\mathcal{F}| / \varepsilon\delta)}{\varepsilon^2}\right)$  samples. This procedure achieves the same simple regret rate as converting SquareCB's cumulative regret into simple regret In contrast with the eluder planning algorithm the uniform sampling strategy does not require a planning phase. Instead it consists of running of the Sampler (Algorithm 2) setting  $\widetilde{\pi}_j = \mathrm{Uniform}(\mathcal{A})$  for all  $j\in [T]$ . Given a dataset  $\tilde{\mathcal{D}}_T$  of contexts, actions and rewards collected during the sampling phase, we solve the least squares problem:
$$
\widehat{f}_{T} = \operatorname *{arg  min}_{f\in \mathcal{F}}\sum_{(x_{i},a_{i},r_{i})\in \tilde{\mathcal{D}}_{T}}\left(f(x_{i},a_{i}) - r_{i}\right)^{2}
$$
and define the policy  $\widehat{\pi}_T$  as  $\widehat{\pi}_T(x) = \arg \max_{a\in A}\widehat{f}_T(x,a)$ . The main result of this section is,
The regret bound of the SquareCB algorithm scales as  $\mathcal{O}\left(\sqrt{|\mathcal{A}|\log(\mathcal{F})T\log(T / \delta)}\right)$ .
Theorem 5.1. There exists a universal constant  $\tilde{c} >0$  such that if  $T\geqslant \tilde{c}\frac{\max^2(B,\bar{B})|\mathcal{A}|\log(|\mathcal{F}| / \varepsilon\delta)}{\varepsilon^2}$  then with probability at least  $1 - \delta$  the uniform planning algorithm's policy  $\hat{\pi}_T$  is  $\varepsilon$ -optimal.
The proof of Theorem 5.1 can be found in Appendix D.
Comparison Between OptLS and SquareCB. The regret rate after  $T$  steps of the OptLS algorithm applied to a contextual bandit problem with discrete function class  $\mathcal{F}$  scales (up to logarithmic factors) as  $\mathcal{O}\left(\sqrt{d_{\mathrm{eluder}}(\mathcal{F},B / T)\log(|\mathcal{F}| / \delta)T}\right)$ . In contrast, the regret of the SquareCB algorithm satisfies a regret guarantee (up to logarithmic factors) of order  $\mathcal{O}\left(\sqrt{|\mathcal{A}|\log(|\mathcal{F}| / \delta)T}\right)$  where the eluder dimension dependence is substituted by a polynomial dependence on the number of actions  $|\mathcal{A}|$ . When the number of actions is small, or even constant, the regret rate of SquareCB can be much smaller than that of OptLS. The opposite is true when the number of actions is large or even infinite. Converting these cumulative regret bounds to simple regret implies the number of samples required to produce an  $\varepsilon$ -optimal policy from the adaptive OptLS policy sequence scales as  $\frac{d_{\mathrm{eluder}}(\mathcal{F},B / T)\log(|\mathcal{F}| / \delta)}{\varepsilon^2}$  whereas for the adaptive SquareCB policy sequence it scales as  $\frac{|\mathcal{A}|\log(|\mathcal{F}| / \delta)}{\varepsilon^2}$ . The results of Theorems 4.4 and 5.1 recover these rates in the experiment planning setting.
6 Gap Between Experiment Planning and Adaptive Learning
The results of Sections 4 and 5 imply planning bounds that are comparable to the corresponding online-to-batch guarantees for OptLS [42] and SquareCB [17]. The main result of this section Theorem 6.1 shows there are problems where the number of samples required for experiment planning can be substantially larger than the number of samples required of an adaptive learning algorithm. This result implies the suboptimality of algorithms such as SquareCB and OptLS for adaptive learning.
In order to state our results we consider an action set  $\mathcal{A}_{\mathrm{tree}}$  indexed by the nodes of a height  $L$  binary tree defined here as having  $L$  levels and  $2^{L} - 1$  nodes.
We call  $a_{l,i}$  the  $i$ -th action of the  $l$ -th level of the tree. For an example see Figure  $\square$ . Let  $\varepsilon > 0$ . We define a function class  $\mathcal{F}_{\mathrm{tree}}$  indexed by paths from the root node to a leaf. For any such path  $\mathbf{p} = \{a_{1,1}, a_{2,i_2}, \dots, a_{L,i_L}\}$  the function  $f^{(\mathbf{p})}$  equals,
Figure 1: Binary Tree
$$
f ^ {(\mathbf {p})} (a) = \left\{ \begin{array}{l l} 1 & \text {i f} a = a _ {L, i _ {L}} \\ 1 - 2 \varepsilon & \text {i f} a \in \mathbf {p} \backslash \{a _ {L, i _ {L}} \} \\ 1 - 1 2 \varepsilon & \text {o . w .} \end{array} \right.
$$
The following result fleshes out a separation between planning and adaptive learning with
action set  $\mathcal{A}_{\mathrm{tree}}$  and function class  $\mathcal{F}_{\mathrm{tree}}$  in the setting where at time  $T$  the learner will produce a guess for the optimal policy  $\hat{\pi}_T$ .
Theorem 6.1. Let  $\varepsilon >0,T\in \mathbb{N}$ . Consider the action set  $\mathcal{A}_{\mathrm{tree}}$  and function class  $\mathcal{F}_{\mathrm{tree}}$  and a reward noise process such that  $\xi_t\sim \mathcal{N}(0,1)$  conditionally for all  $t\in [T]$ . For any planning algorithm Alg there is a function  $f_{\star}\in \mathcal{F}_{\mathrm{tree}}$  such that when  $T\leqslant \frac{2^{L - 5}}{9\varepsilon^2}$  and Alg interacts with  $f_{\star}$  then,
$$
\mathbb {E} _ {\operatorname {A l g}, f _ {\star}} \left[ \mathbb {E} _ {a \sim \widehat {\pi} _ {T}} \left[ f _ {\star} (a) \right] \right] <   \max  _ {a \in \mathcal {A} _ {\text {t r e e}}} f _ {\star} (a) - \varepsilon .
$$
Moreover, there is an adaptive algorithm  $\mathrm{Alg}_{\mathrm{adaptive}}$  such that if  $T \geqslant \frac{2L\log(2L / \varepsilon)}{\varepsilon^2}$ ,
$$
\mathbb {E} _ {\operatorname {A l g} _ {\text {a d a p t i v e}}} \left[ \mathbb {E} _ {a \sim \hat {\pi} _ {T}} [ f (a) ] \right] \geqslant \max  _ {a \in \mathcal {A} _ {\text {t r e e}}} f (a) - \varepsilon .
$$
for all  $f \in \mathcal{F}_{\mathrm{tree}}$ . Where  $\mathbb{E}_{\widetilde{\mathrm{Alg}},\widetilde{f}}[\cdot]$  is the expectation over the randomness of  $\widetilde{\mathrm{Alg}}$  and the environment for target function  $\widetilde{f}$ , and  $\hat{\pi}_T$  is the algorithm's final policy guess after the sampling phase.
<sup>5</sup>We obviate the  $T$  dependence on  $d_{\mathrm{eluder}}$  for readability.
The main insight behind Theorem 6.1 is that adaptive strategies to find an optimal action in the  $\mathcal{F}_{\mathrm{tree}}$  function class can make use of the conditional structure of the action space by trying to determine a path of actions from the root to the leaf containing the optimal action. An adaptive strategy can determine this path by querying only nodes that are adjacent to it. In contrast, a static experiment planning strategy cannot leverage this structure and has to query all leaf nodes. Theorem 6.1 implies a gap between the adaptive and experiment planning problems. Moreover, since the eluder dimension of  $\mathcal{F}_{\mathrm{tree}}$  scales with  $2^{L}$  (see Lemma D.1), OptLS and SquareCB are suboptimal adaptive algorithms for this model class. In contrast with the  $\mathcal{O}\left(\frac{2L\log(2L / \varepsilon)}{\varepsilon^2}\right)$  upper bound in Theorem 6.1, converting the cumulative regret bounds of OptLS and SquareCB yield guarantees scaling as  $\mathcal{O}\left(\frac{2^L\log(|\mathcal{F}_{\mathrm{tree}}|)}{\varepsilon^2}\right)$ .
7 Model Selection
In this section we consider a setting where the learner has access to  $\mathcal{F}_1, \dots, \mathcal{F}_M$  function (model) classes all with domain  $\mathcal{X} \times \mathcal{A}$ . The learner is promised there exists an index  $i_\star$  such that  $f_\star \in \mathcal{F}_{i_\star}$ . The value of index  $i_\star$  is not known by the learner. We will show the uniform sampling strategy has a sample complexity that scales logarithmically with the number of models and with the complexity of the optimal model class  $|\mathcal{F}_{i_\star}|$ . In this setting the Sampler will collect two uniform actions datasets  $(\widetilde{\mathcal{D}}_T^{(\mathrm{Train})}, \widetilde{\mathcal{D}}_T^{(\mathrm{Test})})$  of size  $T/2$  each. Using the "train" dataset  $\widetilde{\mathcal{D}}_T^{(\mathrm{Train})}$  the learner computes candidate reward models  $\widehat{f}_T^{(i)}$  for all  $i \in [M]$  by solving the least squares problems:
$$
\widehat{f}_{T}^{(i)} = \operatorname *{arg  min}_{f\in \mathcal{F}_{i}}\sum_{\substack{(x_{i},a_{i},r_{i})\in \widehat{\mathcal{D}}_{T}^{\mathrm{(Train)}}}}\big(f(x_{i},a_{i}) - r_{i}\big)^{2}.
$$
Using the "test" dataset the learner extracts a guess for the optimal model class index  $\underline{\mathrm{i}}$  by solving,
$$
\dot {\mathbf {i}} = \underset {i \in [ M ]} {\arg \min } \sum_ {(x _ {\ell}, a _ {\ell}, r _ {\ell}) \in \tilde {\mathcal {D}} _ {T} ^ {(\mathrm {T e s t})}} \left(\hat {f} _ {T} ^ {(i)} \left(x _ {\ell}, a _ {\ell}\right) - r _ {\ell}\right) ^ {2}. \tag {7}
$$
The candidate policy  $\widehat{\pi}_T$  is defined as  $\widehat{\pi}_T(x) = \arg \max_{a\in A}\widehat{f}_T^{(\underline{1})}(x,a)$ . Let's start by relating the expected least squares loss of the candidate model  $\widehat{f}_T^{(\underline{1})}$  to the size of the optimal model class  $|\mathcal{F}_{i_\star}|$ , Proposition 7.1. There exists a universal constant  $\underline{c} >0$  such that,
$$
\mathbb {E} _ {x \sim \mathcal {P}, a \sim \mathrm {U n i f o r m} (\mathcal {A})} \left[ \left(f _ {\star} (x, a) - \widehat {f} _ {T} ^ {\underline {{i}}} (x, a)\right) ^ {2} \right] \leqslant \frac {c \max ^ {2} (B , \bar {B}) \log (T \max (M , | \mathcal {F} _ {i _ {\star}} |) / \delta)}{T}.
$$
With probability at least  $1 - \delta$ .
The proof of Proposition 7.1 can be found in Appendix E.1 The uniform sampling model-selection algorithm has the following performance guarantees,
Theorem 7.2. There exists a universal constant  $\tilde{c}^{\prime} > 0$  s.t. if  $T\geqslant \tilde{c}^{\prime}\frac{\max^{2}(B,\bar{B})|\mathcal{A}|\log\left(\frac{\max(M_{,}|F_{\star}|)}{\varepsilon\delta}\right)}{\varepsilon^{2}}$  then with probability at least  $1 - \delta$ , the candidate policy  $\hat{\pi}_T$  of the uniform sampling model-selection algorithm is  $\varepsilon$ -optimal.
Proof. Due to the realizability assumption  $(r_t = f_\star(x_t, a_t) + \xi_t)$  the instantaneous regret of  $\widehat{\pi}_T$  on context  $x \in \mathcal{X}$  equals  $\max_a f_\star(x, a) - f_\star(x, \widehat{\pi}_T(x))$ . Let  $\pi_\star(x) = \arg \max_{a \in \mathcal{A}} f_\star(x, a)$ . Just like in the proof of Theorem 5.1 (see Equation 25), we can relate the suboptimality of  $\widehat{\pi}_T$  with the expected least squares loss of  $\widehat{f}_T^{(1)}$  under the uniform policy,
$$
\mathbb {E} _ {x \sim \mathcal {P}} \left[ \max  _ {a \in \mathcal {A}} f _ {\star} (x, a) - f _ {\star} (x, \hat {\pi} _ {T} (x)) \right] \leqslant 2 \sqrt {| \mathcal {A} | \mathbb {E} _ {x \sim \mathcal {P} , a \sim \text {U n i f o r m} (\mathcal {A})} \left[ \left(f _ {\star} (x , a) - \hat {f} _ {T} ^ {(\dot {\mathrm {i}})} (x , a)\right) ^ {2} \right]}. \tag {8}
$$
Finally Proposition 7.1 implies thre is a universal constant  $\underline{c} >0$  such that,
$$
\mathbb {E} _ {x \sim \mathcal {P}, a \sim \mathrm {U n i f o r m} (\mathcal {A})} \left[ \left(f _ {\star} (x, a) - \widehat {f} _ {T} ^ {(\underline {{\mathbf {i}}})} (x, a)\right) ^ {2} \right] \leqslant \frac {\underline {{\mathsf {c}}} (\bar {B} ^ {2} + B ^ {2}) \log (T \max (M , | \mathcal {F} _ {i _ {\star}} |) / \delta)}{T}
$$
with probability at least  $1 - \delta$ . Plugging this result back into Equation 8 we see the suboptimality of  $\widehat{\pi}_T$  can be upper bounded as,
$$
\mathbb {E} _ {x \sim \mathcal {P}} \left[ \max _ {a \in \mathcal {A}} f _ {\star} (x, a) - f _ {\star} (x, \widehat {\pi} _ {T} (x)) \right] \leqslant 2 \sqrt {\frac {\underline {{\mathsf {c}}} | \mathcal {A} | (\bar {B} ^ {2} + B ^ {2}) \log (T \max (M , | \mathcal {F} _ {i _ {\star}} |) / \delta)}{T}}
$$
Setting  $g(T) = 4\frac{\mathcal{C}|\mathcal{A}|(\bar{B}^2 + B^2)\log(T\max(M,|\mathcal{F}_{i_\star}|) / \delta)}{T}$  in Lemma A.6 implies there exists a universal constant  $\tilde{c}' > 0$  such that  $g(T) \leqslant \varepsilon^2$  for all  $T \geqslant \tilde{c}' \frac{\max^2(B,\bar{B})|\mathcal{A}|\log\left(\frac{\max(M,|\mathcal{F}_{i_\star}|)}{\varepsilon\delta}\right)}{\varepsilon^2}$ .
The results of Theorem 7.2 can be best understood by contrasting them to the uniform sampling algorithm with input model class equal to the union of the function classes  $\mathcal{F}_{\mathrm{all}} = \cup_{i\in [M]}\mathcal{F}_i$ . Applying the results of Theorem 5.1 to  $\mathcal{F}_{\mathrm{all}}$  yields a sample complexity scaling with  $\max_{i\in [M]}\log (|\mathcal{F}|_i)$ , a quantity that could be much larger than  $\log (|\mathcal{F}_{i_\star}|)$ . In contrast, the uniform sampling model-selection algorithm achieves a sample complexity scaling with  $\log (|\mathcal{F}_{i_\star}|)$  at a price logarithmic in the number of models classes  $M$ . This logarithmic dependence on  $M$  stands apart from model selection algorithms for cumulative regret scenarios such as Corral [5, 39], ECE [30] and RegretBalancing [38, 12] that instead have a polynomial dependence on  $M$ . The uniform sampling model-selection algorithm is agnostic to the value of  $\varepsilon$ . The results of Theorem 7.2 hold for any  $\varepsilon$ . If  $\varepsilon$  is known in advance the learner can compute the model class index  $\hat{i} = \arg \min \left\{i\in [M]\text{s.t.} T\geqslant \tilde{c}\frac{\max^2(B,\bar{B})|\mathcal{A}|\log(|\mathcal{F}_i| / \varepsilon\delta)}{\varepsilon^2}\right\}$  and use the uniform sampling strategy for  $\mathcal{F}_{\hat{i}}$ . For this choice of  $\varepsilon$ , Theorem 5.1 guarantees similar bounds to those of Theorem 7.2 Unfortunately in contrast with the uniform sampling model-selection algorithm this method would be valid for a single choice of  $\varepsilon$ .