SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Broad Institute & Boston University apacchia@broadinstitute.org
Jonathan N. Lee
Stanford University jnl@stanford.edu
Emma Brunskill
Stanford University ebrun@cs.stanford.edu
We study the problem of experiment planning with function approximation in contextual bandit problems. In settings where there is a significant overhead to deploying adaptive algorithms—for example, when the execution of the data collection policies is required to be distributed, or a human in the loop is needed to implement these policies—producing in advance a set of policies for data collection is paramount. We study the setting where a large dataset of contexts but not rewards is available and may be used by the learner to design an effective data collection strategy. Although when rewards are linear this problem has been well studied [53], results are still missing for more complex reward models. In this work we propose two experiment planning strategies compatible with function approximation. The first is an eluder planning and sampling procedure that can recover optimality guarantees depending on the eluder dimension [42] of the reward function class. For the second, we show that a uniform sampler achieves competitive optimality rates in the setting where the number of actions is small. We finalize our results introducing a statistical gap fleshing out the fundamental differences between planning and adaptive learning and provide results for planning with model selection.
Data-driven decision-making algorithms have achieved impressive empirical success in various domains such as online personalization [4, 48], games [36, 43], dialogue systems [32] and robotics [25, 35]. In many of these decision-making scenarios, it is often advantageous to consider contextual information when making decisions. This recognition has sparked a growing interest in studying adaptive learning algorithms in the setting of contextual bandits [26, 33, 6] and reinforcement learning (RL) [46]. Adaptive learning scenarios involve the deployment of data collection policies, where learners observe rewards or environment information and utilize this knowledge to shape subsequent data collection strategies. Nonetheless, the practical implementation of adaptive policies in real-world experiments currently presents significant challenges. First, there is significant infrastructure requirements and associated overheads which require skills and resources many organizations lack. For example, while there are end-user services that enable organizations to automatically send different text messages to different individuals, such services typically do not offer adaptive bandit algorithms. Second, the resulting reward signal may be significantly delayed. As an example, the effect of personalized health screening reminders on a patient making a doctors appointments may take weeks. Therefore, while it is increasingly recognized that there exist other settings where context-specific policies are likely to be beneficial, including behavioural science [8], many organizations working in these settings may find it infeasible to run experiments with adaptive policies. To bridge this gap, there is a need to explore the deployment of non-adaptive or static strategies that can effectively collect data with minimal or no updates. Surprisingly, limited research has been conducted to investigate this particular setting, highlighting the importance of further exploration in this area.
In this work, we consider how to design a static experimental sampling strategy for contextual multi-armed bandit settings which, when executed, will yield a dataset from which we can compute a near-optimal contextual policy (one with small simple regret). Our framework closely adheres to the experiment planning problem formulation originally introduced by [53]. The problem involves a learner interacting with a contextual bandit problem where the reward function is unknown. Our assumption is that the learner possesses a substantial offline dataset comprising of  $m$  sample contexts, none of which include reward information. The learner's objective is to utilize this data to design a static policy sequence of length  $T$  (where  $m \geqslant T$ ) that enables the collection of valuable reward information when deployed in the real world. The ultimate goal is to produce an almost optimal policy using the data gathered during deployment. To address this scenario, the authors of [53] propose a two-phase approach involving Planner and Sampler algorithms. In the planner phase, the learner employs the context data to generate a set of sampling policies, which are then utilized in the sampler phase. The primary focus of [53] lies in the analysis of the linear case, assuming the reward function to be a linear function of the context vectors. Their key algorithmic contribution is the linear Planner Algorithm, which encompasses a reward-free instance of LinUCB. The analysis presented in [53] demonstrates that the required number of samples under the static sampler to achieve an  $\varepsilon$ -optimal policy (also referred to as simple regret) scales as  $\mathcal{O}\left(d^{2} / \varepsilon^{2}\right)$ , where  $d$  represents the dimension of the underlying space. This result matches the online minimax sample complexity for the simple regret problem [11, 1].
The algorithm proposed by [53] effectively constructs a static policy sequence of adequate span, utilizing the linear structure of the problem. However, in many problem settings, linearity alone may not be sufficient to capture the true nature of the underlying reward model. This limitation is particularly evident in scenarios such as genetic perturbation experimentation [40] or other similar contexts. In such cases, the availability of algorithms that do not rely on linearity becomes crucial. Unfortunately, extending the results and techniques presented in [53] to the generic function approximation regime is not straightforward.
Here we consider when the reward function is realized by an unknown function  $f_{\star}$  belonging to a known function class  $\mathcal{F}$  (which can be more complex than linear).
Adaptive Learning with Function Approximation. In adaptive learning settings where a learner can change the data collection policies as it observes rewards and has much more flexibility than in experiment planning scenarios, various adaptive learning procedures compatible with generic function approximation have been proposed for contextual bandit problems. Among these, two significant methods relevant to our discussion are the Optimistic Least Squares algorithm (OptLS) introduced by [42] and the SquareCB algorithm introduced by [17]. Both of these methods offer guarantees for cumulative regret. Specifically, the cumulative regret of OptLS scales as  $\mathcal{O}(\sqrt{d_{\mathrm{eluder}}\log(|\mathcal{F}|)T})$ , while the cumulative regret of SquareCB scales as  $\mathcal{O}\left(\sqrt{|\mathcal{A}|\log(|\mathcal{F}|)T}\right)$ , where  $\mathcal{A}$  corresponds to the set of actions. The eluder dimension  $[d_{\mathrm{eluder}}]$  is a statistical complexity measure introduced by [42], which enables deriving guarantees for adaptive learning algorithms based on the principle of optimism in the face of uncertainty in contextual bandits and reinforcement learning [31, 22, 37, 9]. By employing an online-to-batch conversion strategy, these two methods imply that the number of samples required to achieve  $\varepsilon$ -simple regret using an adaptive algorithm is at most  $\mathcal{O}\left(d_{\mathrm{eluder}}\log(|\mathcal{F}|)/\varepsilon^{2}\right)$  and  $\mathcal{O}\left(|\mathcal{A}|\log(|\mathcal{F}|)/\varepsilon^{2}\right)$  respectively.
Contributions. In this paper, we address the function approximation setting within the contextual bandit (static) experiment planning problem. Although in experiment planning the data collection policies have to be produced before interacting with the environment, we establish that surprisingly the adaptive  $\varepsilon$ -simple regret rates achieved by OptLS and SquareCB are also attainable by a static experiment sampling strategy. To achieve this, we introduce and analyze two experimental planning algorithms. The PlannerEluder algorithm (see Section 4) utilizes confidence intervals derived from least squares optimization to construct a static set of policies to be executed during the sampling phase. When the algorithm has access to a sufficient number of offline samples to generate  $T = \Omega \left(d_{\mathrm{eluder}} \log(|\mathcal{F}|) / \varepsilon^2\right)$  static policies, the learner can produce an  $\varepsilon$ -optimal policy denoted as  $\hat{\pi}_T$  thus matching the online-to-batch  $\varepsilon$ -simple regret rates of OptLS. Since the eluder dimension of linear function classes of dimension  $d$  is -up to logarithmic factors- of order  $\mathcal{O}(d)$ , our results
<sup>1</sup>We formally introduce this quantity in Section 4. Here we use a simpler notation to avoid confusion.
recover the linear experiment planning rates of [53]. Additionally, in Section 5, we demonstrate that collecting  $T = \Omega \left( |\mathcal{A}| \log (|\mathcal{F}|) / \varepsilon^2 \right)$  uniform samples is sufficient to obtain an  $\varepsilon$ -optimal policy  $\widehat{\pi}_T$ . This matches the online-to-batch  $\varepsilon$ -simple regret rates of SquareCB. These two results yield conclusions similar to those known in the linear setting [53], but for general function classes.
These results prompt two important questions: (1) are existing adaptive cumulative regret algorithms already optimal for simple regret minimization, and (2) is static experiment planning sufficient to match the simple regret guarantees of adaptive learning for realizable contextual bandits?
In Section 6, we provide a negative answer to both questions. We present a certain structured class of bandit problems where a different adaptive algorithm can require significantly less samples than that implied by the bounds from OptLS and SquareCB. Our result is for the realizable setting [2,18,17,44,9], where the true reward function  $f_{\star}$  belongs to a known function class  $\mathcal{F}$ , where the true complexity of simple regret problems in adaptive learning scenarios has not been known in this setting. This result complements recent results [34] that online-to-batch strategies are suboptimal for minimizing simple regret in the agnostic contextual bandits setting.
For the second question, our statistical lower bound shows that for this class of problems, a significantly smaller number of samples are needed to find an  $\varepsilon$ -optimal policy when using adaptive learning, than when using a static policy. This result complements related work in reinforcement learning with realizable linear state-action value function approximation[52].
Finally, we address the problem of model selection when the learner is presented with a family of reward function classes  $\{\mathcal{F}_i\}_{i=1}^M$  and is guaranteed that the true reward function  $f_\star$  belongs to  $\mathcal{F}_{i_\star}$  where the index  $i_\star$  is unknown. We demonstrate that as long as  $T = \Omega(|\mathcal{A}| \log(\max(M, |\mathcal{F}_{i_\star}|)) / \varepsilon^2)$  uniform samples are available, it is possible to construct an  $\varepsilon$ -optimal policy denoted as  $\widehat{\pi}_T$ . Further details regarding these results can be found in Section 7.
2 Further Related Work
Best Arm Identification and Design of Experiments. Previous work to efficiently produce an almost optimal policy for non-contextual linear and multi-armed bandits settings is based on best-arm identification procedures [16, 20, 13, 45, 47, 51]. These are strategies that react adaptively to the collected rewards and achieve instance dependent sample complexity. Other papers in the design of experiments literature [24, 15, 29, 41] introduce methods for designing a non-adaptive policy that can be used to find an optimal arm with high probability in non-contextual scenarios.
Safe Optimal Design. When safety concerns are imposed in the exploration policy, works such as [54] and [49] have studied the problem of producing a safe and efficient static policy used to collect data that can then be used for policy evaluation in the MAB and linear settings. This is in contrast with the unconstrained contextual function approximation scenario we study.
Reward Free Reinforcement Learning. In Reinforcement Learning settings a related line of research [21, 50, 10] considers the problem of producing enough exploratory data to allow for policy optimization for all functions in a reward class. These works allow for the learner to interact with the world for a number of steps, enough to collect data that will allow for zero-shot estimation of the optimal policy within a reward family. This stands in contrast with the experiment planning setup, where the objective is to produce a sequence of policies for static deployment at test time with the objective of learning the unknown reward function.
We study a two-phase interaction between a learner and its environment consisting of a Planning and a Sampling phase. During the planning phase the learner has access to  $m \geq T$  i.i.d. offline
In the agnostic case the learner is instead presented with an arbitrary policy class  $\Pi$  that may or may not be related to the mean reward function  $f_{\star}$ . In this setting a recent study by [34] introduces an algorithm that achieves sharp rates characterized in terms of a quantity  $\rho_{\Pi}$ .
context samples from a context distribution  $\mathcal{P}$  supported on a context space  $\mathcal{X}$  as well as knowledge of a reward function class  $\mathcal{F}$  where each element  $f\in \mathcal{F}$  has domain  $\mathcal{X}\times \mathcal{A}$  where  $\mathcal{A}$  is the action set. During the sampling phase the learner interacts with a contextual bandit problem for  $T$  steps where at each time-step  $t$  a context  $x_{t}\sim \mathcal{P}$  is revealed to the learner, the learner takes an action  $a_{t}$  and receives a reward  $r_t$ . We adopt the commonly used realizability assumption, which is widely employed in the contextual bandit literature [2, 18, 17, 44, 9].
Assumption 3.1 (Realizability). There exists a function  $f_{\star} \in \mathcal{F}$  such that  $r_t = f_{\star}(x_t, a_t) + \xi_t$  where  $f_{\star} \in \mathcal{F}$  and  $\xi_t$  is a conditionally zero mean 1-subgaussian random variable for all  $t \in [T]$  during the sampling phase.
During the planning phase the learner is required to build a static policy sequence  $\{\pi_i\}_{i=1}^T$  (a policy is a mapping from  $\mathcal{X}$  to  $\Delta_{\mathcal{A}}$ , the set of distributions over actions) that, without knowledge of the reward, can be deployed to collect data during the sampling phase. In contrast with adaptive learning procedures, in the experiment planning problem the policies in  $\{\pi_i\}_{i=1}^T$  have to be fully specified during the planning phase and cannot depend on the learner's observed reward values during the sampling phase. The learner's objective is to use the reward data collected during the sampling phase to produce an  $\varepsilon$ -optimal policy  $\widehat{\pi}_T$  such that,
$$
\mathbb {E} _ {x \sim \mathcal {P}, a \sim \hat {\pi} _ {T} (x)} \left[ f _ {\star} (x, a) \right] + \varepsilon \geqslant \max  _ {\pi} \mathbb {E} _ {x \sim \mathcal {P}, a \sim \pi (x)} \left[ f _ {\star} (x, a) \right].
$$
for a suboptimality parameter  $\varepsilon > 0$ . Because we have assumed realizability of the reward function, the optimal policy among all possible policies  $\pi_{\star} = \arg \max_{\pi} \mathbb{E}_{x \sim \mathcal{P}, a \sim \pi(x)}[f_{\star}(x, a)]$  can be written as  $\pi_{\star}(x) = \arg \max_{a \in \mathcal{A}} f_{\star}(x, a)$  for all  $x \in \mathcal{X}$ .
Throughout this work we make the following boundedness assumptions,
Assumption 3.2 (Bounded Function Range). The range of  $\mathcal{F}$  is bounded, i.e. for all  $x, a \in \mathcal{X} \times \mathcal{A}$ ,  $|f(x, a)| \leqslant B$  for some constant  $B > 0$ .
Assumption 3.3 (Bounded Noise). The noise variables are bounded  $|\xi_t| \leqslant \bar{B}$  for all  $t \in [T]$  for some constant  $\bar{B} > 0$ .
Assumptions 3.2 and 3.3 imply  $|r_t| \leqslant B + \bar{B}$  for all  $t \in [T]$ .
4 Planning Under the Eluder Dimension
The eluder dimension is a sequential notion of complexity for function classes that was originally introduced by [42]. The eluder dimension can be used to bound the regret of optimistic least squares algorithms in contextual bandits [42, 9] and reinforcement learning [37]. Informally speaking the eluder dimension is the length of the longest sequence of points one must observe to accurately estimate the function value at any other point. We use the eluder dimension definition from [42].
Definition 4.1. ( $\varepsilon$ -dependence) Let  $\mathcal{G}$  be a scalar function class with domain  $\mathcal{Z}$  and  $\varepsilon > 0$ . An element  $z \in \mathcal{Z}$  is  $\varepsilon$ -dependent on  $\{z_1, \dots, z_n\} \subseteq \mathcal{Z}$  w.r.t.  $\mathcal{G}$  if any pair of functions  $g, g' \in \mathcal{G}$  satisfying  $\sqrt{\sum_{i=1}^{n}(g(z_i) - g'(z_i))^2} \leqslant \varepsilon$  also satisfies  $g(z) - g'(z) \leqslant \varepsilon$ . Furthermore,  $z \in \mathcal{Z}$  is  $\varepsilon$ -independent of  $\{z_1, \dots, z_n\}$  w.r.t.  $\mathcal{G}$  if it is not  $\varepsilon$ -dependent on  $\{z_1, \dots, z_n\}$ .
Definition 4.2. ( $\varepsilon$ -eluder) The  $\varepsilon$ -non monotone eluder dimension  $\overline{d_{\mathrm{elud}}}_{\mathcal{G}}(\mathcal{G},\varepsilon)$  of  $\mathcal{G}$  is the length of the longest sequence of elements in  $\mathcal{Z}$  such that every element is  $\varepsilon$ -independent of its predecessors. Moreover, we define the  $\varepsilon$ -eluder dimension  $d_{\mathrm{elud}}(\mathcal{G},\varepsilon)$  as  $d_{\mathrm{elud}}(\mathcal{G},\varepsilon) = \max_{\varepsilon' \geqslant \varepsilon} \overline{d_{\mathrm{elud}}}_{\mathcal{G}}(\mathcal{G},\varepsilon)$ .
By definition the  $\varepsilon$ -eluder dimension increases as  $\varepsilon$  is driven down to zero. Let  $\mathcal{D}$  be a dataset of  $\mathcal{X}$ ,  $\mathcal{A}$  pairs. For any  $f, f'$  with support  $\mathcal{X} \times \mathcal{A}$  we use the notation  $\|f - f'\|_{\mathcal{D}}$  to denote the data norm of the difference between functions  $f$  and  $f'$ :
$$
\| f - f ^ {\prime} \| _ {\mathcal {D}} = \sqrt {\sum_ {(x , a) \in \mathcal {D}} \left(f (x , a) - f ^ {\prime} (x , a)\right) ^ {2}}.
$$
This assumption is based on the fact that gathering offline context samples can be significantly less costly compared to conducting multiple rounds of experimental deployment.
Algorithm 1 EluderPlanner
1: Input:  $m \geqslant T$  samples  $\{x_{\ell} \sim \mathcal{P}\}_{\ell=1}^{m}$ , function class  $\mathcal{F}$ , confidence radius function  $\beta_{\mathcal{F}}: t, \delta \to \mathbb{R}$ .
2: Initialize data buffer  $\mathcal{D}_1 = \emptyset$
3: for  $t = 1 \dots T$  do
4: For any  $x \in \mathcal{X}$  define the uncertainty radius of action  $a \in \mathcal{A}$  as,
$$
\omega (x,a,\mathcal{D}_{t}) = \max_{f,f^{\prime}\in \mathcal{F}\text{s.t.} \| f - f^{\prime}\|_{\mathcal{D}_{t}\leqslant 4\beta_{\mathcal{F}}(t,\delta)}}f(x,a) - f^{\prime}(x,a).
$$
5: Define the policy  $\pi_t$  as  $\pi_t(x) = \arg \max_{a\in A}\omega (x,a,\mathcal{D}_t)$  for all  $x\in \mathcal{X}$ .
6: Update  $\mathcal{D}_{t + 1}\gets \mathcal{D}_t\cup \{(x_t,\pi_t(x_t))\}$
Algorithm 2 Sampler
1: Input: number of time-steps  $T$ .
2: Initialize data buffer  $\widetilde{\mathcal{D}}_1 = \varnothing$
3: for  $t = 1 \dots T$  do
4: Define deployment policy  $\tilde{\pi}_t$  as  $\pi_t$ .
5: Observe context  $\tilde{x}_t\sim \mathcal{P}$
6: Play action  $\tilde{a}_t = \tilde{\pi}_t(\tilde{x}_t)$  and receive reward  $\tilde{r}_t$ .
7: Update  $\widetilde{\mathcal{D}}_{t + 1}\gets \widetilde{\mathcal{D}}_t\cup \{(\tilde{x}_t,\tilde{a}_t,\tilde{r}_t)\} .$
8: end for
The EluderPlanner algorithm takes as input  $m\geqslant T$  i.i.d. samples from the context distribution  $\{x_{\ell}\sim \mathcal{P}\}_{\ell = 1}^{m}$  and a realizable function class  $\mathcal{F}$  satisfying Assumption3.1 The learner iterates over  $T$  out of these  $m$  samples in a sequential manner. We use the name  $x_{t}$  to denote the  $t$  -th input sample and call  $\pi_t$  to the (deterministic) exploration policies produced upon processing samples  $1,\dots ,t - 1$  .We use the notation  $\mathcal{D}_t = \{(x_\ell ,\pi_\ell (x_\ell))\}_{\ell = 1}^{t - 1}$  to denote the dataset composed of the first  $t - 1$  (ordered) samples from  $\{x_{\ell}\}_{\ell = 1}^{m}$  and actions from policies
$\{\pi_{\ell}\}_{\ell = 1}^{t - 1}$ . We adopt the convention that  $\mathcal{D}_1 = \varnothing$ . The output of the EluderPlanner algorithm is the sequence of policies  $\{\pi_t\}_{t = 1}^T$ . Policy  $\pi_t$  is defined as  $\pi_t(x) = \arg \max_{a\in \mathcal{A}}\omega (x,a,\mathcal{D}_t)$  such that  $\omega (x,a,\mathcal{D}_t)$  is an uncertainty measure defined as,
$$
\omega (x, a, \mathcal {D} _ {t}) = \max  _ {f, f ^ {\prime} \in \mathcal {F} \text {s . t .} \| f - f ^ {\prime} \| _ {\mathcal {D} _ {t}} \leqslant 4 \beta_ {\mathcal {F}} (t, \delta)} f (x, a) - f ^ {\prime} (x, a). \tag {1}
$$
Where the confidence radius function  $\beta_{\mathcal{F}}(\delta ,t)$  equals
$$
\beta_ {\mathcal {F}} (\delta , t) = \bar {C} (B + \bar {B}) \sqrt {\log \left(\frac {| \mathcal {F} | t}{\delta}\right)} \tag {2}
$$
for some universal constant  $\bar{C} > 0$  defined in Proposition A.7. This is a complexity radius implied by the guarantees of least squares regression (Lemma A.8) and constraints imposed by Lemma 4.6. See Appendix A.1 and Proposition A.7 for a detailed discussion about this definition. We call the combination of the EluderPlanner and Sampler as the eluder planning algorithm. Its performance guarantees are obtained by relating the regret of a constructed sequence of optimistic policies  $\{\widetilde{\pi}_t^{\mathrm{opt}}\}_{t=1}^T$  based on the data collected by  $\{\pi_t\}_{t=1}^T$  to the sum of uncertainty radii  $\sum_{t=1}^T \omega(x_t, \pi_t(x_t), \mathcal{D}_t)$  of the context-actions collected during the planning phase. Using optimism when constructing this policy sequence is important. In contrast with the sequence of greedy policies  $\pi_t^{\mathrm{greedy}}(\cdot|x) = \arg \max_{a \in \mathcal{A}} \hat{f}_t(x, a)$ , where  $\hat{f}_t = \arg \min_{f \in \mathcal{F}} \sum_{(x,a,r) \in \tilde{\mathcal{D}}_t} (f(x,a) - r)^2$  resulting from a least squares over  $\tilde{\mathcal{D}}_t$ , the sequence of optimistic policies satisfies a regret bound. Thus, playing a uniform policy from the sequence  $\{\widetilde{\pi}_t^{\mathrm{opt}}\}_{t=1}^T$  satisfies a simple regret guarantee. The sum of uncertainty radii  $\sum_{t=1}^T \omega(x_t, \pi_t(x_t), \mathcal{D}_t)$  is bounded using a Lemma we borrow from [9].
Lemma 4.3. [Lemma 3 from [9]] Let  $\mathcal{F}$  be a function class satisfying Assumption 3.2 with  $\varepsilon$ -eluder dimension  $d_{\mathrm{eluder}}(\mathcal{F},\varepsilon)$ . For all  $T\in \mathbb{N}$  and any dataset sequence  $\{\bar{D}_t\}_{t = 1}^\infty$  with  $\bar{D}_1 = \varnothing$  and  $\bar{D}_t = \{\{\bar{x}_\ell ,\bar{a}_\ell \} \}_{\ell = 1}^{t - 1}$  of context-action pairs, the following inequality on the sum of the uncertainty radii holds,
$$
\sum_ {t = 1} ^ {T} \omega (\bar {x} _ {t}, \bar {a} _ {t}, \bar {D} _ {t}) \leqslant \mathcal {O} \left(\min  \left(B T, B d _ {\text {e l u d e r}} (\mathcal {F}, B / T) + \beta_ {\mathcal {F}} (T, \delta) \sqrt {d _ {\text {e l u d e r}} (\mathcal {F} , B / T) T}\right)\right).
$$
Extracting Policy  $\widehat{\pi}_T$  from Data. At the end of the execution of the Sampler algorithm, the learner will have access to a sequence of datasets  $\{\widetilde{D}_t\}_{t=1}^T$  of contexts, actions and reward triplets. The learner will use this sequence of datasets to compute the sequence of regression functions,