Title: Experiment Planning with Function Approximation
Venue: NeurIPS 2023
Accepted: yes
Keywords: regret

model selection

planning

static

lower bound
Abstract: We study the problem of experiment planning with function approximation in contextual bandit problems. In settings where there is a significant overhead to deploying adaptive algorithms---for example, when the execution of the data collection policies is required to be distributed, or a human in the loop is needed to implement these policies---producing in advance a set of policies for data collection is paramount. We study the setting where a large dataset of contexts but not rewards is available and may be used by the learner to design an effective data collection strategy. Although when rewards are linear this problem has been well studied, results are still missing for more complex reward models. In this work we propose two experiment planning strategies compatible with function approximation. The first is an eluder planning and sampling procedure that can recover optimality guarantees depending on the eluder dimension of the reward function class. For the second, we show that a uniform sampler achieves competitive optimality rates in the setting where the number of actions is small. We finalize our results introducing a statistical gap fleshing out the fundamental differences between planning and adaptive learning and provide results for planning with model selection.

Experiment Planning with Function Approximation

Abstract

We study the problem of experiment planning with function approximation in contextual bandit problems. In settings where there is a significant overhead to deploying adaptive algorithms-for example, when the execution of the data collection policies is required to be distributed, or a human in the loop is needed to implement these policies-producing in advance a set of policies for data collection is paramount. We study the setting where a large dataset of contexts but not rewards is available and may be used by the learner to design an effective data collection strategy. Although when rewards are linear this problem has been well studied [53], results are still missing for more complex reward models. In this work we propose two experiment planning strategies compatible with function approximation. The first is an eluder planning and sampling procedure that can recover optimality guarantees depending on the eluder dimension [42] of the reward function class. For the second, we show that a uniform sampler achieves competitive optimality rates in the setting where the number of actions is small. We finalize our results introducing a statistical gap fleshing out the fundamental differences between planning and adaptive learning and provide results for planning with model selection.

1 Introduction

Data-driven decision-making algorithms have achieved impressive empirical success in various domains such as online personalization [4, 48], games [36, 43], dialogue systems [32] and robotics [25, 35]. In many of these decision-making scenarios, it is often advantageous to consider contextual information when making decisions. This recognition has sparked a growing interest in studying adaptive learning algorithms in the setting of contextual bandits [26, 33, 6] and reinforcement learning (RL) [46]. Adaptive learning scenarios involve the deployment of data collection policies, where learners observe rewards or environment information and utilize this knowledge to shape subsequent data collection strategies. Nonetheless, the practical implementation of adaptive policies in realworld experiments currently presents significant challenges. First, there is significant infrastructure requirements and associated overheads which require skills and resources many organizations lack. For example, while there are end-user services that enable organizations to automatically send different text messages to different individuals, such services typically do not offer adaptive bandit algorithms. Second, the resulting reward signal may be significantly delayed. As an example, the effect of personalized health screening reminders on a patient making a doctors appointments may take weeks. Therefore, while it is increasingly recognized that there exist other settings where contextspecific policies are likely to be beneficial, including behavioural science[8], many organizations working in these settings may find it infeasible to run experiments with adaptive policies. To bridge this gap, there is a need to explore the deployment of non-adaptive or static strategies that can effectively collect data with minimal or no updates. Surprisingly, limited research has been conducted to investigate this particular setting, highlighting the importance of further exploration in this area.

In this work, we consider how to design a static experimental sampling strategy for contextual multi-armed bandit settings which, when executed, will yield a dataset from which we can compute a near-optimal contextual policy (one with small simple regret). Our framework closely adheres to the experiment planning problem formulation originally introduced by [53]. The problem involves a learner interacting with a contextual bandit problem where the reward function is unknown. Our assumption is that the learner possesses a substantial offline dataset comprising of m sample contexts, none of which include reward information. The learner's objective is to utilize this data to design a static policy sequence of length T (where m ě T ) that enables the collection of valuable reward information when deployed in the real world. The ultimate goal is to produce an almost optimal policy using the data gathered during deployment. To address this scenario, the authors of [53] propose a two-phase approach involving Planner and Sampler algorithms. In the planner phase, the learner employs the context data to generate a set of sampling policies, which are then utilized in the sampler phase. The primary focus of [53] lies in the analysis of the linear case, assuming the reward function to be a linear function of the context vectors. Their key algorithmic contribution is the linear Planner Algorithm, which encompasses a reward-free instance of LinUCB . The analysis presented in [53] demonstrates that the required number of samples under the static sampler to achieve an ε -optimal policy (also referred to as simple regret) scales as O ` d 2 { ε 2 ˘ , where d represents the dimension of the underlying space. This result matches the online minimax sample complexity for the simple regret problem [11, 1].

The algorithm proposed by [53] effectively constructs a static policy sequence of adequate span, utilizing the linear structure of the problem. However, in many problem settings, linearity alone may not be sufficient to capture the true nature of the underlying reward model. This limitation is particularly evident in scenarios such as genetic perturbation experimentation [40] or other similar contexts. In such cases, the availability of algorithms that do not rely on linearity becomes crucial. Unfortunately, extending the results and techniques presented in [53] to the generic function approximation regime is not straightforward.

Here we consider when the reward function is realized by an unknown function f ‹ belonging to a known function class F (which can be more complex than linear).

Adaptive Learning with Function Approximation. In adaptive learning settings where a learner can change the data collection policies as it observes rewards and has much more flexibility than in experiment planning scenarios, various adaptive learning procedures compatible with generic function approximation have been proposed for contextual bandit problems. Among these, two significant methods relevant to our discussion are the Optimistic Least Squares algorithm ( OptLS ) introduced by [42] and the SquareCB algorithm introduced by [17]. Both of these methods offer guarantees for cumulative regret. Specifically, the cumulative regret of OptLS scales as O p a d eluder log p| F |q T q , while the cumulative regret of SquareCB scales as O ´ a | A | log p| F |q T ¯ ,

Contributions. In this paper, we address the function approximation setting within the contextual bandit (static) experiment planning problem. Although in experiment planning the data collection policies have to be produced before interacting with the environment, we establish that surprisingly the adaptive ε -simple regret rates achieved by OptLS and SquareCB are also attainable by a static experiment sampling strategy. To achieve this, we introduce and analyze two experimental planning algorithms. The PlannerEluder algorithm (see Section 4) utilizes confidence intervals derived from least squares optimization to construct a static set of policies to be executed during the sampling phase. When the algorithm has access to a sufficient number of offline samples to generate T ' Ω ` d eluder log p| F |q{ ε 2 ˘ static policies, the learner can produce an ε -optimal policy denoted as p π T thus matching the online-to-batch ε -simple regret rates of OptLS . Since the eluder dimension of linear function classes of dimension d is -up to logarithmic factors- of order O p d q , our results where A corresponds to the set of actions. The eluder dimension 1 ( d eluder ) is a statistical complexity measure introduced by [42], which enables deriving guarantees for adaptive learning algorithms based on the principle of optimism in the face of uncertainty in contextual bandits and reinforcement learning [31, 22, 37, 9]. By employing an online-to-batch conversion strategy, these two methods imply that the number of samples required to achieve ε -simple regret using an adaptive algorithm is at most O ` d eluder log p| F |q{ ε 2 ˘ and O ` | A | log p| F |q{ ε 2 ˘ respectively.

recover the linear experiment planning rates of [53]. Additionally, in Section 5, we demonstrate that collecting T ' Ω ` | A | log p| F |q{ ε 2 ˘ uniform samples is sufficient to obtain an ε -optimal policy p π T . This matches the online-to-batch ε -simple regret rates of SquareCB . These two results yield conclusions similar to those known in the linear setting [53], but for general function classes.

These results prompt two important questions: (1) are existing adaptive cumulative regret algorithms already optimal for simple regret minimization, and (2) is static experiment planning sufficient to match the simple regret guarantees of adaptive learning for realizable contextual bandits?

In Section 6, we provide a negative answer to both questions. We present a certain structured class of bandit problems where a different adaptive algorithm can require significantly less samples than that implied by the bounds from OptLS and SquareCB . Our result is for the realizable setting [2, 18, 17, 44, 9], where the true reward function f ‹ belongs to a known function class F , where the true complexity of simple regret problems in adaptive learning scenarios has not be known known in this setting. This result complements recent results [34] that online-to-batch strategies are suboptimal for minimizing simple regret in the agnostic contextual bandits setting. 2

For the second question, our statistical lower bound shows that for this class of problems, a significantly smaller number of samples are needed to find an ε -optimal policy when using adaptive learning, than when using a static policy. This result complements related work in reinforcement learning with realizable linear state-action value function approximation[52].

Finally, we address the problem of model selection when the learner is presented with a family of reward function classes t F i u i M ' 1 and is guaranteed that the true reward function f ‹ belongs to F i ‹ , where the index i ‹ is unknown. We demonstrate that as long as T ' Ω ` | A | log p max p M, | F i ‹ |qq{ ε 2 ˘ uniform samples are available, it is possible to construct an ε -optimal policy denoted as p π T . Further details regarding these results can be found in Section 7.

2 Further Related Work

Best Arm Identification and Design of Experiments. Previous work to efficiently produce an almost optimal policy for non-contextual linear and multi-armed bandits settings is based on best-arm identification procedures [16, 20, 13, 45, 47, 51]. These are strategies that react adaptively to the collected rewards and achieve instance dependent sample complexity. Other papers in the design of experiments literature [24, 15, 29, 41] introduce methods for designing a non-adaptive policy that can be used to find an optimal arm with high probability in non-contextual scenarios.

Safe Optimal Design. When safety concerns are imposed in the exploration policy, works such as [54] and [49] have studied the problem of producing a safe and efficient static policy used to collect data that can then be used for policy evaluation in the MAB and linear settings. This is in contrast with the unconstrained contextual function approximation scenario we study.

Reward Free Reinforcement Learning. In Reinforcement Learning settings a related line of research [21, 50, 10] considers the problem of producing enough exploratory data to allow for policy optimization for all functions in a reward class. These works allow for the learner to interact with the world for a number of steps, enough to collect data that will allow for zero-shot estimation of the optimal policy within a reward family. This stands in contrast with the experiment planning setup, where the objective is to produce a sequence of policies for static deployment at test time with the objective of learning the unknown reward function.

3 Problem Definition

We study a two-phase interaction between a learner and its environment consisting of a Planning and a Sampling phase. During the planning phase the learner has access to m ě T i.i.d. offline

context samples 3 from a context distribution P supported on a context space X as well as knowledge of a reward function class F where each element f P F has domain X ˆ A where A is the action set. During the sampling phase the learner interacts with a contextual bandit problem for T steps where at each time-step t a context x t ' P is revealed to the learner, the learner takes an action a t and receives a reward r t . We adopt the commonly used realizability assumption, which is widely employed in the contextual bandit literature [2, 18, 17, 44, 9].

Assumption 3.1 (Realizability) . There exists a function f ‹ P F such that r t ' f ‹ p x t , a t q ` ξ t where f ‹ P F and ξ t is a conditionally zero mean 1 ´ subgaussian random variable for all t P r T s during the sampling phase.

During the planning phase the learner is required to build a static policy sequence t π i u T i ' 1 (a policy is a mapping from X to ∆ A , the set of distributions over actions) that, without knowledge of the reward, can be deployed to collect data during the sampling phase. In contrast with adaptive learning procedures, in the experiment planning problem the policies in t π i u T i ' 1 have to be fully specified during the planning phase and cannot depend on the learner's observed reward values during the sampling phase. The learner's objective is to use the reward data collected during the sampling phase to produce an ε ´ optimal policy π T such that, for a suboptimality parameter ε ą 0 . Because we have assumed realizability of the reward function, the optimal policy among all possible policies π ‹ ' arg max π E x ' P ,a ' π p x q r f ‹ p x, a qs can be written as π ‹ p x q ' arg max a P A f ‹ p x, a q for all x P X .

Throughout this work we make the following boundedness assumptions,

Assumption 3.2 (Bounded Function Range) . The range of F is bounded, i.e. for all x, a P X ˆ A , | f p x, a q| ď B for some constant B ą 0 .

Assumption 3.3 (Bounded Noise) . The noise variables are bounded | ξ t | ď ¯ B for all t P r T s for some constant ¯ B ą 0 .

Assumptions 3.2 and 3.3 imply | r t | ď B ` ¯ B for all t P r T s .

4 Planning Under the Eluder Dimension

The eluder dimension is a sequential notion of complexity for function classes that was originally introduced by [42]. The eluder dimension can be used to bound the regret of optimistic least squares algorithms in contextual bandits [42, 9] and reinforcement learning [37]. Informally speaking the eluder dimension is the length of the longest sequence of points one must observe to accurately estimate the function value at any other point. We use the eluder dimension definition from [42].

Definition 4.1. ( ε ´ dependence) Let G be a scalar function class with domain Z and ε ą 0 . An element z P Z is ε ´ dependent on t z 1 , ¨ ¨ ¨ , z n u Ď Z w.r.t. G if any pair of functions g, g 1 P G satisfying a ř n i ' 1 p g p z i q ´ g 1 p z i qq 2 ď ε also satisfies g p z q ´ g 1 p z q ď ε . Furthermore, z P Z is ε ´ independent of t z 1 , ¨ ¨ ¨ , z n u w.r.t. G if it is not ε ´ dependent on t z 1 , ¨ ¨ ¨ , z n u .

By definition the ε ´ eluder dimension increases as ε is driven down to zero. Let D be a dataset of X , A pairs. For any f, f 1 with support X ˆ A we use the notation } f ´ f 1 } D to denote the data norm of the difference between functions f and f 1 :

Definition 4.2. ( ε -eluder) The ε ´ non monotone eluder dimension Ğ d eluder p G , ε q of G is the length of the longest sequence of elements in Z such that every element is ε ´ independent of its predecessors. Moreover, we define the ε ´ eluder dimension d eluder p G , ε q as d eluder p G , ε q ' max ε 1 ě ε Ğ d eluder p G , ε q .

Algorithm 1 EluderPlanner

1: Input: m ě T samples t x /lscript ' P u m /lscript ' 1 , function class F , confidence radius function β F : t, δ Ñ R .

3: for t ' 1 . . . T do

2: Initialize data buffer D 1 ' H

4: For any x P X define the uncertainty radius of action a P A as,

5: Define the policy π t as π t p x q ' arg max a P A ω p x, a, D t q for all x P X .

7: end for

6: Update D t ` 1 Ð D t Ytp x t , π t p x t qqu

Algorithm 2 Sampler

1: Input: number of time-steps T .

2: Initialize data buffer r D 1 ' H 3: for t ' 1 . . . T do

4: Define deployment policy ˜ π t as π t .

5: Observe context ˜ x t ' P .

7: Update r D t ` 1 Ð r D t Ytp ˜ x t , ˜ a t , ˜ r t qu . 8: end for

The EluderPlanner algorithm takes as input m ě T i.i.d. samples from the context distribution t x /lscript ' P u m /lscript ' 1 and a realizable function class F satisfying Assumption 3.1. The learner iterates over T out of these m samples in a sequential manner. We use the name x t to denote the t ´ th input sample and call π t to the (deterministic) exploration policies produced upon processing samples 1 , ¨ ¨ ¨ , t ´ 1 . We use the notation D t ' tp x /lscript , π /lscript p x /lscript qqu t ´ 1 /lscript ' 1 to denote the dataset composed of the first t ´ 1 (ordered) samples from t x /lscript u m /lscript ' 1 and actions from policies

6: Play action ˜ a t ' ˜ π t p ˜ x t q and receive reward ˜ r t .

t π /lscript u t ´ 1 /lscript ' 1 . We adopt the convention that D 1 ' H . The output of the EluderPlanner algorithm is the sequence of policies t π t u T t ' 1 . Policy π t is defined as π t p x q ' arg max a P A ω p x, a, D t q such that ω p x, a, D t q is an uncertainty measure defined as,

Where the confidence radius function β F p δ , t q equals

for some universal constant ¯ C ą 0 defined in Proposition A.7. This is a complexity radius implied by the guarantees of least squares regression (Lemma A.8) and constraints imposed by Lemma 4.6. See Appendix A.1 and Proposition A.7 for a detailed discussion about this definition. We call the combination of the EluderPlanner and Sampler as the eluder planning algorithm . Its performance guarantees are obtained by relating the regret of a constructed sequence of optimistic policies t r π opt t u T t ' 1 based on the data collected by t π t u T t ' 1 to the sum of uncertainty radii ř T t ' 1 ω p x t , π t p x t q , D t q of the context-actions collected during the planning phase. Using optimism when constructing this policy sequence is important. In contrast with the sequence of greedy policies π greedy t p¨| x q ' arg max a P A p f t p x, a q , where p f t ' arg min f P F ř p x,a,r qP r D t p f p x, a q ´ r q 2 resulting from a least squares over r D t , the sequence of optimistic policies satisfies a regret bound. Thus, playing a uniform policy from the sequence t r π opt t u T t ' 1 satisfies a simple regret guarantee. The sum of uncertainty radii ř T t ' 1 ω p x t , π t p x t q , D t q is bounded using a Lemma we borrow from [9].

Lemma 4.3. [Lemma 3 from [9]] Let F be a function class satisfying Assumption 3.2 with ε -eluder dimension d eluder p F , ε q . For all T P N and any dataset sequence t ¯ D t u 8 t ' 1 with ¯ D 1 ' H and ¯ D t ' tt ¯ x /lscript , ¯ a /lscript uu t ´ 1 /lscript ' 1 of context-action pairs, the following inequality on the sum of the uncertainty radii holds,

Extracting Policy p π T from Data. At the end of the execution of the Sampler algorithm, the learner will have access to a sequence of datasets t r D t u T t ' 1 of contexts, actions and reward triplets. The learner will use this sequence of datasets to compute the sequence of regression functions,

Standard Least Squares (LS) results (see Lemma A.8) imply that } p f t ´ f ‹ } ˜ D t ď β p t, δ q with high probability for all t P N . These confidence sets are used to define a sequence of optimistic (deterministic) policies t π opt t u T t ' 1 :

Theorem 4.4. Let ε ą 0 . There exists a universal constant c ą 0 such that if

The candidate optimal policy p π T is then defined as p π T ' Uniform p r π opt 1 , ¨ ¨ ¨ , r π opt T q . The main result in this section is the following Theorem.

then with probability at least 1 ´ δ the eluder planning algorithm's policy p π T is ε ´ optimal . Remark 4.5. The results of this section hold for the structured bandits setting [23, 27]; that is, scenarios where X ' tHu .

Both sides of the inequality defining T in Theorem 4.4 depend on T . Provided d eluder p F , B { T q is sublinear as a function of T , there exists a finite value of T satisfying Equation 5. When d eluder p F , ¯ ε q ' ¯ d eluder log p 1 { ¯ ε q for some ¯ d eluder and for all ¯ ε ą 0 setting T ě c max 2 p B, ¯ B, 1 q d eluder p F ,B ε q log p| F |{ εδ q ε 2 is enough to guarantee the conditions of Theorem 4.4 are satisfied. Examples 4 and 5 from [42] show the eluder dimension of linear or generalized linear function classes can be written in such way. In these cases ¯ d eluder is a constant multiple of the underlying dimension of the space.

Using standard techniques, the results of Theorem 4.4 can be adapted to the case when the function class F is infinite but admits a metrization and has a covering number. In this case the sample complexity will scale not with log p| F |q but instead with the logarithm of the covering number of F . For example, in the case of linear functions, the logarithm of the covering number of F under the /lscript 2 norm will scale as d , the ambient dimension of the space up to logarithmic factors of T (see for example Lemma D.1 of [14]). Plugging this into the sample guarantees of Theorem 4.4 recovers the r O ` d 2 { ε 2 ˘ sample complexity for linear experiment planning from [53].

4.1 Proof Sketch of Theorem 4.4

Let ˜ x 1 , ¨ ¨ ¨ , ˜ x T be the sampler's context sequence. The proof works by showing that with probability at least 1 ´ δ the regret of the t π opt t p¨qu T t ' 1 sequence satisfies the regret bound

A key part of the proof involves relating the balls defined by the data norms of the planner datasets t D t u T t ' 1 and those induced by the data norms of the sampler datasets t r D t u T t ' 1 . The precise statement of this relationship is provided in Lemma 4.6, and its proof can be found in Appendix C.

Lemma 4.6. With probability at least 1 ´ δ 12 ,

Simultaneously for all t P r T s . Where t D t u T t ' 1 is the dataset sequence resulting of the execution of EluderPlanner while t r D t u T t ' 1 is the dataset sequence resulting of the execution of the Sampler and β F p δ , t q is the confidence radius function defined in Equation 2.

Let p f t as in Equation 3 and define E as the event where the Standard Least Squares results (see Lemma A.8) hold i.e. } p f t ´ f ‹ } r D t ď β F p t, δ q for all t P r T s and also Equation 6 from Lemma 4.6 holds for all t P r T s . The results of Lemmas A.8 and 4.6 imply P p E q ě 1 ´ δ 6 .

For context x and action a let's denote by ˜ f x,a t as a function achieving the inner maximum in the definition of ˜ π opt t (see Equation 4). When E holds the policies t ˜ π opt t u T t ' 1 are optimistic in the sense that ˜ f x, ˜ π opt t p x q t p x, ˜ π opt t p x qq ě max a P A f ‹ p x, a q and therefore,

We relate the sum of uncertainty radii t ω p ˜ x t , π t p ˜ x t q , D t qu T t ' 1 with those of the planner t ω p x t , π t p x t q , D t qu T t ' 1 via Hoeffding Inequality (see Lemma A.1) and conclude that w.h.p,

Lemma 4.3 allows us to bound the sum of these uncertainty radii as

and therefore w.h.p,

Converting this cumulative regret bound into a simple regret one (Lemma B.1) finalizes the result. A detailed version of the proof can be found in Appendix C.1.

5 Uniform Sampling Strategies

In this section we show that a uniform sampling strategy can produce an ε optimal policy with probability at least 1 ´ δ after collecting O ´ max 2 p B, ¯ B q| A | log p| F |{ εδ q ε 2 ¯ samples. This procedure achieves the same simple regret rate as converting SquareCB 's cumulative regret into simple regret 4 .

In contrast with the eluder planning algorithm the uniform sampling strategy does not require a planning phase. Instead it consists of running of the Sampler (Algorithm 2) setting r π j ' Uniform p A q for all j P r T s . Given a dataset r D T of contexts, actions and rewards collected during the sampling phase, we solve the least squares problem:

r and define the policy π T as π T p x q ' arg max a P A p f T p x, a q . The main result of this section is,

Theorem 5.1. There exists a universal constant ˜ c ą 0 such that if T ě ˜ c max 2 p B, ¯ B q| A | log p| F |{ εδ q ε 2 then with probability at least 1 ´ δ the uniform planning algorithm's policy p π T is ε ´ optimal. The proof of Theorem 5.1 can be found in Appendix D.

Comparison Between OptLS and SquareCB . The regret rate after T steps of the OptLS algorithm applied to a contextual bandit problem with discrete function class F scales 5 (up to logarithmic factors) as O ´ a d eluder p F , B { T q log p| F |{ δ q T ¯ . In contrast, the regret of the SquareCB algorithm satisfies a regret guarantee (up to logarithmic factors) of order O ´ a | A | log p| F |{ δ q T ¯ where the eluder dimension dependence is substituted by a polynomial dependence on the number of actions | A | . When the number of actions is small, or even constant, the regret rate of SquareCB can be much smaller than that of OptLS . The opposite is true when the number of actions is large or even infinite. Converting these cumulative regret bounds to simple regret implies the number of samples required to produce an ε ´ optimal policy from the adaptive OptLS policy sequence scales as d eluder p F ,B { T q log p| F |{ δ q ε 2 whereas for the adaptive SquareCB policy sequence it scales as | A | log p| F |{ δ q ε 2 . The results of Theorems 4.4 and 5.1 recover these rates in the experiment planning setting.

6 Gap Between Experiment Planning and Adaptive Learning

The results of Sections 4 and 5 imply planning bounds that are comparable to the corresponding online-to-batch guarantees for OptLS [42] and SquareCB [17]. The main result of this section Theorem 6.1 shows there are problems where the number of samples required for experiment planning can be substantially larger than the number of samples required of an adaptive learning algorithm. This result implies the suboptimality of algorithms such as SquareCB and OptLS for adaptive learning.

In order to state our results we consider an action set A tree indexed by the nodes of a height L binary tree defined here as having L levels and 2 L ´ 1 nodes.

a

1

,

1

a

2

,

1

a

3

,

1

a

3

,

2

a

2

,

2

a

3

,

3

a

3

,

4

We call a l,i the i -th action of the l -th level of the tree. For an example see Figure 1. Let ε ą 0 . We define a function class F tree indexed by paths from the root node to a leaf. For any such path p ' t a 1 , 1 , a 2 ,i 2 , ¨ ¨ ¨ , a L,i L u the function f p p q equals,

The following result fleshes out a separation between planning and adaptive learning with action set A tree and function class F tree in the setting where at time T the learner will produce a guess for the optimal policy π T .

p Theorem 6.1. Let ε ą 0 , T P N . Consider the action set A tree and function class F tree and a reward noise process such that ξ t ' N p 0 , 1 q conditionally for all t P r T s . For any planning algorithm Alg there is a function f ‹ P F tree such that when T ď 2 L ´ 5 9 ε 2 and Alg interacts with f ‹ then,

Moreover, there is an adaptive algorithm Alg adaptive such that if T ě 2 L log p 2 L { ε q ε 2 ,

for all f P F tree . Where E Ą Alg , r f r¨s is the expectation over the randomness of Ą Alg and the environment for target function r f , and p π T is the algorithm's final policy guess after the sampling phase. 5 We obviate the T dependence on d eluder for readability.

The main insight behind Theorem 6.1 is that adaptive strategies to find an optimal action in the F tree function class can make use of the conditional structure of the action space by trying to determine a path of actions from the root to the leaf containing the optimal action. An adaptive strategy can determine this path by querying only nodes that are adjacent to it. In contrast, a static experiment planning strategy cannot leverage this structure and has to query all leaf nodes. Theorem 6.1 implies a gap between the adaptive and experiment planning problems. Moreover, since the eluder dimension of F tree scales with 2 L (see Lemma D.1), OptLS and SquareCB are suboptimal adaptive algorithms for this model class. In contrast with the O ´ 2 L log p 2 L { ε q ε 2 ¯ upper bound in Theorem 6.1, converting the cummulative regret bounds of OptLS and SquareCB yield guarantees scaling as O ´ 2 L log p| F tree |q ε 2 ¯ .

7 Model Selection

In this section we consider a setting where the learner has access to F 1 , ¨ ¨ ¨ , F M function (model) classes all with domain X ˆ A . The learner is promised there exists an index i ‹ such that f ‹ P F i ‹ . The value of index i ‹ is not known by the learner. We will show the uniform sampling strategy has a sample complexity that scales logarithmically with the number of models and with the complexity of the optimal model class | F i ‹ | . In this setting the Sampler will collect two uniform actions datasets p r D p Train q T , r D p Test q T q of size T { 2 each. Using the 'train" dataset r D p Train q T the learner computes candidate reward models f p i q T for all i P r M s by solving the least squares problems:

Using the 'test" dataset the learner extracts a guess for the optimal model class index i by solving,

The candidate policy p π T is defined as p π T p x q ' arg max a P A p f p i q T p x, a q . Let's start by relating the expected least squares loss of the candidate model p f p i q T to the size of the optimal model class | F i ‹ | , Proposition 7.1. There exists a universal constant c ą 0 such that,

With probability at lest 1 ´ δ .

The proof of Proposition 7.1 can be found in Appendix E.1. The uniform sampling model-selection algorithm has the following performance guarantees,

Theorem 7.2. There exists a universal constant r c 1 ą 0 s.t. if T ě ˜ c 1 max 2 p B, ¯ B q| A | log p max p M, | F ‹|q εδ q ε 2 then with probability at least 1 ´ δ , the candidate policy p π T of the uniform sampling model-selection algorithm is ε ´ optimal.

Proof. Due to the realizability assumption p r t ' f ‹ p x t , a t q ` ξ t q the instantaneous regret of p π T on context x P X equals max a f ‹ p x, a q ´ f ‹ p x, p π T p x qq . Let π ‹ p x q ' arg max a P A f ‹ p x, a q . Just like in the proof of Theorem 5.1 (see Equation 25), we can relate the suboptimality of p π T with the expected least squares loss of f p i q T under the uniform policy,

Finally Proposition 7.1 implies thre is a universal constant c ą 0 such that,

with probability at least 1 ´ δ . Plugging this result back into Equation 8 we see the suboptimality of p π T can be upper bounded as,

The results of Theorem 7.2 can be best understood by contrasting them to the uniform sampling algorithm with input model class equal to the union of the function classes F all ' Y i Pr M s F i . Applying the results of Theorem 5.1 to F all yields a sample complexity scaling with max i Pr M s log p| F | i q , a quantity that could be much larger than log p| F i ‹ |q . In contrast, the uniform sampling model-selection algorithm achieves a sample complexity scaling with log p| F i ‹ |q at a price logarithmic in the number of models classes M . This logarithmic dependence on M stands apart from model selection algorithms for cumulative regret scenarios such as Corral [5, 39], ECE [30] and RegretBalancing [38, 12] that instead have a polynomial dependence on M . The uniform sampling model-selection algorithm is agnostic to the value of ε . The results of Theorem 7.2 hold for any ε . If ε is known in advance the learner can compute the model class index p i ' arg min ! i P r M s s.t. T ě ˜ c max 2 p B, ¯ B q| A | log p| F i |{ εδ q ε 2 ) and use the uniform sampling strategy for F p i . For this choice of ε , Theorem 5.1 guarantees similar bounds to those of Theorem 7.2. Unfortunately in contrast with the uniform sampling model-selection algorithm this method would be valid for a single choice of ε .

8 Conclusion

In this work we have introduced the first set of algorithms for the experiment planning problem for contextual bandits with general function approximation. We have developed the EluderPlanning algorithm that produces a static policy sequence that after deployment can be used to recover an ε ´ optimal policy. We showed it is enough for the number of static policies and therefore samples during the sampling phase to be as large as the number of samples required from an adaptive procedure based on an online-to-batch conversion of the OptLS algorithm. Similarly we also demonstrated the uniform sampling strategy enjoys the same online-to-batch conversion sample complexity as the SquareCB algorithm. These results seemingly suggest that simple regret rates for adaptive learning may also be achieved in experiment planning scenarios. We show this is not the case. There exist structured bandit problems for which adaptive learning may require a number of samples that is substantially smaller than the number of samples required by a static policy sequence. This is significant because it implies the suboptimality of the rates achieved by existing adaptive learning algorithms such as OptLS and SquareCB and also because it draws a clear distinction between adaptive learning and experiment planning. This implies these algorithms are either suboptimal or their upper bound analysis is not tight. We believe the first to be correct. This is an important open question we hope to see addressed in future research. We have also introduced the first model selection results for the experiment planning problem.

Many important questions remain regarding this setting. Chief among them is to characterize the true statistical complexity of experimental design for contextual bandits with general function approximations. Our results indicate the eluder dimension is not the sharpest statistical complexity measure to characterize learning here. Developing a more new form of complexity, as well as an accompanying algorithm that can achieve the true statistical lower bound for the problem of experiment planning remains an exciting and important open question to tackle in future research.

Acknowledgments and Disclosure of Funding

We thank Akshay Krishnamurthy for helpful discussions. Aldo Pacchiano would like to thank the support of the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard. This work was supported in part by funding from the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard. This work was supported in part by NSF grant #2112926.

References

Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24:2312-2320, 2011.

A. Agarwal, M. Dudík, S. Kale, J. Langford, and R. Schapire. Contextual bandit learning with predictable rewards. In Artificial Intelligence and Statistics , pages 19-26. PMLR, 2012.

A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646. PMLR, 2014.

A. Agarwal, S. Bird, M. Cozowicz, L. Hoang, J. Langford, S. Lee, J. Li, D. Melamed, G. Oshri, O. Ribas, et al. Making contextual decisions with low technical debt. arXiv preprint arXiv:1606.03966 , 2016.

A. Agarwal, H. Luo, B. Neyshabur, and R. E. Schapire. Corralling a band of bandit algorithms. In Conference on Learning Theory , pages 12-38. PMLR, 2017.

S. Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013.

A. Beygelzimer, J. Langford, L. Li, L. Reyzin, and R. Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 19-26. JMLR Workshop and Conference Proceedings, 2011.

C. J. Bryan, E. Tipton, and D. S. Yeager. Behavioural science is unlikely to change the world without a heterogeneity revolution. Nature human behaviour , 5(8):980-989, 2021.

J. Chan, A. Pacchiano, N. Tripuraneni, Y. S. Song, P. Bartlett, and M. I. Jordan. Parallelizing contextual bandits. arXiv preprint arXiv:2105.10590 , 2021.

J. Chen, A. Modi, A. Krishnamurthy, N. Jiang, and A. Agarwal. On the statistical efficiency of reward-free exploration in non-linear rl. Advances in Neural Information Processing Systems , 35:20960-20973, 2022.

W. Chu, L. Li, L. Reyzin, and R. Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214, 2011.

A. Cutkosky, C. Dann, A. Das, C. Gentile, A. Pacchiano, and M. Purohit. Dynamic balancing for model selection in bandits and rl. In International Conference on Machine Learning , pages 2276-2285. PMLR, 2021.

R. Degenne, P. Ménard, X. Shang, and M. Valko. Gamification of pure exploration for linear bandits. In International Conference on Machine Learning , pages 2432-2442. PMLR, 2020.

S. S. Du, S. M. Kakade, J. D. Lee, S. Lovett, G. Mahajan, W. Sun, and R. Wang. Bilinear classes: A structural framework for provable generalization in rl. arXiv preprint arXiv:2103.10897 , 2021.

H. Esfandiari, A. Karbasi, A. Mehrabian, and V. Mirrokni. Regret bounds for batched bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7340-7348, 2021.

T. Fiez, L. Jain, K. G. Jamieson, and L. Ratliff. Sequential experimental design for transductive linear bandits. Advances in neural information processing systems , 32, 2019.

D. Foster and A. Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning , pages 3199-3210. PMLR, 2020.

D. Foster, A. Agarwal, M. Dudik, H. Luo, and R. Schapire. Practical contextual bandits with regression oracles. In International Conference on Machine Learning , pages 1539-1548. PMLR, 2018.

D. A. Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

Y. Jedra and A. Proutiere. Optimal best-arm identification in linear bandits. Advances in Neural Information Processing Systems , 33:10007-10017, 2020.

C. Jin, A. Krishnamurthy, M. Simchowitz, and T. Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pages 4870-4879. PMLR, 2020.

C. Jin, Q. Liu, and S. Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems , 34: 13406-13418, 2021.

K.-S. Jun and C. Zhang. Crush optimism with pessimism: Structured bandits beyond asymptotic optimality. Advances in Neural Information Processing Systems , 33:6366-6376, 2020.

J. Kiefer and J. Wolfowitz. The equivalence of two extremum problems. Canadian Journal of Mathematics , 12:363-366, 1960.

J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.

J. Langford and T. Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Advances in neural information processing systems , 20(1):96-1, 2007.

T. Lattimore and R. Munos. Bounded regret for finite-armed structured bandits. Advances in Neural Information Processing Systems , 27, 2014.

T. Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

T. Lattimore, C. Szepesvari, and G. Weisz. Learning with good feature representations in bandits and in rl with a generative model. In International Conference on Machine Learning , pages 5662-5670. PMLR, 2020.

J. Lee, A. Pacchiano, V. Muthukumar, W. Kong, and E. Brunskill. Online model selection for reinforcement learning with function approximation. In International Conference on Artificial Intelligence and Statistics , pages 3340-3348. PMLR, 2021.

G. Li, P. Kamath, D. J. Foster, and N. Srebro. Understanding the eluder dimension. Advances in Neural Information Processing Systems , 35:23737-23750, 2022.

J. Li, W. Monroe, A. Ritter, M. Galley, J. Gao, and D. Jurafsky. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541 , 2016.

L. Li, W. Chu, J. Langford, and R. E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Z. Li, L. Ratliff, K. G. Jamieson, L. Jain, et al. Instance-optimal pac algorithms for contextual bandits. Advances in Neural Information Processing Systems , 35:37590-37603, 2022.

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

I. Osband and B. Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems , 27, 2014.

A. Pacchiano, C. Dann, C. Gentile, and P. Bartlett. Regret bound balancing and elimination for model selection in bandits and rl. arXiv preprint arXiv:2012.13045 , 2020.

A. Pacchiano, M. Phan, Y. Abbasi Yadkori, A. Rao, J. Zimmert, T. Lattimore, and C. Szepesvari. Model selection in contextual stochastic bandit problems. Advances in Neural Information Processing Systems , 33:10328-10337, 2020.

A. Pacchiano, D. Wulsin, R. A. Barton, and L. Voloch. Neural design for genetic perturbation experiments. arXiv preprint arXiv:2207.12805 , 2022.

Y. Ruan, J. Yang, and Y. Zhou. Linear bandits with limited adaptivity and learning distributional optimal design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 74-87, 2021.

D. Russo and B. Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

D. Simchi-Levi and Y. Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research , 2021.

M. Soare, A. Lazaric, and R. Munos. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems , 27, 2014.

R. S. Sutton. Introduction: The challenge of reinforcement learning. In Reinforcement Learning , pages 1-3. Springer, 1992.

C. Tao, S. Blanco, and Y. Zhou. Best arm identification in linear bandits with linear dimension dependency. In International Conference on Machine Learning , pages 4877-4886. PMLR, 2018.

A. Tewari and S. A. Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile Health: Sensors, Analytic Methods, and Applications , pages 495-517, 2017.

R. Wan, B. Kveton, and R. Song. Safe exploration for efficient policy evaluation and comparison. In International Conference on Machine Learning , pages 22491-22511. PMLR, 2022.

R. Wang, S. S. Du, L. Yang, and R. R. Salakhutdinov. On reward-free reinforcement learning with linear function approximation. Advances in neural information processing systems , 33: 17816-17826, 2020.

L. Xu, J. Honda, and M. Sugiyama. A fully adaptive algorithm for pure exploration in linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 843-851. PMLR, 2018.

A. Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl. In International Conference on Machine Learning , pages 12287-12297. PMLR, 2021.

A. Zanette, K. Dong, J. N. Lee, and E. Brunskill. Design of experiments for stochastic contextual linear bandits. Advances in Neural Information Processing Systems , 34:22720-22731, 2021.

R. Zhu and B. Kveton. Safe optimal design with applications in off-policy learning. In International Conference on Artificial Intelligence and Statistics , pages 2436-2447. PMLR, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of static experiment planning under the function approximation setting, which contrasts with the adaptive setting where the reward is not observable during the static planning phase. The paper proposes two static planning strategies: (1) EluderPlanner based on Eluder Dimension, and (2) Naive uniform sampler, and analyzes their regret, achieving a cumulative regret similar to the OLS and SquareCB algorithms in their adaptive counterparts.\n\nThe paper takes a step further by designing a special tree-structured function class that highlights the gap between the proposed static planning algorithms and the lower bound of adaptive learning in terms of regret.\n\nFinally, the paper addresses the problem of model selection and demonstrates the possibility of constructing an $\\varepsilon$-optimal policy.\n\nSTRENGTHS:\nClear motivation and good writing:\n- I enjoyed reading the motivational example in the introduction that explains why adaptive learning scenarios can be challenging.\n- All the problem setups and the relations with the related literature are clearly explained.\n- The structure of the paper is good and easy to follow.\n- Assumptions are well explained, and the definition of Eluder dimension is helpful.\n\nNovelty:\n- Combining Eluder dimension and static experiment planning appears to be a novel approach.\n- The lower bound construction in Section 5 is informative and demonstrates a fundamental gap between static planning and adaptive learning.\n\nTheoretical soundness:\n- I read through the main text, and all the derivations seem correct to me.\n\nWEAKNESSES:\nRelated work on reward-free RL: I found reward-free RL to be closely related, especially [1], which also describes reward-free RL under function approximation. It would be good to have a short paragraph discussing reward-free RL in general.\n\n[1] Shuang Qiu, Jieping Ye, Zhaoran Wang, Zhuoran Yang. \"On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Game.\"\n\nQUESTIONS:\nI am not familiar with Eluder dimension, but I find the construction of the function class in Section 5 extremely interesting. Will there be a similar gap for some of our familiar non-linear function classes, such as neural networks or kernel spaces, between static planning and adaptive learning?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of experiment planning with function approximation in contextual bandit problems. \nIt is intended to solve the scenario that the datasets include a large amount of contexts while no rewards.\nThe authors propose two experiment planning strategies that are compatible with function approximation: an eluder planning and sampling procedure. The theoretical results guarantee that the policy converges to optimially under the time constrained by the eluder dimension.\n\nSTRENGTHS:\nThis algorithm achieves in this setting: the reward signal is not abundant while there are a large amount of the contexts. The uniform experiment planning algorithms help solve the setting. \n\nTwo important questions are answered in this problem:\n\n(1)are existing adaptive cumulative regret algorithms already optimal for simple regret minimization? \n\n(2) is static experiment planning sufficient to match the simple regret guarantees of adaptive learning for realizable contextual bandits?\n\nWEAKNESSES:\n1.The paper does not provide a detailed comparison with existing experiment planning algorithms to show the improvement of the algorithm, especially in the function approximation like linear function. Enough related works and comparisons are needed in this paper.\n\n2.The main results and the proof is lack of proof intuition to highlight the keypoint. The results need a clear and intuitive proof sketch to help readers follow the idea.\nThe results are lack of a lower bound to guarantee the optimality of the algorithm. A tight bound is needed. If not, the gap of the lower bound and the upper bound is needed to state.\n\n3.The paper could benefit from a more detailed discussion of the limitations of the proposed approach.\n\nQUESTIONS:\n1. Is there a specific experiment to show the performance of the algorithm?\n\nLIMITATIONS:\nNo. The paper could benefit from a more detailed discussion of the limitations of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of planning for efficient data collection. Given initial data with a lot of contexts but no reward, the question is how do you devise a policy for data collection such that when executed in the real world, it learns the reward optimally to finally learn a policy with maximum rewards.\n\nThey propose two methods for it, i) eluder planning and a sampling procedure \nii) uniform sampler for the case when the number of actions is small\n\nSTRENGTHS:\nProposed algorithms for experiment planning and theoretically analyze them. They also showed suboptimality of the existing adaptive learning algorithms which is interesting.\n\nLine 138-139, the intuition of the eluder dimension helps to understand its use case\n\nWEAKNESSES:\nThe paper was hard for me to grab fully. Partially I lack background but majorly, I think it was the writing part. Some ideas to improve them:\n1) Either go with a short, vague intro or a precise long intro: currently, it is a long but vague intro since the problem setting was not defined early, and it becomes difficult to understand. I have seen theoretical papers which define problem statements in the intro, and then it's a precise long intro.\n\n2) Since you continuously refer to the sampling and planning phase, may be use a block diagram to represent them.\n\nIn general, I believe the writing style can be improved.\n\nSecondly, I was wondering if it would make sense to also add at least a preliminary experiment showcasing your theory? It is hard to get intuition on how tight the bounds are.\n\nQUESTIONS:\nIntuitively when is it possible to have $d_{eluder}(F, B/T)$ a sublinear function in T? \n\nI am not able to interpret theorem 4.1 properly, on what factors does $\\tilde{c}$ depends. Consider writing it as: let T be the smallest integer satisfying the $T\\geq....$, then \\for all $T' \\geq T$ $\\pi_{T'}$ is optimal?\n\nline 94,166: typos\n\nLIMITATIONS:\nno negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe study focuses on the experiment planning problem for contextual bandits with function approximation. \nThe paper gives two algorithms for the problem: (1) EluderPlanning algorithm whose sample complexity depends on the elder dimension of the function class and matches the sample complexity of OLS algorithm by a online-to-batch conversion.\n(2) It also suggest that simple uniform sampling is good when action space is small, and the sample complexity matches the SquareCB algorithm by online-to-batch conversion.\n\nThen, the paper suggest that there is an exponential gap between adaptive sampling and passive sampling (experimental planning) by finding a class of tree-structured function. Besides, for this class of function, the paper show that SquareCB and OLS (two adaptive sampling algorithm) have exponential large sample complexity while there exist an algorithm with reliability (called adaptive tree sampling in appendix) can achieve polynomial sample complexity in this tree-structured function class. This suggest the sub-optimality of SquareCB and OLS.\n\nFinally, address the problem of model selection when the learner is presented with a family of reward function classes and the true reward function is inside one of the reward function classes.\n\nSTRENGTHS:\nThe tree-structured function class constructed in section 5 is interesting and elegant. \nWith this function class, the paper point out an exponential separation between adaptive sampling algorithm. \nAlso, it points out that in realisable setting, the online-to-batch conversion of SquareCB and OLS is suboptimal for simple regret minimization.\n\nWEAKNESSES:\n(1) The proof of Lemma A.4 has a *major flaw*:\nin Line 439, it is not valid to choose $\\eta_t$ as a random variable that is dependent on the martingale difference sequence. \nThis is because the moment generating function (MGF) is required to derive lemma A.3 and thus also for lemma A.4.\nIt is valid only when the MGF coefficient $\\eta$ is a fixed value, which is also reflected in the claim of lemma A.3 (eta is fixed outside the probability bound). \nMore generally, MGF coefficient $\\eta$ can be a random variable that is independent of whole martingale difference sequence and then distribution of sum of martingale difference sequence is not affected by conditioning on $\\eta$.\n\n(2) Because of (1) and Lemma A.4 serves a fundamental step for most theoretical claims in appendix A, section 3 and section 6. All claims related to lemma A.4 may not be correct, including but not limited to Lemma A.5, Lemma A.7, Lemma A.8, Lemma 3.6, Theorem 3.4 and Proposition 6.1.\n\n(3) There are a few typos that would affect the understanding of the technical details.\n\n(4) Even if all claims can be corrected, the main contribution is not clean in the paper where too many issues are discussed.\n\nQUESTIONS:\n(1) See weaknesses. If all problems can be correctly resolved, I would consider increasing the score.\n\n(2) There is a claim \"Our results indicate the eluder dimension is not the sharpest statistical complexity measure to characterize learning in this setting.\" in line 326, but I don't find any support to this claim.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}

comments: SUMMARY:
We study the static experiment planning for policy learning problem in contextual bandits, with focus on the general realizable case. The paper first presents an algorithm using reward free, extending similar ideas in [37] and leveraging the  eluder dimension. The paper then shares a few theoretical results, including the competitive guarantee for uniform sampling policy, the gap between static and adaptive policies in a special case, and the results in model selection.

STRENGTHS:
* The paper is technically sound. 
* The writing is very clear.
* The paper studies an important problem and the first proposed algorithm is useful.

WEAKNESSES:
Numerical results on the finite sample performance would be important to have. For example, although the asymptotic rate for uniform sampling is competitive and this fact might be surprising, it is expected that its finite-sample performance is worse. 

It is also not fully clear how significant these theoretical findings are. For example, it is expected that there is some performance gap between adaptive and static policies, when the task is hard (though I appreciate formally proving it).

QUESTIONS:
1. It would be helpful to add outputs to your Algorithm tables
2. In the main text, it would help to discuss the connection to [37] in terms of the reward-free emulation idea. 
3. "Extracting Policy from Data" - the intuition for this part is hard to understand. Why uniform sampling + optimistic, instead of just using the greedy (or pessimistic) policy?

LIMITATIONS:
Extensions to safe exploration and/or policy evaluation tasks would be meaningful and worthy some discussions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-14