Design of Experiments for Stochastic Contextual Linear Bandits

1 Introduction . Many settings may substantially benefit from data-driven contextualized decision policies that optimize the desired expected outcome . Online machine learning methods like multi-armed bandits and reinforcement learning , that adaptively change interventions in response to outcomes in a closed loop process ( see Figure 1a ) , may not yet be practical for all domains due to the expertise and infrastructure needed . However running an experiment with a fixed decision policy to identify a good personalized policy is likely to be both simple logistically ( since currently such decision policies are often specified by hand ) and more easily accepted , since many areas ( education , healthcare , social sciences ) commonly deploy experiments across a few conditions to find the best approach . For example , education startups , political campaigns , and governmental agencies can use email and ∗The first three authors contributed equally . The work was fully completed while Andrea Zanette was a PhD Candidate in the Institute for Computational and Mathematical Engineering at Stanford University . Future revision of this article will be made available at https : //arxiv.org/abs/2107.09912 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . text messages to provide targeted information and opportunities , such as information to encourage vaccination , or tips to parents to support their child ’ s developmental stage . Such organizations are familiar with standard experimental design , and but generally lack the infrastructure for continuous online contextual MAB learning . Experiments that involve deploying a nonadaptive policy that fit in with standard workflows but enable data-efficient learning of contextualized policies might offer substantial benefits over AB testing or relying on other segmentation methods that may not directly optimize desired outcomes . For these reasons , a key opportunity is to design static or nonadaptive policies that can be used to gather data to identify optimal contextualized decision policies . Indeed a fixed data collection strategy is practically desirable ( 1 ) whenever multiple agents collect data asynchronously and communication to update their policy is difficult or impossible , and ( 2 ) whenever changing the policy requires a significant overhead either in the engineering infrastructure or in the training of human personnel . Several prior papers limit the number of policy switches with minimal sample complexity impact ( Han et al. , 2020 ; Ruan et al. , 2020 ; Ren et al. , 2020 ; Bai et al. , 2019 ; Wang et al. , 2021 ) . Motivated by the above settings , we look for a single , nonadaptive policy for data collection . Setting and goal We consider the linear stochastic contextual bandit setting where each context s ∈ S is sampled from a distribution µ and a context-dependent action setAs is made available to the learner . The bandit instance is defined by a feature extractor φ ( s , a ) ∈ Rd and some unknown parameter θ ? ∈ Rd . Upon choosing an action a ∈ As , the linear reward function r ( s , a ) = φ ( s , a ) > θ ? +η is revealed to the learner corrupted by mean zero 1-subGaussian noise η . Our goal is to construct an exploration policy πe to gather a dataset , such that after that dataset is gathered , we can extract a near optimal policy π̂ from it . In particular , we want to minimize the number of exploration samples . Perhaps surprisingly , there has been relatively little work on this setting . Prior work on exploration to quickly identify a near-optimal policy focuses on best-arm identification using adaptive policies that react to the observed rewards ( Soare et al. , 2014 ; Tao et al. , 2018 ; Jedra and Proutiere , 2020 ) or design of experiments that produces a nonadaptive policy for data collection ( Kiefer and Wolfowitz , 1960 ; Esfandiari et al. , 2019 ; Lattimore and Szepesvari , 2020 ) ; both lines of work assume that a single , repeated context with unchanging action set is presented to the learner . In contrast we are interested in identifying near-optimal context-specific decision policies . The closest related work ( Ruan et al. , 2020 ) investigates our task as a subtask for online regret learning , but requires an amount of data that scales as Ω ( d16 ) , which is impractical in applications with even moderate d.2 Without any apriori information , no algorithm can do much better than deploying a random policy , which can require an amount of data that scales exponentially in d , see appendix G.1 . However , in many common applications , prior data about the context distribution µ ( s ) and the state–action feature representation φ is available . For example , an organization will often know information about its customers and specify the feature representation used for state–action spaces in advance.3 The initially available state contexts are referred to as offline ( state ) contexts data . Our algorithm leverages historical context data to enable data efficient design of experiments for stochastic linear contextual bandits . It uses offline context-only data C to design a nonadaptive policy πe to collect new , online data where reward feedback is observed ( see Figure 1b ) , and uses the resulting dataset D′ to learn a near-optimal on average decision policy π̂ for future use . We highlight that the algorithm does not get to adjust the exploratory policy πe while the online data is being collected . Contributions We make the following contributions . • Using past state contexts only , we design a single , nonadaptive policy to acquire online data that can be used to compute a context-dependent decision policy that is near-optimal in expectation across the contexts with high probability , for future use . • To identify an -optimal policy , our algorithm achieves the minimax lower bound Ω ( min { d log ∑ sAs , d2 ) } / 2 ) on the number of online samples ( ignoring log and constants ) , while keeping the number of offline state contexts required polynomially small ( O ( d2/ 2 ) or O ( d3/ 2 ) ) . 2Note that the number of offline data in ( Ruan et al. , 2020 ) is independent of 1/ . But for most of the practical settings , our bound d3/ 2 can be much smaller than d16 . 3In other words , given a set of previously observed states s1 , . . . , sM , and a known state–action representation φ , for any potential action a , we can compute the resulting representation φ ( s , a ) . • Our experiments on a simulation and on a learning to rank Yahoo dataset show our strategic design of experiments approach can learn better decision policies with less exploration data compared to standard static exploration strategies that fail to exploit structure . 2 Related Work . Most linear bandit papers focus on online regret . Research on sample complexity results has focused primarily on adaptive algorithms for non-contextual bandits with a simple or vectored action space ( Soare et al. , 2014 ; Tao et al. , 2018 ; Xu et al. , 2018 ; Fiez et al. , 2019 ; Jedra and Proutiere , 2020 ; Degenne et al. , 2020 ) . Their sample complexity bounds depend on the suboptimality gap of the action set , and are therefore , instance-dependent . The design of experiments literature ( Kiefer and Wolfowitz , 1960 ; Esfandiari et al. , 2019 ; Lattimore and Szepesvari , 2020 ; Ruan et al. , 2020 ) has also focused on non-contextual bandits , but in designing a non-adaptive policy such that the resulting dataset can be used to identify an optimal arm with high probability . A few recent papers consider learning the optimal policy for contextual bandits in the pure exploration setting ( Deshmukh et al. , 2018 ; Ruan et al. , 2020 ) . Deshmukh et al . ( 2018 ) propose an algorithm that yields an online reactive policy for when the reward is generated by a RKHS function on the input space : in contrast we focus on nonadaptive policies for linear contextual bandits . There has been recent interest in reward-free reinforcement learning in Markov decision processes ( e.g . ( Jin et al. , 2020 ; Wang et al. , 2020a ) ) . This work is focused on online adaptive policies . Jin et al . ( 2020 ) ’ s algorithm is similar in structure to our ’ s and could likely be adapted to the offline , bandit setting , but they focus on tabular domains . Note for simple contextual multi-armed bandits ( tabular states , no shared structure , Jin et al. ’ s setting with H=1 ) , a good strategy in the non-adaptive setting is to simply sample actions uniformly ; indeed this is optimal given the lack of shared information between state-actions . No sophisticated algorithm is needed in that case . For linear bandits , the closest reward-free work we are aware of is work on RL with linear function approximationWang et al . ( 2020a ) . While their algorithm can be specialized to the linear bandit setting and modified to match our algorithm by using offline data , the theoretical analysis would need to be very different from that in ( Wang et al. , 2020b ) ’ s paper ( and thus it would not yield the results presented in our paper ) . A key challenge in our case is to compute a non-adaptive policy for gathering data and be able to ensure the covariance matrix of the resulting data will be very similar to the covariance matrix that could result by a simulated adaptive policy . This issue does not arise in ( Wang et al. , 2020b ) because they focus on reactive policies.4 The most related paper to this work is by Ruan et al . ( 2020 ) . The main focus of Ruan et al . ( 2020 ) is to minimize the number of policy switches in the online phase while keeping the expected regret optimal . Directly applying Theorem 6 of Ruan et al . ( 2020 ) yields a Õ ( d2/ 2 ) online sample complexity and Õ ( d16 ) offline sample complexity . Their offline sample complexity is independent of . As a trade-off , Ruan et al . ( 2020 ) have a highly suboptimal dependence on d , which leads to burnin phase too large to be practically useful ; their algorithm also suffers from a high computational complexity . Note their work is focused on a different objective than our ’ s . 4In particular , we will shortly introduce a planner-sampler two stage algorithm . We decouple the planner from the sampler to identify a non-reactive policy . Due to the decoupling , the planner needs to accurately predict what will happen when the sampler is run . To ensure this , a key challenge is to ensure that the planner inverse covariance matrix is very similar to the sampler inverse covariance matrix . 3 Setup . We consider the stochastic linear contextual bandit model with stochastic contexts . A bandit instance is characterized by a tuple 〈S , A , µ , r〉 where S is the context space and µ is the context distribution . For a context s ∈ S , the action space is denoted by As . The feature map φ : ( s , a ) 7→ φ ( s , a ) ∈ Rd is assumed to be known and defines the linear reward model r ( s , a ) = φ ( s , a ) > θ ? + η for some θ ? ∈ Rd parameter and some mean-zero 1-subgaussian random variable η . We occasionally use the Õ notation to suppress polylog factors of the input parameters d , λ , 1δ . We write f . g if f = O ( g ) , and f / g if f = Õ ( g ) . We say f = Ω ( g ) if there exists a positive constant c such that f ≥ cg . For a positive semi-definite matrix Σ ∈ Rd×d and a vector x ∈ Rd , let ‖x‖Σ = √ x > Σx . For two symmetric matrices A , B , we say A 4 B if B − A is positive semi-definite . Our analysis consists of an offline and online component . We often add ′ to indicate the online quantities , e.g. , we denote with s1 , s2 , . . . the offline contexts and with s′1 , s ′ 2 , . . . the online contexts .

Summary: This paper studies the setting where it is not possible (e.g. not allowed or difficult) to update a policy while it interacts with a system. The authors suggest to build, from a batch of historical contexts, an explorative policy, non-reactive, that is not updated online as it interacts with the system, to gather informative data that are finally used to build a near-optimal policy.  The explorative policy (Planner) consists in a mixture of policies built offline from the historical contexts so as to minimize the uncertainty. The main idea is that using this mixture of policies on the real system will also gather data so as to reduce the uncertainty.  Sample complexity bounds and bounds on the performance of the final policy are derived. The algorithm is also compared with other strategies in numerical experiments.