Doubly Robust Thompson Sampling with Linear Payoffs

√ T ) , where φ2 is the minimum eigenvalue of the covariance matrix of contexts . This is the first regret bound of LinTS using φ2 without the dimension of the context , d. Applying the relationship between φ2 and d , the regret bound of the proposed algorithm is Õ ( d √ T ) in many practical scenarios , improving the bound of LinTS by a factor of √ d. A benefit of the proposed method is that it utilizes all the context data , chosen or not chosen , thus allowing to circumvent the technical definition of unsaturated arms used in theoretical analysis of LinTS . Empirical studies show the advantage of the proposed algorithm over LinTS . 1 Introduction . Contextual bandit has been popular in sequential decision tasks such as news article recommendation systems . In bandit problems , the learner sequentially pulls one arm among multiple arms and receives random rewards on each round of time . While not knowing the compensation mechanisms of rewards , the learner should make his/her decision to maximize the cumulative sum of rewards . In the course of gaining information about the compensation mechanisms through feedback , the learner should carefully balance between exploitation , pulling the best arm based on information accumulated so far , and exploration , pulling the arm that will assist in future choices , although it does not seem to be the best option at the moment . Therefore in the bandit problem , estimation or learning is an important element besides decision making . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A challenging aspect of estimation in the bandit problem is that a stochastic reward is observed only for the chosen arm . Consequently , only the context and reward pair of the chosen arm is used for estimation , which causes dependency of the context data at the round on the past contexts and rewards . To handle this difficulty , we view bandit problems as missing data problems . The first step in handling missing data is to define full , observed , and missing data . In bandit settings , full data consist of rewards and contexts of all arms ; observed data consist of full contexts for all arms and the reward for the chosen arm ; missing data consist of the rewards for the arms that are not chosen . Typical estimation procedures require both rewards and contexts pairs to be observed , and the observed contexts from the unselected are discarded ( see Table 1 ) . The analysis based on the completely observed pairs only is called complete record analysis . Most stochastic bandit algorithms utilize estimates based on complete record analysis . Estimators from complete record analysis are known to be inefficient . In bandit setting , using the observed data whose probability of observation depends on previous rewards requires special theoretical treatment . There are two main approaches to missing data : imputation and inverse probability weighting ( IPW ) . Imputation is to fill in the predicted value of missing data from a specified model , and IPW is to use the observed records only but weight them by the inverse of the observation probability . The doubly robust ( DR ) method [ Robins et al. , 1994 , Bang and Robins , 2005 ] is a combination of imputation and IPW tools . We provide a review of missing data and DR methods in supplementary materials . The robustness against model misspecification in missing data settings is insignificant in the bandit setting since the probability of observation or allocation to an arm is known . The merit of the DR method in the bandit setting is its ability to employ all the contexts including unselected arms . We propose a novel multi-armed contextual bandit algorithm called Doubly Robust Thompson Sampling ( DRTS ) that applies the DR technique used in missing data literature to Thompson Sampling with linear contextual bandits ( LinTS ) . The main thrust of DRTS is to utilize contexts information for all arms , not just chosen arms . By using the unselected , yet observed contexts , along with a novel algorithmic device , the proposed algorithm renders a unique regret decomposition which leads to a novel regret bound without resorting to the technical definition of unsaturated arms used by Agrawal and Goyal [ 2014 ] . Since categorizing the arms into saturated vs. unsaturated plays a critical role in costing extra √ d , by circumventing it , we prove a Õ ( d √ T ) bound of the cumulative regret in many practical occasions compared to Õ ( d3/2 √ T ) shown in Agrawal and Goyal [ 2014 ] . The main contributions of this paper are as follows . • We propose a novel contextual bandit algorithm that improves the cumulative regret bound of LinTS by a factor of √ d ( Theorem 1 ) in many practical scenarios ( Section 4.1 ) . This improvement is attained mainly by defining a novel set called super-unsaturated arms , that is utilizable due to the proposed estimator and resampling technique adopted in the algorithm . • We provide a novel estimation error bound of the proposed estimator ( Theorem 3 ) which depends on the minimum eigenvalue of the covariance matrix of the contexts from all arms without d. • We develop a novel dimension-free concentration inequality for sub-Gaussian vector martingale ( Lemma 4 ) and use it in deriving our regret bound in place of the self-normalized theorem by Abbasi-Yadkori et al . [ 2011 ] . • We develop a novel concentration inequality for the bounded matrix martingale ( Lemma 6 ) which improves the existing result ( Proposition 5 ) by removing the dependency on d in the bound . Lemma 6 also allows eliminating the forced sampling phases required in some bandit algorithms relying on Proposition 5 [ Amani et al. , 2019 , Bastani and Bayati , 2020 ] . All missing proofs are in supplementary materials . 2 Related works . Thompson Sampling [ Thompson , 1933 ] has been extensively studied and shown solid performances in many applications ( e.g . Chapelle and Li [ 2011 ] ) . Agrawal and Goyal [ 2013 ] is the first to prove theoretical bounds for LinTS and an alternative proof is given by Abeille et al . [ 2017 ] . Both papers show Õ ( d3/2 √ T ) regret bound , which is known as the best regret bound for LinTS . Recently , Hamidi and Bayati [ 2020 ] points out that Õ ( d3/2 √ T ) could be the best possible one can get when the estimator used by LinTS is employed . In our work , we improve this regret bound by a factor of√ d in many practical scenarios through a novel definition of super-unsaturated arms , which becomes utilizable due to the proposed estimator and resampling device implemented in the algorithm . Our work assumes the independence of the contexts from all arms across time rounds . Some notable works have used the assumption that the contexts are independently identically distributed ( IID ) . Leveraging the IID assumption with a margin condition , Goldenshluger and Zeevi [ 2013 ] derives a two-armed linear contextual bandit algorithm with a regret upper bound of order O ( d3logT ) . Bastani and Bayati [ 2020 ] has extended this algorithm to any number of arms and improves the regret bound to O ( d2log 3 2 d · logT ) . The margin condition states that the gap between the expected rewards of the optimal arm and the next best arm is nonzero with some constant probability . This condition is crucial in achieving a O ( logT ) regret bound instead of Õ ( √ T ) . In this paper , we do not assume this margin condition , and focus on the dependence on the dimension of contexts d. From a missing data point of view , most stochastic contextual bandit algorithms use the estimator from complete record analysis except Dimakopoulou et al . [ 2019 ] and Kim and Paik [ 2019 ] . Dimakopoulou et al . [ 2019 ] employs an IPW estimator that is based on the selected contexts alone . Dimakopoulou et al . [ 2019 ] proves a Õ ( d √ −1T 1+ N ) regret bound for their algorithm which depends on the number of arms , N . Kim and Paik [ 2019 ] considers the high-dimensional settings with sparsity , utilizes a DR technique , and improves the regret bound in terms of the sparse dimension instead of the actual dimension of the context , d. Kim and Paik [ 2019 ] is different from ours in several aspects : the mode of exploration ( -greedy vs. Thompson Sampling ) , the mode of regularization ( Lasso vs. ridge regression ) ; and the form of the estimator . A sharp distinction between the two estimators lies in that Kim and Paik [ 2019 ] aggregates contexts and rewards over the arms although they employ all the contexts . If we apply this aggregating estimator and DR-Lasso bandit algorithm to the low-dimensional setting , we obtain a regret bound of order O ( Ndφ2 √ T ) when the contexts from the arms are independent . This bound is bigger than our bound by a factor of d and N . It is because the aggregated form of the estimator does not permit the novel regret decomposition derived in Section 4.2 . The proposed estimator coupled with a novel algorithmic device renders the additive regret decomposition which in turn improves the order of the regret bound . 3 Proposed estimator and algorithm . 3.1 Settings and assumptions . We denote a d-dimensional context for the ith arm at round t by Xi ( t ) ∈ Rd , and the corresponding random reward by Yi ( t ) for i = 1 , . . . , N . We assume E [ Yi ( t ) |Xi ( t ) ] = Xi ( t ) Tβ for some unknown parameter β ∈ Rd . At round t , the arm that the learner chooses is denoted by at ∈ { 1 , . . . , N } , and the optimal arm by a∗t : = arg maxi=1 , ... , N { Xi ( t ) Tβ } . Let regret ( t ) be the difference between the expected reward of the chosen arm and the optimal arm at round t , i.e. , regret ( t ) : = Xa∗t ( t ) Tβ − Xat ( t ) Tβ . The goal is to minimize the sum of regrets over T rounds , R ( T ) : = ∑T t=1 regret ( t ) . The total round T is finite but possibly unknown . We also make the following assumptions . Assumption 1 . Boundedness for scale-free regrets . For all i = 1 , . . . , N and t = 1 , . . . , T , we have ‖Xi ( t ) ‖2 ≤ 1 and ‖β‖2 ≤ 1 . Assumption 2 . Sub-Gaussian error . Let Ht : = ⋃t−1 τ=1 [ { Xi ( τ ) } Ni=1 ∪ { aτ } ∪ { Yaτ ( τ ) } ] ∪ { Xi ( t ) } Ni=1 be the set of observed data at round t. For each t and i , the error ηi ( t ) : = Yi ( t ) −Xi ( t ) Tβ is conditionally zero-mean σ-sub-Gaussian for a fixed constant σ ≥ 0 , i.e , E [ ηi ( t ) |Ht ] = 0 and E [ exp ( ληi ( t ) ) |Ht ] ≤ exp ( λ2σ2/2 ) , for all λ ∈ R. Furthermore , the distribution of ηi ( t ) does not depend on the choice at round t , i.e . at . Assumption 3 . Independently distributed contexts . The stacked contexts vectors { Xi ( 1 ) } Ni=1 , . . . , { Xi ( T ) } Ni=1 ∈ RdN are independently distributed . Assumption 4 . Positive minimum eigenvalue of the average of covariance matrices . For each t , there exists a constant φ2 > 0 such that λmin ( E [ 1 N ∑N i=1Xi ( t ) Xi ( t ) T ] ) ≥ φ2 . Assumptions 1 and 2 are standard in stochastic bandit literature Agrawal and Goyal [ 2013 ] . We point out that given round t , Assumption 3 allows that the contexts among different arms , X1 ( t ) , . . . , XN ( t ) are correlated to each other . Assumption 3 is weaker than the assumption of IID , and the IID condition is considered by Goldenshluger and Zeevi [ 2013 ] and Bastani and Bayati [ 2020 ] . As Bastani and Bayati [ 2020 ] points out , the IID assumption is reasonable in some practical settings , including clinical trials , where health outcomes of patients are independent of those of other patients . Both Goldenshluger and Zeevi [ 2013 ] and Bastani and Bayati [ 2020 ] address the problem where the contexts are equal across all arms , i.e . X ( t ) = X1 ( t ) = . . . = XN ( t ) , while our work admits different contexts over all arms . Assumption 4 guarantees that the average of covariance matrices of contexts over the arms is well-behaved so that the inverse of the sample covariance matrix is bounded by the spectral norm . This assumption helps controlling the estimation error of β in linear regression models . Similar assumptions are adopted in existing works in the bandit setting [ Goldenshluger and Zeevi , 2013 , Amani et al. , 2019 , Li et al. , 2017 , Bastani and Bayati , 2020 ] .

Summary: The paper proposes a new Thompson Sampling-based algorithm for the canonical N-armed T period contextual bandit problem that  achieves an expected cumulative regret of $\mathcal{O}\left( \phi^{-2}\sqrt{T} \right)$, where $\phi^2$ is the min eigenvalue of the context covariance matrix. It is easily observed that $\phi^{-2} \geqslant d$, where $d$ is the dimension of the context space. However, the authors point out that $\phi^{-2} = \mathcal{O}(d)$, in fact, holds in many practical scenarios. In such cases, therefore, the $\phi$-dependent scaling corresponds to an improvement upon the well-known $\tilde{\mathcal{O}}\left( d^{3/2}\sqrt{T}\right)$ standard by a factor of $\sqrt{d}$. The primary algorithmic innovation driving this result is the use of a "doubly robust" model for unobserved rewards, and a ridge estimator for the underlying $d$-dimensional (true) parameter vector, which facilitates a regret decomposition that is amenable to the aforementioned improvement in the upper bound. Numerical studies illustrate a significant performance improvement over standard well-known algorithms for the problem. The robust performance is plausibly due to a full utilization of the entire $N\times d \times T$ context stack, and use of the ridge estimator.