Bandit Phase Retrieval

p n ) , which matches known lower bounds up to logarithmic factors and improves on the best known upper bound by a factor of p d. We also show that the minimax simple regret is ⇥̃ ( d/ p n ) and that this is only achievable by an adaptive algorithm . Our analysis shows that an apparently convincing heuristic for guessing lower bounds can be misleading and that uniform bounds on the information ratio for information-directed sampling [ Russo and Van Roy , 2014 ] are not sufficient for optimal regret . 1 Introduction . We study an instantiation of the low-rank bandit problem [ Jun et al. , 2019 ] that in the statistical setting is called phase retrieval . Although this model is interesting in its own right , our main focus is on the curious information structure of this problem and how it impacts algorithm design choices . Notably , we were not able to prove optimal regret for standard approaches based on optimism , Thompson sampling or even information-directed sampling . Instead , our algorithm is a variant of explore-then-commit with an adaptive exploration phase that learns to gain information at a faster rate than what is achievable with non-adaptive exploration . Problem setting Let k · k be the standard euclidean norm and Bdr = { x 2 Rd : kxk r } and Sd 1r = { x 2 Rd : kxk = r } . At the start of the game the environment secretly chooses a vector ✓ ? 2 Sd 1r with r 2 [ 0 , 1 ] a constant that is known to the learner . The assumption that r is known can be relaxed at essentially no cost ( Section 8 ) . The game then proceeds over n rounds . In round t the learner chooses an action At 2 Bd1 and observes a reward Xt = hAt , ✓ ? i2 + ⌘t , where ( ⌘t ) nt=1 is a sequence of independent standard Gaussian random variables . As is standard in bandit problems , the conditional law of At should be chosen as a ( measurable ) function of the previous actions ( As ) t 1s=1 and rewards ( Xs ) t 1 s=1 and possibly an exogenous source of randomness . The performance of a policy ⇡ is measured in terms of the expected regret , Rn ( ⇡ , ✓ ? ) = max a2Bd1 E `` nX t=1 ha , ✓ ? i2 hAt , ✓ ? i2 # = nr2 E `` nX t=1 hAt , ✓ ? i2 # . The minimax regret is R ? n = supr2 [ 0,1 ] inf⇡ sup✓ ? 2Sd 1r Rn ( ⇡ , ✓ ? ) , where the infimum is over all policies . We also study the pure exploration setting , where at the end of the game the learner uses the observed data ( At ) nt=1 and ( Xt ) nt=1 to make a prediction bA ? 2 Bd1 of the optimal action . The simple regret of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . policy ⇡ is rn ( ⇡ , ✓ ? ) = max a2Bd1 E h ha , ✓ ? i2 h bA ? , ✓ ? i2 i = r2 E h h bA ? , ✓ ? i2 i . As expected , the minimax simple regret is r ? n = supr2 [ 0,1 ] inf⇡ sup✓ ? 2Sd 1r rn ( ⇡ , ✓ ? ) . Contributions Our main contribution is an upper bound on R ? n that matches existing lower bounds up to logarithmic factors . For the simple regret we provide a near-optimal upper bound and a lower bound showing that non-adaptive policies must be at least a factor of ⌦ ( p d ) suboptimal . In all of the following , const is a universal non-negative constant that may vary from one expression to the next . Theorem 1 . R ? n const d p n log ( n ) log ( d ) . A corollary of the proof of the lower bound by Kotłowski and Neu [ 2019 ] shows that the minimax regret is at least ⌦ ( d p n ) , so the minimax regret for bandit phase retrieval is now known up to logarithmic factors . For the simple regret we provide the following upper and lower bounds : Theorem 2. r ? n const d p log ( n ) log ( d ) /n . Theorem 3 . Assume that n d 8 . Then there exists an r 2 [ 0 , 1 ] such that for all policies ⇡ with ( At ) nt=1 independent of ( Xt ) n t=1 , sup✓ ? 2Sd 1r rn ( ⇡ , ✓ ? ) const p d3/n . We also show that worst-case bounds on the information ratio for information-directed sampling are not sufficient to achieve optimal regret . Our results suggest that the conjectured lower bounds for low-rank bandits [ Jun et al. , 2019 , Lu et al. , 2021 ] are not true and that existing upper bounds may be loose . The same phenomenon may explain the gap between upper and lower bounds for bandit principle component analysis [ Kotłowski and Neu , 2019 ] , as we discuss in Section 8 . Notation The first n integers are [ n ] = { 1 , 2 , . . . , n } and the standard basis vectors in Rd are e1 , . . . , ed . The span of a collection of vectors is denoted by span ( v1 , . . . , vm ) and the orthogonal complement of a linear subspace V ⇢ Rd is V ? = { x 2 Rd : hx , yi = 0 for all y 2 V } . The mutual information between random elements X and Y on the same probability space is I ( X ; Y ) and the relative entropy between probability measures P and Q on the same measurable space is KL ( P , Q ) . The dimension of a set ⇥ ⇢ Rd is defined as the dimension of the affine hull of ⇥ . 2 Related work . The most related work is by Huang et al . [ 2021 ] . As happens surprisingly often , this is a case where multiple groups tackled the same problem and developed similar methods to arrive at approximately the same solution . Their work provides the same upper bound in the cumulative regret setting as we do using an explore-then-commit principle and the noisy power method . They also generalise the setting to consider higher powers . For example , when the reward is hAt , ✓ ? i3 . What is different is that we use a more ad-hoc ( but still interesting ) construction during the exploration period and our work is more focussed on the information-theoretic implications . Phase retrieval Phase retrieval is a classical problem in signal processing and statistics [ Candès et al. , 2015b , a , Cai et al. , 2016 , Chen and Candès , 2017 , Chen et al. , 2019 , Sun et al. , 2018 ] . These works are focused on learning ✓ ? where the covariates ( At ) nt=1 are uncontrolled , either random or fixed design . Linear bandits Our problem can be written as a stochastic linear bandit by noticing that hAt , ✓ ? i2 = hAtA > t , ✓ ? ✓ > ? i , where the inner product between matrices on the right-hand side should be interpreted coordinate-wise and the action set is { aa > : a 2 Bd1 } . There is an enormous literature on stochastic linear bandits [ Auer , 2002 , Dani et al. , 2008 , Rusmevichientong and Tsitsiklis , 2010b , Chu et al. , 2011 , Abbasi-Yadkori et al. , 2011 ] . This reduction immediately yields an upper bound on the minimax regret of O ( d2 p n log ( n ) ) . Low-rank bandits Low-rank bandits are a kind of linear bandit where the environment is determined by an unknown matrix and the actions of the learner are also matrices . Let E ⇢ Rd1⇥d2 and A ⇢ Rd1⇥d2 . A low-rank bandit problem over E and with actions A is characterised by a matrix ⇥ ? 2 E . The learner plays actions At 2 A and the reward is Xt = hAt , ⇥ ? i+ ⌘t , where ⌘t is noise and the inner product between matrices is interpreted coordinate-wise . So far this is nothing more than a complicated way of defining a linear bandit . The name comes from the fact that in general elements of E are assumed to be low-rank . The precise nature of the problem is determined by assumptions on E and the action set A . Our setup is recovered by assuming that E = { xx > : x 2 Sd 1r } and A = { xx > : x 2 Bd1 } . Jun et al . [ 2019 ] assume that E consists of rank p matrices and A = { xy > : x 2 X , y 2 Y } for some reasonably bounded sets X ⇢ Rd1 and Y ⇢ Rd2 . They prove that the regret is bounded by Õ ( ( d1 + d2 ) 3/2 p pn ) . These results can not be applied directly to the phase retrieval bandit because of the product assumption on the action set . Lu et al . [ 2021 ] retain the assumption that E consists of rank-p matrices , but relax the product form of the action set ( while also allowing for generalised linear models ) . Relying only on mild boundedness assumptions , they show that the regret can be bounded by Õ ( ( d1 + d2 ) 3/2 p pn ) . For the bandit phase retrieval problem , d1 = d2 = d and p = 1 , so this algorithm yields an upper bound on the regret for bandit phase retrieval of Õ ( d3/2 p n ) . Both Jun et al . [ 2019 ] and Lu et al . [ 2021 ] conjecture that their upper bounds are optimal . Our results show that this is not true for this sub-problem , despite the fact that the heuristic argument used by these authors holds in this case , as we explain in Section 3 . We summarize these comparisons in Table 1 . Some authors use a model where the noise is in the parameter rather than additive , which means the reward is hAt , ⇥ti with ( ⇥t ) nt=1 an independent and identically distributed sequence of low-rank matrices ( with unknown distribution ) . For example , Katariya et al . [ 2017b , a ] and Trinh et al . [ 2020 ] assume that ⇥t is rank-1 almost surely and A = { eie > j : 1 i , j d } , which means the learner is trying to identify the largest entry in a matrix . Adversarial setting A similar problem has been studied in the adversarial framework by Kotłowski and Neu [ 2019 ] . They assume that ( ✓t ) nt=1 is a sequence of vectors chosen in secret by an adversary at the start of the game and the learner observes hAt , ✓ti2 . They design an algorithm for which the regret is at most Õ ( d p n ) , while the best lower bound is ⌦ ( p dn ) .

Summary: The paper studies a stochastic bandit optimization problem in which the expected reward can be expressed as $\langle a , \theta_\star \rangle^2$, where $\theta_\star$ is an unknown parameter. The authors provides an algorithm that achieve a nearly tight (up to logarithmic factors) cumulative regret bounds. Nearly tight bounds for simple regret have been shown as well.