Bandit Phase Retrieval

p n ) , which matches known lower bounds up to logarithmic factors and improves on the best known upper bound by a factor of p d. We also show that the minimax simple regret is ⇥̃ ( d/ p n ) and that this is only achievable by an adaptive algorithm . Our analysis shows that an apparently convincing heuristic for guessing lower bounds can be misleading and that uniform bounds on the information ratio for information-directed sampling [ Russo and Van Roy , 2014 ] are not sufficient for optimal regret . 1 Introduction . We study an instantiation of the low-rank bandit problem [ Jun et al. , 2019 ] that in the statistical setting is called phase retrieval . Although this model is interesting in its own right , our main focus is on the curious information structure of this problem and how it impacts algorithm design choices . Notably , we were not able to prove optimal regret for standard approaches based on optimism , Thompson sampling or even information-directed sampling . Instead , our algorithm is a variant of explore-then-commit with an adaptive exploration phase that learns to gain information at a faster rate than what is achievable with non-adaptive exploration . Problem setting Let k · k be the standard euclidean norm and Bdr = { x 2 Rd : kxk r } and Sd 1r = { x 2 Rd : kxk = r } . At the start of the game the environment secretly chooses a vector ✓ ? 2 Sd 1r with r 2 [ 0 , 1 ] a constant that is known to the learner . The assumption that r is known can be relaxed at essentially no cost ( Section 8 ) . The game then proceeds over n rounds . In round t the learner chooses an action At 2 Bd1 and observes a reward Xt = hAt , ✓ ? i2 + ⌘t , where ( ⌘t ) nt=1 is a sequence of independent standard Gaussian random variables . As is standard in bandit problems , the conditional law of At should be chosen as a ( measurable ) function of the previous actions ( As ) t 1s=1 and rewards ( Xs ) t 1 s=1 and possibly an exogenous source of randomness . The performance of a policy ⇡ is measured in terms of the expected regret , Rn ( ⇡ , ✓ ? ) = max a2Bd1 E `` nX t=1 ha , ✓ ? i2 hAt , ✓ ? i2 # = nr2 E `` nX t=1 hAt , ✓ ? i2 # . The minimax regret is R ? n = supr2 [ 0,1 ] inf⇡ sup✓ ? 2Sd 1r Rn ( ⇡ , ✓ ? ) , where the infimum is over all policies . We also study the pure exploration setting , where at the end of the game the learner uses the observed data ( At ) nt=1 and ( Xt ) nt=1 to make a prediction bA ? 2 Bd1 of the optimal action . The simple regret of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . policy ⇡ is rn ( ⇡ , ✓ ? ) = max a2Bd1 E h ha , ✓ ? i2 h bA ? , ✓ ? i2 i = r2 E h h bA ? , ✓ ? i2 i . As expected , the minimax simple regret is r ? n = supr2 [ 0,1 ] inf⇡ sup✓ ? 2Sd 1r rn ( ⇡ , ✓ ? ) . Contributions Our main contribution is an upper bound on R ? n that matches existing lower bounds up to logarithmic factors . For the simple regret we provide a near-optimal upper bound and a lower bound showing that non-adaptive policies must be at least a factor of ⌦ ( p d ) suboptimal . In all of the following , const is a universal non-negative constant that may vary from one expression to the next . Theorem 1 . R ? n const d p n log ( n ) log ( d ) . A corollary of the proof of the lower bound by Kotłowski and Neu [ 2019 ] shows that the minimax regret is at least ⌦ ( d p n ) , so the minimax regret for bandit phase retrieval is now known up to logarithmic factors . For the simple regret we provide the following upper and lower bounds : Theorem 2. r ? n const d p log ( n ) log ( d ) /n . Theorem 3 . Assume that n d 8 . Then there exists an r 2 [ 0 , 1 ] such that for all policies ⇡ with ( At ) nt=1 independent of ( Xt ) n t=1 , sup✓ ? 2Sd 1r rn ( ⇡ , ✓ ? ) const p d3/n . We also show that worst-case bounds on the information ratio for information-directed sampling are not sufficient to achieve optimal regret . Our results suggest that the conjectured lower bounds for low-rank bandits [ Jun et al. , 2019 , Lu et al. , 2021 ] are not true and that existing upper bounds may be loose . The same phenomenon may explain the gap between upper and lower bounds for bandit principle component analysis [ Kotłowski and Neu , 2019 ] , as we discuss in Section 8 . Notation The first n integers are [ n ] = { 1 , 2 , . . . , n } and the standard basis vectors in Rd are e1 , . . . , ed . The span of a collection of vectors is denoted by span ( v1 , . . . , vm ) and the orthogonal complement of a linear subspace V ⇢ Rd is V ? = { x 2 Rd : hx , yi = 0 for all y 2 V } . The mutual information between random elements X and Y on the same probability space is I ( X ; Y ) and the relative entropy between probability measures P and Q on the same measurable space is KL ( P , Q ) . The dimension of a set ⇥ ⇢ Rd is defined as the dimension of the affine hull of ⇥ . 2 Related work . The most related work is by Huang et al . [ 2021 ] . As happens surprisingly often , this is a case where multiple groups tackled the same problem and developed similar methods to arrive at approximately the same solution . Their work provides the same upper bound in the cumulative regret setting as we do using an explore-then-commit principle and the noisy power method . They also generalise the setting to consider higher powers . For example , when the reward is hAt , ✓ ? i3 . What is different is that we use a more ad-hoc ( but still interesting ) construction during the exploration period and our work is more focussed on the information-theoretic implications . Phase retrieval Phase retrieval is a classical problem in signal processing and statistics [ Candès et al. , 2015b , a , Cai et al. , 2016 , Chen and Candès , 2017 , Chen et al. , 2019 , Sun et al. , 2018 ] . These works are focused on learning ✓ ? where the covariates ( At ) nt=1 are uncontrolled , either random or fixed design . Linear bandits Our problem can be written as a stochastic linear bandit by noticing that hAt , ✓ ? i2 = hAtA > t , ✓ ? ✓ > ? i , where the inner product between matrices on the right-hand side should be interpreted coordinate-wise and the action set is { aa > : a 2 Bd1 } . There is an enormous literature on stochastic linear bandits [ Auer , 2002 , Dani et al. , 2008 , Rusmevichientong and Tsitsiklis , 2010b , Chu et al. , 2011 , Abbasi-Yadkori et al. , 2011 ] . This reduction immediately yields an upper bound on the minimax regret of O ( d2 p n log ( n ) ) . Low-rank bandits Low-rank bandits are a kind of linear bandit where the environment is determined by an unknown matrix and the actions of the learner are also matrices . Let E ⇢ Rd1⇥d2 and A ⇢ Rd1⇥d2 . A low-rank bandit problem over E and with actions A is characterised by a matrix ⇥ ? 2 E . The learner plays actions At 2 A and the reward is Xt = hAt , ⇥ ? i+ ⌘t , where ⌘t is noise and the inner product between matrices is interpreted coordinate-wise . So far this is nothing more than a complicated way of defining a linear bandit . The name comes from the fact that in general elements of E are assumed to be low-rank . The precise nature of the problem is determined by assumptions on E and the action set A . Our setup is recovered by assuming that E = { xx > : x 2 Sd 1r } and A = { xx > : x 2 Bd1 } . Jun et al . [ 2019 ] assume that E consists of rank p matrices and A = { xy > : x 2 X , y 2 Y } for some reasonably bounded sets X ⇢ Rd1 and Y ⇢ Rd2 . They prove that the regret is bounded by Õ ( ( d1 + d2 ) 3/2 p pn ) . These results can not be applied directly to the phase retrieval bandit because of the product assumption on the action set . Lu et al . [ 2021 ] retain the assumption that E consists of rank-p matrices , but relax the product form of the action set ( while also allowing for generalised linear models ) . Relying only on mild boundedness assumptions , they show that the regret can be bounded by Õ ( ( d1 + d2 ) 3/2 p pn ) . For the bandit phase retrieval problem , d1 = d2 = d and p = 1 , so this algorithm yields an upper bound on the regret for bandit phase retrieval of Õ ( d3/2 p n ) . Both Jun et al . [ 2019 ] and Lu et al . [ 2021 ] conjecture that their upper bounds are optimal . Our results show that this is not true for this sub-problem , despite the fact that the heuristic argument used by these authors holds in this case , as we explain in Section 3 . We summarize these comparisons in Table 1 . Some authors use a model where the noise is in the parameter rather than additive , which means the reward is hAt , ⇥ti with ( ⇥t ) nt=1 an independent and identically distributed sequence of low-rank matrices ( with unknown distribution ) . For example , Katariya et al . [ 2017b , a ] and Trinh et al . [ 2020 ] assume that ⇥t is rank-1 almost surely and A = { eie > j : 1 i , j d } , which means the learner is trying to identify the largest entry in a matrix . Adversarial setting A similar problem has been studied in the adversarial framework by Kotłowski and Neu [ 2019 ] . They assume that ( ✓t ) nt=1 is a sequence of vectors chosen in secret by an adversary at the start of the game and the learner observes hAt , ✓ti2 . They design an algorithm for which the regret is at most Õ ( d p n ) , while the best lower bound is ⌦ ( p dn ) .

Summary: This paper studies the phase retrieval problem under bandit feedback. Regret bounds for this problem can be obtained from prior work by casting the problem as a low-rank bandit, which yields a $\tilde{O}(d^{3/2} \sqrt{n})$ regret upper bound. While several authors have conjectured that the upper bound for low-rank bandits is tight using information-theoretic heuristics, and these heuristics carry over also to the phase retrieval setting, this paper shows that these heuristics can be quite misleading. Indeed, this paper proves that the optimal regret for this setting is $\Theta(d \sqrt{n})$ (providing matching upper and lower bounds on the regret), which they obtain by ensuring that their algorithm can _increase_ the rate of information accumulated over time (thus circumventing the information-theoretic heuristics of prior works).