Bandit Phase Retrieval

p n ) , which matches known lower bounds up to logarithmic factors and improves on the best known upper bound by a factor of p d. We also show that the minimax simple regret is ⇥̃ ( d/ p n ) and that this is only achievable by an adaptive algorithm . Our analysis shows that an apparently convincing heuristic for guessing lower bounds can be misleading and that uniform bounds on the information ratio for information-directed sampling [ Russo and Van Roy , 2014 ] are not sufficient for optimal regret . 1 Introduction . We study an instantiation of the low-rank bandit problem [ Jun et al. , 2019 ] that in the statistical setting is called phase retrieval . Although this model is interesting in its own right , our main focus is on the curious information structure of this problem and how it impacts algorithm design choices . Notably , we were not able to prove optimal regret for standard approaches based on optimism , Thompson sampling or even information-directed sampling . Instead , our algorithm is a variant of explore-then-commit with an adaptive exploration phase that learns to gain information at a faster rate than what is achievable with non-adaptive exploration . Problem setting Let k · k be the standard euclidean norm and Bdr = { x 2 Rd : kxk r } and Sd 1r = { x 2 Rd : kxk = r } . At the start of the game the environment secretly chooses a vector ✓ ? 2 Sd 1r with r 2 [ 0 , 1 ] a constant that is known to the learner . The assumption that r is known can be relaxed at essentially no cost ( Section 8 ) . The game then proceeds over n rounds . In round t the learner chooses an action At 2 Bd1 and observes a reward Xt = hAt , ✓ ? i2 + ⌘t , where ( ⌘t ) nt=1 is a sequence of independent standard Gaussian random variables . As is standard in bandit problems , the conditional law of At should be chosen as a ( measurable ) function of the previous actions ( As ) t 1s=1 and rewards ( Xs ) t 1 s=1 and possibly an exogenous source of randomness . The performance of a policy ⇡ is measured in terms of the expected regret , Rn ( ⇡ , ✓ ? ) = max a2Bd1 E `` nX t=1 ha , ✓ ? i2 hAt , ✓ ? i2 # = nr2 E `` nX t=1 hAt , ✓ ? i2 # . The minimax regret is R ? n = supr2 [ 0,1 ] inf⇡ sup✓ ? 2Sd 1r Rn ( ⇡ , ✓ ? ) , where the infimum is over all policies . We also study the pure exploration setting , where at the end of the game the learner uses the observed data ( At ) nt=1 and ( Xt ) nt=1 to make a prediction bA ? 2 Bd1 of the optimal action . The simple regret of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . policy ⇡ is rn ( ⇡ , ✓ ? ) = max a2Bd1 E h ha , ✓ ? i2 h bA ? , ✓ ? i2 i = r2 E h h bA ? , ✓ ? i2 i . As expected , the minimax simple regret is r ? n = supr2 [ 0,1 ] inf⇡ sup✓ ? 2Sd 1r rn ( ⇡ , ✓ ? ) . Contributions Our main contribution is an upper bound on R ? n that matches existing lower bounds up to logarithmic factors . For the simple regret we provide a near-optimal upper bound and a lower bound showing that non-adaptive policies must be at least a factor of ⌦ ( p d ) suboptimal . In all of the following , const is a universal non-negative constant that may vary from one expression to the next . Theorem 1 . R ? n const d p n log ( n ) log ( d ) . A corollary of the proof of the lower bound by Kotłowski and Neu [ 2019 ] shows that the minimax regret is at least ⌦ ( d p n ) , so the minimax regret for bandit phase retrieval is now known up to logarithmic factors . For the simple regret we provide the following upper and lower bounds : Theorem 2. r ? n const d p log ( n ) log ( d ) /n . Theorem 3 . Assume that n d 8 . Then there exists an r 2 [ 0 , 1 ] such that for all policies ⇡ with ( At ) nt=1 independent of ( Xt ) n t=1 , sup✓ ? 2Sd 1r rn ( ⇡ , ✓ ? ) const p d3/n . We also show that worst-case bounds on the information ratio for information-directed sampling are not sufficient to achieve optimal regret . Our results suggest that the conjectured lower bounds for low-rank bandits [ Jun et al. , 2019 , Lu et al. , 2021 ] are not true and that existing upper bounds may be loose . The same phenomenon may explain the gap between upper and lower bounds for bandit principle component analysis [ Kotłowski and Neu , 2019 ] , as we discuss in Section 8 . Notation The first n integers are [ n ] = { 1 , 2 , . . . , n } and the standard basis vectors in Rd are e1 , . . . , ed . The span of a collection of vectors is denoted by span ( v1 , . . . , vm ) and the orthogonal complement of a linear subspace V ⇢ Rd is V ? = { x 2 Rd : hx , yi = 0 for all y 2 V } . The mutual information between random elements X and Y on the same probability space is I ( X ; Y ) and the relative entropy between probability measures P and Q on the same measurable space is KL ( P , Q ) . The dimension of a set ⇥ ⇢ Rd is defined as the dimension of the affine hull of ⇥ . 2 Related work . The most related work is by Huang et al . [ 2021 ] . As happens surprisingly often , this is a case where multiple groups tackled the same problem and developed similar methods to arrive at approximately the same solution . Their work provides the same upper bound in the cumulative regret setting as we do using an explore-then-commit principle and the noisy power method . They also generalise the setting to consider higher powers . For example , when the reward is hAt , ✓ ? i3 . What is different is that we use a more ad-hoc ( but still interesting ) construction during the exploration period and our work is more focussed on the information-theoretic implications . Phase retrieval Phase retrieval is a classical problem in signal processing and statistics [ Candès et al. , 2015b , a , Cai et al. , 2016 , Chen and Candès , 2017 , Chen et al. , 2019 , Sun et al. , 2018 ] . These works are focused on learning ✓ ? where the covariates ( At ) nt=1 are uncontrolled , either random or fixed design . Linear bandits Our problem can be written as a stochastic linear bandit by noticing that hAt , ✓ ? i2 = hAtA > t , ✓ ? ✓ > ? i , where the inner product between matrices on the right-hand side should be interpreted coordinate-wise and the action set is { aa > : a 2 Bd1 } . There is an enormous literature on stochastic linear bandits [ Auer , 2002 , Dani et al. , 2008 , Rusmevichientong and Tsitsiklis , 2010b , Chu et al. , 2011 , Abbasi-Yadkori et al. , 2011 ] . This reduction immediately yields an upper bound on the minimax regret of O ( d2 p n log ( n ) ) . Low-rank bandits Low-rank bandits are a kind of linear bandit where the environment is determined by an unknown matrix and the actions of the learner are also matrices . Let E ⇢ Rd1⇥d2 and A ⇢ Rd1⇥d2 . A low-rank bandit problem over E and with actions A is characterised by a matrix ⇥ ? 2 E . The learner plays actions At 2 A and the reward is Xt = hAt , ⇥ ? i+ ⌘t , where ⌘t is noise and the inner product between matrices is interpreted coordinate-wise . So far this is nothing more than a complicated way of defining a linear bandit . The name comes from the fact that in general elements of E are assumed to be low-rank . The precise nature of the problem is determined by assumptions on E and the action set A . Our setup is recovered by assuming that E = { xx > : x 2 Sd 1r } and A = { xx > : x 2 Bd1 } . Jun et al . [ 2019 ] assume that E consists of rank p matrices and A = { xy > : x 2 X , y 2 Y } for some reasonably bounded sets X ⇢ Rd1 and Y ⇢ Rd2 . They prove that the regret is bounded by Õ ( ( d1 + d2 ) 3/2 p pn ) . These results can not be applied directly to the phase retrieval bandit because of the product assumption on the action set . Lu et al . [ 2021 ] retain the assumption that E consists of rank-p matrices , but relax the product form of the action set ( while also allowing for generalised linear models ) . Relying only on mild boundedness assumptions , they show that the regret can be bounded by Õ ( ( d1 + d2 ) 3/2 p pn ) . For the bandit phase retrieval problem , d1 = d2 = d and p = 1 , so this algorithm yields an upper bound on the regret for bandit phase retrieval of Õ ( d3/2 p n ) . Both Jun et al . [ 2019 ] and Lu et al . [ 2021 ] conjecture that their upper bounds are optimal . Our results show that this is not true for this sub-problem , despite the fact that the heuristic argument used by these authors holds in this case , as we explain in Section 3 . We summarize these comparisons in Table 1 . Some authors use a model where the noise is in the parameter rather than additive , which means the reward is hAt , ⇥ti with ( ⇥t ) nt=1 an independent and identically distributed sequence of low-rank matrices ( with unknown distribution ) . For example , Katariya et al . [ 2017b , a ] and Trinh et al . [ 2020 ] assume that ⇥t is rank-1 almost surely and A = { eie > j : 1 i , j d } , which means the learner is trying to identify the largest entry in a matrix . Adversarial setting A similar problem has been studied in the adversarial framework by Kotłowski and Neu [ 2019 ] . They assume that ( ✓t ) nt=1 is a sequence of vectors chosen in secret by an adversary at the start of the game and the learner observes hAt , ✓ti2 . They design an algorithm for which the regret is at most Õ ( d p n ) , while the best lower bound is ⌦ ( p dn ) .

Summary:  The paper studies the quadratic bandit problem, where the reward at time $t$ is given by $\langle \theta_\star, a_t \rangle^2 + \varepsilon_t$, where $\varepsilon_t$ is i.i.d. gaussian noise, $\theta_\star$ is an unknown $d$-dimensional vector and $a$ is the action. The action set is fixed to be the unit ball, and the learner can choose any action $a_t \in \mathbb R^d$ such that $\lVert a_t \rVert_2 \le 1$. The norm of the unknown vector $\lVert \theta_\star \rVert$ is revealed to the learner at the beginning of the bandit. (This assumption can be removed, as discussed in the final section.) The main contributions are as follows.  1. The paper proves tight up to log factors upper and lower regret bounds $\tilde{\Theta}(d \sqrt n)$ for the quadratic bandit problem (Theorems 1, 2, respectively). 2. The paper extends the regret results to the pure exploration settings (Theorem 3). 3. The paper suggests that optimal regret is not attainable by fixed policies (Theorem 4).  The paper concludes that standard algorithms in bandits literature, including UCB, Thompson sampling, and information-directed sampling, might be insufficient for the quadratic bandit. Furthermore, the paper refutes previous regret lower bound conjecture, suggesting that naive guess based on signal-to-noise (SNR) ratio is wrong.