Optimal Gradient-based Algorithms for Non-concave Bandit Optimization

1 Introduction . Bandits [ 51 ] are a class of online decision-making problems where an agent interacts with the environment , only receives a scalar reward , and aims to maximize the reward . In many real-world applications , bandit and RL problems are characterized by large or continuous action space . To encode the reward information associated with the action , function approximation for the reward function is typically used , such as linear bandits [ 20 ] . Stochastic linear bandits assume the mean reward to be the inner product between the unknown model parameter and the feature vector associated with the action . This setting has been extensively studied , and algorithms with optimal regret are known [ 20 , 51 , 2 , 13 ] . However , linear bandits suffer from limited representation power unless the feature dimension is prohibitively large . A comprehensive empirical study [ 60 ] found that real-world problems required non-linear models and thus non-concave rewards to attain good performance on a testbed of bandit problems . To take a step beyond the linear setting , it becomes more challenging to design optimal algorithms . Unlike linear bandits , more sophisticated algorithms beyond optimism are necessary . For instance , a natural first step is to look at quadratic [ 44 ] and higher-order polynomial [ 35 ] reward . In ∗Alphabetical order . Correspondence to : Qi Lei , qilei @ princeton.edu , Jason D. Lee , jasonlee @ princeton.edu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the context of phase retrieval , which is a special case for the quadratic bandit , people have derived algorithms that achieve minimax risks in the statistical learning setting [ 9 , 52 , 15 ] . However , the straightforward adaptation of these algorithms results in sub-optimal dimension dependency . In the bandit domain , existing analysis on the nonlinear setting includes eluder dimension [ 62 ] , subspace elimination [ 55 , 43 ] , etc . Their results also suffer from a larger dimension dependency than the best known lower bound in many settings . ( See Table 1 and Section 3 for a detailed discussion of these results . ) Therefore in this paper , we are interested in investigating the following question : What is the optimal regret for non-concave bandit problems , including structured polynomials ( low-rank etc. ) ? Can we design algorithms with optimal dimension dependency ? Contributions : In this paper , we answer the questions and close the gap ( in problem dimension ) for various non-linear bandit problems . 1 . First , we design stochastic zeroth-order gradient-like2 ascent algorithms to attain minimax regret for a large class of structured polynomials . The class of structured polynomials contains bilinear and low-rank linear bandits and symmetric and asymmetric higher-order homogeneous polynomial bandits with action dimension d. Though the reward is non-concave , we combine techniques from two bodies of work , non-convex optimization and numerical linear algebra , to design robust gradient-based algorithms that converge to global maxima . Our algorithms are also computationally efficient , practical , and easily implementable . In all cases , our algorithms attain the optimal dependence on dimension d , which was not previously attainable using existing optimism techniques . As a byproduct , our algorithm refutes3 the conjecture from [ 43 ] on the bilinear bandit , and the conjecture from [ 55 ] on low-rank linear bandit by giving an algorithm that attains the optimal dimension dependence . 2 . We demonstrate that our techniques for non-concave bandits extend to RL in the generative setting , improving upon existing optimism techniques . 3 . When the reward is a general polynomial without noise , we prove that solving polynomial equations achieves regret equal to the intrinsic algebraic dimension of the underlying polynomial class , which is often linear in d for interesting cases . In general , this complexity can not be further improved . 4 . Furthermore , we provide a lower bound showing that all UCB algorithms have a sample complexity of Ω ( dp ) , where p is the degree of the polynomial . The dimension of all homogeneous polynomials of degree p in dimension d is dp , showing that UCB is oblivious to the polynomial class and highly sub-optimal even in the noiseless setting . 1.1 Related Work . Linear Bandits . Linear bandit problems and their variants are studied in [ 20 , 51 , 1 , 13 , 2 , 20 , 61 , 34 , 33 ] . The matching upper bound and minimax lower bound achievesO ( d √ T ) regret . Structured linear bandits , including sparse linear bandit [ 3 ] which developed an online-to-confidence-set technique . This technique yields the optimal O ( √ sdT ) rate for sparse linear bandit . However [ 55 ] employed the same technique for low-rank linear bandits giving an algorithm with regret O ( √ d3poly ( k ) T ) which we improve to O ( √ d2poly ( k ) T ) , which meets the lower bound given in [ 55 ] . Eluder Dimension . [ 62 ] proposed the eluder dimension as a general complexity measure for nonlinear bandits . However , the eluder dimension is only known to give non-trivial bounds for linear function classes and monotone functions of linear function classes . For structured polynomial classes , the eluder dimension simply embeds into an ambient linear space of dimension dp , where d is the dimension and p is the degree . This parallels the linearization/NTK line in supervised learning [ 71 , 30 , 7 ] which show that linearization also incurs a similarly large penalty of dp sample complexity , and more advanced algorithm design is need to circumvent linearization [ 10 , 17 , 25 , 72 , 27 , 59 , 29 , 57 , 36 , 68 , 19 ] . Non-concave Bandits . To our knowledge , there is no general study of non-concave bandits , likely due to the difficulty of globally maximizing non-concave functions . A natural starting point of 2Our algorithm estimates the gradient , but with some irreducible bias for the tensor case . Note that our algorithms converge linearly despite the bias . 3Both papers conjectured regret of the form O ( √ d3poly ( k ) T ) based on convincing but potentially misleading heuristics . studying the non-concave setting are quadratic rewards such as the Rayleigh quotient , or namely bandit PCA [ 47 , 28 ] . In the bilinear [ 43 ] and low-rank linear setting [ 55 ] with rank k parameter matrices achieved Õ ( √ d3poly ( k ) T ) regret . Other literature considered related but different settings that are not comparable to our results [ 44 , 42 , 32 , 48 ] . We note that the regret of all previous work is at least O ( √ d3T ) . This includes the subspace exploration and refinement algorithms from [ 43 , 55 ] , or from eluder dimension [ 62 ] . Recently [ 35 ] considers online problem with a low-rank tensor associated with axis-aligned set of arms , which corresponds to finding the largest entry of the tensor . Finally , [ 47 ] study the bandit PCA problem in the adversarial setting , attaining regret of O ( √ d3T ) . We leave adapting our results to the adversarial setting as an open problem . Due to space limitation we defer the discussion of some equally important literature to the appendix . 2 Preliminaries . 2.1 Setup : Structured Polynomial Bandit . We study structured polynomial bandit problems where the reward function is from a class of structured polynomials ( see below ) . A player plays actions for T rounds ; at each round t ∈ [ T ] , the player chooses one action at from the feasible action set A and receives the reward rt afterward . We consider both the stochastic case where rt = fθ ( at ) + ηt where ηt is the random noise , and the noiseless case rt = fθ ( at ) . Specifically the function fθ is unknown to the player , but lies in a known function class F . We use the notation vec ( M ) to denote the vectorization of a matrix or a tensor M , and v⊗p to denote the p-order tensor product of a vector v. For vectors we use ∥ · ∥2 or ∥ · ∥ to denote its ℓ2 norm . For matrices ∥ · ∥2 or ∥ · ∥ stands for its spectral norm , and ∥ · ∥F is Frobenius norm . For integer n , [ n ] denotes set { 1 , 2 , · · ·n } . For cleaner presentation , in the main paper we use Õ , Θ̃ or Ω̃ to hide universal constants , polynomial factors in p and polylog factors in dimension d , error ε , eigengap ∆ , total round number T or failure rate δ . We now present the outline with the settings considered in the paper : The stochastic bandit eigenvector case , FEV , considers action set A = { a ∈ Rd : ∥a∥2 ≤ 1 } as shown in Section 3.1 , and FEV = { fθ ( a ) = a TMa , M = ∑k j=1 λjvjv ⊤ j , for orthonormal vj M ∈ Rd×d , 1 ≥ λ1 ≥ |λ2| ≥ · · · ≥ |λk| } . The stochastic low-rank linear reward case , FLR considers action sets on bounded matrices A = { A ∈ Rd×d : ∥A∥F ≤ 1 } in Section 3.2 , and FLR = { fθ ( A ) = ⟨M , A⟩ = vec ( M ) ⊤vec ( A ) , M ∈ Rd×d , rank ( M ) = k , M = M⊤ , ∥M∥F ≤ 1 } . We illustrate how to apply the established bandit oracles to attain a better sample complexity for RL problems with the simulator in Section 3.2.1 . The stochastic homogeneous polynomial reward case is presented in Section 3.3 . For the symmetric case in Section 3.3.1 , the action sets are A = { a ∈ Rd : ∥a∥2 ≤ 1 } , and FSYM = { fθ ( a ) = ∑k j=1 λj ( v ⊤ j a ) p for orthonormal vj , 1 ≥ r∗ = λ1 > |λ2| ≥ · · · ≥ |λk| } ; in the asymmetric case in Section D.3.1 , the action sets are A = { a = a ( 1 ) ⊗ a ( 2 ) ⊗ · · · ⊗ a ( p ) ∈ Rdp : ∀q ∈ [ p ] , ∥a ( q ) ∥2 ≤ 1 } , and FASYM = { fθ ( a ) = ∑k j=1 λj ∏p q=1 ( vj ( q ) ⊤a ( q ) ) for orthonormal vj ( q ) for each q , 1 ≥ r∗ = |λ1| ≥ |λ2| ≥ · · · ≥ |λk| } . In the above settings , there is stochastic noise on the observed rewards . We also consider noiseless settings as below : The noiseless polynomial reward case is presented in Section 3.4 . The action sets A are subsets of Rd , and FP = { fθ ( a ) = ⟨θ , ã⊗p⟩ : θ ∈ V , ã = [ 1 , a⊤ ] ⊤ , V ⊆ ( Rd+1 ) ⊗p is an algebraic variety } . Additionally , F needs to be admissible ( Definition 3.20 ) . This class includes two-layer neural networks with polynomial activations ( i.e . structured polynomials ) . We study the fundamental limits of all UCB algorithms in Section G.0.1 as they are Ω ( dp−1 ) worse than our algorithm presented in Section 3.4.1 . In the above settings , we are concerned with the cumulative regret R ( T ) for T rounds . Let f∗θ = sup a∈A fθ ( a ) , Rθ ( T ) : = T∑ t=1 ( f∗θ − fθ ( at ) ) . Since the parameters can be chosen adversarially , we are bounding R ( T ) = supθ Rθ ( T ) in this paper . In all the stochastic settings above , we make the standard assumption on stochasticity that ηt is conditionally zero-mean 1-sub-Gaussian random variable regarding the randomness before t .

Summary: This paper studied a fundamental question in non-linear bandit: suppose the mean reward function f_\theta in the model r_t = f_\theta(a_t) + noise is non-linear or only linear after some high-dimensional reparametrization of \theta, is there a good bandit algorithm to find near-optimal policies with sample complexities dictated by the intrinsic dimension? In this paper, the authors investigated the following examples:   1. low-rank linear reward;  2. bilinear reward (or RL with quadratic Bellman-complete Q-function);  3. homogeneous polynomial reward;  4. noiseless polynomial reward.   In first three examples, the zeroth-order gradient ascent algorithm achieves a PAC sample complexity with the optimal dependence on the dimension d, strictly better than the upper bound given by LinUCB when viewed as a generalized linear model, as well as the eluder dimension argument. In the last example, there is an instance where solving polynomial equations after random sampling only has sample complexity O(d), but any UCB algorithm incurs a sample complexity O(d^p).