Optimal Gradient-based Algorithms for Non-concave Bandit Optimization

1 Introduction . Bandits [ 51 ] are a class of online decision-making problems where an agent interacts with the environment , only receives a scalar reward , and aims to maximize the reward . In many real-world applications , bandit and RL problems are characterized by large or continuous action space . To encode the reward information associated with the action , function approximation for the reward function is typically used , such as linear bandits [ 20 ] . Stochastic linear bandits assume the mean reward to be the inner product between the unknown model parameter and the feature vector associated with the action . This setting has been extensively studied , and algorithms with optimal regret are known [ 20 , 51 , 2 , 13 ] . However , linear bandits suffer from limited representation power unless the feature dimension is prohibitively large . A comprehensive empirical study [ 60 ] found that real-world problems required non-linear models and thus non-concave rewards to attain good performance on a testbed of bandit problems . To take a step beyond the linear setting , it becomes more challenging to design optimal algorithms . Unlike linear bandits , more sophisticated algorithms beyond optimism are necessary . For instance , a natural first step is to look at quadratic [ 44 ] and higher-order polynomial [ 35 ] reward . In ∗Alphabetical order . Correspondence to : Qi Lei , qilei @ princeton.edu , Jason D. Lee , jasonlee @ princeton.edu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the context of phase retrieval , which is a special case for the quadratic bandit , people have derived algorithms that achieve minimax risks in the statistical learning setting [ 9 , 52 , 15 ] . However , the straightforward adaptation of these algorithms results in sub-optimal dimension dependency . In the bandit domain , existing analysis on the nonlinear setting includes eluder dimension [ 62 ] , subspace elimination [ 55 , 43 ] , etc . Their results also suffer from a larger dimension dependency than the best known lower bound in many settings . ( See Table 1 and Section 3 for a detailed discussion of these results . ) Therefore in this paper , we are interested in investigating the following question : What is the optimal regret for non-concave bandit problems , including structured polynomials ( low-rank etc. ) ? Can we design algorithms with optimal dimension dependency ? Contributions : In this paper , we answer the questions and close the gap ( in problem dimension ) for various non-linear bandit problems . 1 . First , we design stochastic zeroth-order gradient-like2 ascent algorithms to attain minimax regret for a large class of structured polynomials . The class of structured polynomials contains bilinear and low-rank linear bandits and symmetric and asymmetric higher-order homogeneous polynomial bandits with action dimension d. Though the reward is non-concave , we combine techniques from two bodies of work , non-convex optimization and numerical linear algebra , to design robust gradient-based algorithms that converge to global maxima . Our algorithms are also computationally efficient , practical , and easily implementable . In all cases , our algorithms attain the optimal dependence on dimension d , which was not previously attainable using existing optimism techniques . As a byproduct , our algorithm refutes3 the conjecture from [ 43 ] on the bilinear bandit , and the conjecture from [ 55 ] on low-rank linear bandit by giving an algorithm that attains the optimal dimension dependence . 2 . We demonstrate that our techniques for non-concave bandits extend to RL in the generative setting , improving upon existing optimism techniques . 3 . When the reward is a general polynomial without noise , we prove that solving polynomial equations achieves regret equal to the intrinsic algebraic dimension of the underlying polynomial class , which is often linear in d for interesting cases . In general , this complexity can not be further improved . 4 . Furthermore , we provide a lower bound showing that all UCB algorithms have a sample complexity of Ω ( dp ) , where p is the degree of the polynomial . The dimension of all homogeneous polynomials of degree p in dimension d is dp , showing that UCB is oblivious to the polynomial class and highly sub-optimal even in the noiseless setting . 1.1 Related Work . Linear Bandits . Linear bandit problems and their variants are studied in [ 20 , 51 , 1 , 13 , 2 , 20 , 61 , 34 , 33 ] . The matching upper bound and minimax lower bound achievesO ( d √ T ) regret . Structured linear bandits , including sparse linear bandit [ 3 ] which developed an online-to-confidence-set technique . This technique yields the optimal O ( √ sdT ) rate for sparse linear bandit . However [ 55 ] employed the same technique for low-rank linear bandits giving an algorithm with regret O ( √ d3poly ( k ) T ) which we improve to O ( √ d2poly ( k ) T ) , which meets the lower bound given in [ 55 ] . Eluder Dimension . [ 62 ] proposed the eluder dimension as a general complexity measure for nonlinear bandits . However , the eluder dimension is only known to give non-trivial bounds for linear function classes and monotone functions of linear function classes . For structured polynomial classes , the eluder dimension simply embeds into an ambient linear space of dimension dp , where d is the dimension and p is the degree . This parallels the linearization/NTK line in supervised learning [ 71 , 30 , 7 ] which show that linearization also incurs a similarly large penalty of dp sample complexity , and more advanced algorithm design is need to circumvent linearization [ 10 , 17 , 25 , 72 , 27 , 59 , 29 , 57 , 36 , 68 , 19 ] . Non-concave Bandits . To our knowledge , there is no general study of non-concave bandits , likely due to the difficulty of globally maximizing non-concave functions . A natural starting point of 2Our algorithm estimates the gradient , but with some irreducible bias for the tensor case . Note that our algorithms converge linearly despite the bias . 3Both papers conjectured regret of the form O ( √ d3poly ( k ) T ) based on convincing but potentially misleading heuristics . studying the non-concave setting are quadratic rewards such as the Rayleigh quotient , or namely bandit PCA [ 47 , 28 ] . In the bilinear [ 43 ] and low-rank linear setting [ 55 ] with rank k parameter matrices achieved Õ ( √ d3poly ( k ) T ) regret . Other literature considered related but different settings that are not comparable to our results [ 44 , 42 , 32 , 48 ] . We note that the regret of all previous work is at least O ( √ d3T ) . This includes the subspace exploration and refinement algorithms from [ 43 , 55 ] , or from eluder dimension [ 62 ] . Recently [ 35 ] considers online problem with a low-rank tensor associated with axis-aligned set of arms , which corresponds to finding the largest entry of the tensor . Finally , [ 47 ] study the bandit PCA problem in the adversarial setting , attaining regret of O ( √ d3T ) . We leave adapting our results to the adversarial setting as an open problem . Due to space limitation we defer the discussion of some equally important literature to the appendix . 2 Preliminaries . 2.1 Setup : Structured Polynomial Bandit . We study structured polynomial bandit problems where the reward function is from a class of structured polynomials ( see below ) . A player plays actions for T rounds ; at each round t ∈ [ T ] , the player chooses one action at from the feasible action set A and receives the reward rt afterward . We consider both the stochastic case where rt = fθ ( at ) + ηt where ηt is the random noise , and the noiseless case rt = fθ ( at ) . Specifically the function fθ is unknown to the player , but lies in a known function class F . We use the notation vec ( M ) to denote the vectorization of a matrix or a tensor M , and v⊗p to denote the p-order tensor product of a vector v. For vectors we use ∥ · ∥2 or ∥ · ∥ to denote its ℓ2 norm . For matrices ∥ · ∥2 or ∥ · ∥ stands for its spectral norm , and ∥ · ∥F is Frobenius norm . For integer n , [ n ] denotes set { 1 , 2 , · · ·n } . For cleaner presentation , in the main paper we use Õ , Θ̃ or Ω̃ to hide universal constants , polynomial factors in p and polylog factors in dimension d , error ε , eigengap ∆ , total round number T or failure rate δ . We now present the outline with the settings considered in the paper : The stochastic bandit eigenvector case , FEV , considers action set A = { a ∈ Rd : ∥a∥2 ≤ 1 } as shown in Section 3.1 , and FEV = { fθ ( a ) = a TMa , M = ∑k j=1 λjvjv ⊤ j , for orthonormal vj M ∈ Rd×d , 1 ≥ λ1 ≥ |λ2| ≥ · · · ≥ |λk| } . The stochastic low-rank linear reward case , FLR considers action sets on bounded matrices A = { A ∈ Rd×d : ∥A∥F ≤ 1 } in Section 3.2 , and FLR = { fθ ( A ) = ⟨M , A⟩ = vec ( M ) ⊤vec ( A ) , M ∈ Rd×d , rank ( M ) = k , M = M⊤ , ∥M∥F ≤ 1 } . We illustrate how to apply the established bandit oracles to attain a better sample complexity for RL problems with the simulator in Section 3.2.1 . The stochastic homogeneous polynomial reward case is presented in Section 3.3 . For the symmetric case in Section 3.3.1 , the action sets are A = { a ∈ Rd : ∥a∥2 ≤ 1 } , and FSYM = { fθ ( a ) = ∑k j=1 λj ( v ⊤ j a ) p for orthonormal vj , 1 ≥ r∗ = λ1 > |λ2| ≥ · · · ≥ |λk| } ; in the asymmetric case in Section D.3.1 , the action sets are A = { a = a ( 1 ) ⊗ a ( 2 ) ⊗ · · · ⊗ a ( p ) ∈ Rdp : ∀q ∈ [ p ] , ∥a ( q ) ∥2 ≤ 1 } , and FASYM = { fθ ( a ) = ∑k j=1 λj ∏p q=1 ( vj ( q ) ⊤a ( q ) ) for orthonormal vj ( q ) for each q , 1 ≥ r∗ = |λ1| ≥ |λ2| ≥ · · · ≥ |λk| } . In the above settings , there is stochastic noise on the observed rewards . We also consider noiseless settings as below : The noiseless polynomial reward case is presented in Section 3.4 . The action sets A are subsets of Rd , and FP = { fθ ( a ) = ⟨θ , ã⊗p⟩ : θ ∈ V , ã = [ 1 , a⊤ ] ⊤ , V ⊆ ( Rd+1 ) ⊗p is an algebraic variety } . Additionally , F needs to be admissible ( Definition 3.20 ) . This class includes two-layer neural networks with polynomial activations ( i.e . structured polynomials ) . We study the fundamental limits of all UCB algorithms in Section G.0.1 as they are Ω ( dp−1 ) worse than our algorithm presented in Section 3.4.1 . In the above settings , we are concerned with the cumulative regret R ( T ) for T rounds . Let f∗θ = sup a∈A fθ ( a ) , Rθ ( T ) : = T∑ t=1 ( f∗θ − fθ ( at ) ) . Since the parameters can be chosen adversarially , we are bounding R ( T ) = supθ Rθ ( T ) in this paper . In all the stochastic settings above , we make the standard assumption on stochasticity that ηt is conditionally zero-mean 1-sub-Gaussian random variable regarding the randomness before t .

Summary: This paper studies the zeroth-order optimization methods for the nonlinear bandit problem. It studies PAC bounds for bandit problems where the reward function is structured polynomials, including a low-rank generalized linear bandit problem and a two-layer neural network with polynomial activations bandit problem. In the low-rank generalized linear bandit setting, the authors provide a minimax optimal algorithm improving the results in [28, 35]. They further apply their algorithms in the RL with the generative model setting, achieving improved sample complexity. In the neural net setting, they propose a bandit algorithm that has sample complexity equal to the intrinsic algebraic dimension.  They also show that the standard optimistic algorithms are sub-optimal for both settings.