Title: Sample Complexity for Quadratic Bandits: Hessian Dependent Bounds and Optimal Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: optimization

quadratic bandits

sample complexity

optimality
Abstract: In stochastic zeroth-order optimization, a problem of practical relevance is understanding how to fully exploit the local geometry of the underlying objective function. We consider a fundamental setting in which the objective function is quadratic, and provide the first tight characterization of the optimal Hessian-dependent sample complexity. Our contribution is twofold. First, from an information-theoretic point of view, we prove tight lower bounds on Hessian-dependent complexities by introducing a concept called \emph{energy allocation}, which captures the interaction between the searching algorithm and the geometry of objective functions. A matching upper bound is obtained by solving the optimal energy spectrum. Then, algorithmically, we show the existence of a Hessian-independent algorithm that universally achieves the asymptotic optimal sample complexities for all Hessian instances. The optimal sample complexities achieved by our algorithm remain valid for heavy-tailed noise distributions, which are enabled by a truncation method.

Sample Complexity for Quadratic Bandits: Hessian Dependent Bounds and Optimal Algorithms

Abstract

In stochastic zeroth-order optimization, a problem of practical relevance is understanding how to fully exploit the local geometry of the underlying objective function. We consider a fundamental setting in which the objective function is quadratic, and provide the first tight characterization of the optimal Hessian-dependent sample complexity. Our contribution is twofold. First, from an information-theoretic point of view, we prove tight lower bounds on Hessian-dependent complexities by introducing a concept called energy allocation , which captures the interaction between the searching algorithm and the geometry of objective functions. A matching upper bound is obtained by solving the optimal energy spectrum. Then, algorithmically, we show the existence of a Hessian-independent algorithm that universally achieves the asymptotic optimal sample complexities for all Hessian instances. The optimal sample complexities achieved by our algorithm remain valid for heavy-tailed noise distributions, which are enabled by a truncation method.

1 Introduction

Stochastic optimization in the zeroth order (gradient-free) setting has attracted significant attention in recent decades. It naturally arises in various applications such as autonomous driving [12], AI gaming [22], robotics [14], healthcare [23], and education [13]. This setting is particularly important when the gradient of the objective function cannot be directly evaluated or is expensive.

An important case of interest in the zeroth order setting is the bandit optimization of smooth and strongly-convex functions. Although there are ample results regarding the minimax rates of this problem [1, 8, 21, 6, 3, 24], little is known about how its complexity depends on the geometry of the objective function f near the global optimum x ∗ , as specified by the quadratic approximation 1 2 ( x -x ∗ ) ⊤ ∇ 2 f ( x ∗ )( x -x ∗ ) . As an initial step, we investigate the following natural questions:

For zeorth-order bandit optimization problems of quadratic functions of the form 1 2 ( x -x 0 ) ⊤ A ( x -x 0 ) , what is the optimal instance-dependent upper bound with respect to A ?

Is there an algorithm that universally achieves the optimal instance-dependent bounds for all quadratic functions, but without the knowledge of Hessian?

As our main contributions, we fully addressed the above questions as follows. First, we established the tight Hessian-dependent upper and lower bounds of the simple regret. Our bounds indicate asymptotic sample complexity bounds of Tr 2 ( A -1 2 ) / (2 ϵ ) to achieve ϵ accuracy. This covers the

minimax lower bound of Ω( d 2 /ϵ ) in [21]. Second, we prove the existence of a Hessian-independent algorithm with a matching upper bound of O ( Tr 2 ( A -1 2 ) /ϵ ) . Thus, we complete the theory of zeroth-order bandit optimization on quadratic functions.

Related works beyond linear or convex bandit optimization. Compared to their linear/convex counterparts, much less is known for finding optimal bandit under nonlinear/non-convex reward function. A natural next step beyond linear bandits is to look at quadratic reward functions, as studied in this paper and some prior work (see reference on bandit PCA [15, 4], its rank-1 special cases [18, 11], bilinear [10], low-rank linear [19] and some other settings [9, 5, 16]).

When getting beyond quadratic losses, prior work on non-convex settings mainly focuses on finding achieving ϵ -stationary points instead of ϵ -optimal reward [17], except for certain specific settings (see, e.g., [7, 25, 26, 20, 26, 20]).

2 Problem Formulation and Main Results

We first present a rigorous formulation for the stochastic zeroth-order optimization problem studied in this paper. Given a fixed dimension parameter d , let f : B → R be an unknown objective function defined on the closed unit L 2 -ball B of the Euclidean space R d , which takes the following form for some positive-semidefinite (PSD) matrix A .

For each time t ∈ [ T ] , an optimization algorithm A produces a query point x t ∈ B , and receives

The algorithm can be adaptive , so that A is described by a sequence of conditional distributions for choosing each x t based on all prior observations { x τ , y τ } τ<t . Then we only assume that the noises { w t } T -1 t =1 are independent random variables with zero mean and unit variance, i.e., E [ w t | x t ] = 0 and E [ w 2 t | x t ] ≤ 1 .

For brevity, we use F ( A ) to denote all functions f satisfying equation (1) for some x 0 ∈ B . We are interested in the following minimax simple regret for any PSD matrix A .

The above quantity characterizes the simple regrets achievable by algorithms with perfect Hessian information. We also aim to identify the existence of algorithms that universally achieves the minimax regret for all A , without having access to that knowledge.

Our first result provides a tight characterization for the asymptotics of the minimax regret.

Theorem 2.1. For any PSD matrix A , we have

where A -1 2 denotes the pseudo inverse of A 1 2 . Moreover, if the distributions of w t are i.i.d. standard Gaussian, the above bound provides a tight characterization, i.e., there is a matching lower bound that the following equality is implied.

We prove Theorem 2.1 in Section 3. More generally, we also provide a full characterization of R ( T ; A ) for the non-asymptotic regime, stated as follows and proved in Appendix C.

Theorem 2.2. For any PSD matrix A and T > 3 dim A , where dim A denotes the rank of A , we have

where λ j is the j th smallest eigenvalue of A and k ∗ is the largest integer in { 0 , 1 , ..., dim A } satisfying T ≥ ( ∑ k ∗ k =1 λ -1 2 k )( ∑ k ∗ k =1 λ -3 2 k ) .

Remark 2.3. While our formulation requires the algorithm to return estimates from the bounded domain, i.e., x T ∈ B , our lower bound applies to algorithms without this constraint as well. This can be proved by showing that the worst-case simple regret over all f ∈ F ( A ) is always achievable by estimators satisfying x T ∈ B . Their construction can be obtained using the projection step in Algorithm 1, and proof details can be found in Appendix A.

Finally, we provide a positive answer to the existence of universally optimal algorithms, stated in the following Theorem. We present the algorithm and prove its achievability guarantee in Section 4.

Theorem 2.4. There exists an algorithm A , which does not depend on the Hessian parameter A , such that for A being any PSD matrix, the achieved minimax simple regret satisfies

3 Proof of Theorem 2.1

To motivate the main construction ideas, we start by proving a weaker version of the lower bound in Theorem 2.1. We use the provided intuition to construct a matching upper bound. Then we complete the proof by strengthening the lower bound through a Bayes analysis and the uncertainty principle.

3.1 Proof of a Weakened Lower Bound

Assuming A being positive definite, we prove the following inequality when the additive noises w t 's are i.i.d. standard Gaussian.

Throughout this section, we fix any basis such that A = diag ( λ 1 , ..., λ d ) . We construct a class of hard instances by letting

We investigate a Bayes setting where the objective function is defined by equation (1) and x 0 is uniformly random on the above set. For convenience, let x t = ( x 1 ,t , ..., x d,t ) be the action of the algorithm at time t . We define the energy spectrum to be a vector R = ( R 1 , ..., R d ) with each entry given by

Intuitively, our proof is to show that an allocation of at least R k ≥ Ω( λ -2 k x -2 k ) energy is required to correctly estimate each x k with Θ(1) probability. Note that for any entry that is incorrectly estimated, a penalty of Ω( λ k x 2 k ) is applied to the simple regret. This penalty is proportional to the required energy, which is due to the design of each x k . Meanwhile, the total expected energy is upper bounded by T , which is no greater than the summation of the individual requirements. Therefore, an Ω(1) fraction of the penalty is guaranteed, resulting in an overall effect of Ω (∑ k λ k x 2 k ) = Ω ( ( Tr ( A -1 2 )) 2 /T ) on the simple regret.

Rigorously, consider any fixed algorithm, we define the estimation cost of each entry as follows.

For brevity, let s k = sign ( x k ) . The above function is minimized by the minimum mean square error (MMSE) estimator (e.g., see [2, Sec. 4.6]), which depends on the following log-likelihood ratio (LLR) function.

Specifically, the MMSE estimator is given by ̂ x k,T ≜ | x k | tanh L k 2 , and the resulting error conditioned on any fixed { x τ , y τ } τ<T has an expectation of x 2 k · sech 2 L k 2 . Hence, we have the following lower bound for each cost entry.

By the Gaussian noise assumption, the conditional expectation of LLR can be written as follows.

The above quantity equals the KL divergence between the distributions of action-reward sequences generated by s k = ± 1 . Clearly, it has the following connection to the energy spectrum.

Recall inequality (6), it also implies the following lower bound of the mean-squared error.

Combine inequality (7) and (8), the overall simple regret can be bounded as follows.

Note that the definition of X H implies that the value of each x 2 k is fixed. Hence,

where the last inequality uses the fact that the total energy is upper bounded by the number of samples.

Finally, when T is sufficiently large, we have || x 0 || 2 ≤ 1 almost surely, which implies that our hard-instance functions belong to the set of objective functions F ( A ) . Therefore, E [ f ( x T )] provides a lower bound of the asymptotic minimax regret, and we can conclude that

Remark 3.1. The validity of the hard-instance functions requires T = Ω (( ∑ k λ -1 2 k )( ∑ k λ -3 2 k )) , which is consistent with the transition threshold in Theorem 2.2 to the non-asymptotic regimes.

3.2 Proof of the Upper Bound

Now we provide a proof for equation (4), which is implied by the following result.

Proposition 3.2. For any PSD matrix A , let k ∗ be the rank, λ 1 , ..., λ k ∗ be the non-zero eigenvalues, and e 1 , e 2 , ..., e k ∗ be the associated orthonormal eigenvectors. Then the expected simple regret achieved by algorithm 1 satisfies

Algorithm 1 Hessian Dependent Algorithm

Proof. Consider an eigenbasis of A with e 1 , ..., e k ∗ being the first k ∗ vectors. For each k ∈ [ k ∗ ] , the algorithm allocates R k energy to estimate the k th entry of x 0 . The value of R k is chosen such that the hard instances in the earlier subsection maximizes (modulo a constant factor) the lower bound in inequality (9) (i.e., R k satisfies x 2 k ≍ 1 / ( λ 2 k R k ) ) while ensuring the total number of samples does not exceed T -1 . By the zero-mean and unit-variance assumptions of the noise distribution, we have

This essentially provides an unbounded estimator ̂ x ≜ ∑ k ∗ k =1 α k e k that satisfies

Then, to obtain x T within the bounded constraint set, ̂ x is projected to B under a pseudometric defined by matrix A . Due to the convexity of set B , we have f ( ̂ x ) ≥ f ( x T ) with probability 1 (see Appendix A for a proof). Therefore, inequality (10) is implied by inequality (11).

3.3 Proof of the Lower Bound with Tight Constant Factors

To complete the proof of Theorem 2.1, it remains to show that for standard Gaussian noise, we have

Notice that the object function and reward feedbacks depend only on the projections of x 0 and query points onto the column (or row) space of A . Hence, it suffices to focus on algorithms with actions that are constrained on this subspace. This reduces the original problem to an equivalent instance that is defined on a (possibly) lower dimensional space and by a non-singular A . Therefore, we only need to prove inequality (12) for those reduced cases (i.e., when A is full-rank).

We lower bound the minimax regret by comparing them with a Bayes estimation error, where x 0 has a prior distribution that violates the bounded-norm constraint. Formally, for any fixed algorithm A , we can consider its expected simple regret E [ f ( x T )] over an extended class of objective functions where f is defined by equation (1) but x 0 is chosen from the entire Euclidian space. Then for any distribution of x 0 , the overall expectation is upper bounded by the following inequality.

where the first term on the RHS above is obtained by taking the supremum over all objective functions that satisfies x 0 ∈ B , then the second term is obtained from the adversarial choice over all estimators for x 0 / ∈ B . Compare the RHS of inequality (13) with equation (3), a lower bound of the minimax simple regret R ( T ; A ) can be obtained by taking the infimum over algorithm A on both sides. Hence, it remains to characterize the optimal Bayes estimation error on the LHS.

To provide a concrete analysis, let x 0 be a Gaussian vector with zero mean and a covariance of T -2 3 · I d , where I d denotes the identity matrix. 1 Under this setting, conditioned on any realization of queries x 1 , ..., x T -1 and feedbacks y 1 , ..., y T -1 , the posterior distribution of x 0 is proportional to

Clearly, the Bayesian error can be lower bounded by the expectation of the conditional covariance, which is further characterized by the following principle (see Appendix B for a proof).

Proposition 3.3 (Uncertainty Principle) . Let Z be a random variable on any measurable space and θ be a real-valued random variable dependent of Z . If the conditional distribution of θ given Z has a density function f Z ( θ ) , and ln f Z ( θ ) has a second derivative that is integrable over f Z , then

Hence, by taking the second derivative, the squared estimation error for the k th entry of x 0 is lower bounded by the inverse of the following expectation.

where ( · ) kk denotes the k th diagonal entry of the given matrix. Recall that we chose a basis where A is diagonal. The overall Bayes error is bounded as follows

where λ k = ( A ) kk is the k th eigenvalue of A . By Cauchy's inequality, the RHS above can be further bounded by

Note that by triangle inequality

where d is the dimension of the action space. We have obtained a lower bound of E x 0 [ E [ f ( x T )]] that is independent of the algorithm. Formally,

We apply the above estimate to inequality (13). By taking the infimum over all algorithms,

Observe that for our Gaussian prior,

Hence, all terms above can be replace by closed-form functions. and we have

4 Proof of Theorem 2.4

To prove the universal achievability result, we need to develop a new algorithm to learn and incorporate the needed Hessian information. Particularly, the procedure required for achieving the optimal rates is beyond simply adding an initial stage with an arbitrary Hessian estimator, for there are two challenges. First, any Hessian estimation algorithm would result in a mean squared error of Ω(1 /T ) in the worst case, which translates to a cost of Ω(1 /T ) in the minimax simple regret. This induced cost often introduces a multiplicative factor that is order-wise larger than the desired O ( ( Tr ( A -1 2 )) 2 ) . Second, to utilize any estimated Hessian, the analysis for the subsequent stages often requires the estimation error to have a light-tail distribution, which is not guaranteed for general linear estimators when the additive noise in the observation model has a heavy-tail distribution.

To overcome these challenges, we present two main algorithmic building blocks in the following subsections. The first uses o ( T ) samples to obtain a rough estimate of the Hessian, and achieves low-error guarantees with high probability through the introduction of a truncation method. The second estimates the global minimum with carefully designed sample points to minimize the error contributed from the Hessian estimation. We show that this sample phase can be written in the form of a two-step descent. Finally, we show the combination of two stages provides an algorithm that proves Theorem 2.4.

4.1 Initial Hessian Estimate and Truncation Method

Consider any sufficiently large T , we first use T 0 = ⌈ T 0 . 8 ⌉ samples to find a rough estimate of A . Particularly, we rely on the following result.

Proposition 4.1. For any fixed dimension d , there is an algorithm that samples on T 0 predetermined points and returns an estimation ̂ A , such that for any fixed α ∈ (0 , 0 . 5) , β ∈ (0 , + ∞ ) , and PSD matrix A ,

where || · || F denotes the Frobenius norm.

Proof. We first consider the 1D case. Let y + , y -, and y each be samples of f at 1 , -1 , and 0 , respectively. When the additive noises are subgaussian, we have that ( y + + y --2 y ) is an unbiased estimator of A with an error that is subgaussian. By repeating and averaging over this process ⌊ T 0 / 3 ⌋ times, the squared estimation error is reduced to O (1 /T 0 ) , and the normalized error is subgaussian, which satisfies the statement in the proposition.

However, recall that in Section 2 we did not assume the subgaussianity of w t . A modification of the above estimator is required to achieve the same guanrantee in equation (14). Here we propose a truncation method, which projects each measurement ( y + + y --2 y ) to a bounded interval [ -T 0 . 5 0 , T 0 . 5 0 ] . Specifically, the returned ̂ A is the average of ⌊ T 0 / 3 ⌋ samples of max { min { y + + y --2 y, T 0 . 5 0 } , -T 0 . 5 0 } , which provides a guaranteed superpolynomial tail bound. A more detailed discussion and analysis for the truncation method is provided in Appendix D.1.

For general d , one can return an estimator that satisfies the same light-tail requirement, for example, by applying the above 1D estimator repetitively poly ( d ) times to obtain estimates of all entries of A . Then the overall error probability is controlled by the union bound.

4.2 Two-Step Descent Stage

Let ̂ A be any estimator that satisfies the condition in Proposition 4.1 for α = 0 . 4 and β = 1 . 6 . Asymptotically, this implies the eigenvalues and eigenvectors of ̂ A are close to that of A . We choose an eigenbasis of ̂ A and remove vectors with vanishingly small eigenvalues to approximate the row space of A . Then, we estimate the entries of x 0 in these remaining directions following an energy allocation defined based on ̂ A .

Specifically, consider any fixed realization of ̂ A , let ̂ λ 1 , ̂ λ 2 , ... be its eigenvalues in the non-increasing order, ̂ e 1 , ̂ e 2 , ... be the corresponding eigenvectors, and k ∗ be the largest integer such that ̂ λ k ∗ ≥ T -0 . 2 . We present the detailed steps in Algorithm 2, where T 1 denotes the remaining available samples, i.e., T -T 0 .

Algorithm 2

procedure QUADRATIC SEARCH( λ , λ , ..., λ ∗ , e , ..., e ∗ , T )

We use the eigenbasis of ̂ A and let ̂ A 0 be the diagonal matrix given by diag ( ̂ λ 1 , ̂ λ 2 , ..., ̂ λ k ∗ , 0 , ..., 0) . In the first for-loop, we essentially estimated A x 0 , and then computed the first k ∗ entries of its product with the pseudo-inverse of ̂ A 0 (note that the rest of the entries are all zero). Since ̂ A 0 is a good estimate of A with high probability, we use it to compute the optimal energy spectrum similar to the instance-dependent case, and allocate the measurements accordingly.

Recall the proof in Section 3. By the analysis of the truncation method in Appendix D.1, the variable ̂ x could have served as an estimator of x 0 if ̂ A 0 = A , where the estimation process reduces to the Hessian-dependent case. However, ̂ A 0 relies only on O ( T 0 . 8 ) samples, which generally leads to an expected penalty of Θ( T -0 . 8 ) = ω (1 /T ) in simple regret if used in place of A . We reserve a fraction of samples for a second for-loop to fine-tune the estimation in order to achieve the optimal Hessian-dependent simple regrets.

4.3 Regret Analysis

Recall our assumption on the Hessain Estimator and T 0 = O ( T 0 . 8 ) . The probability for || ̂ A -A || F ≥ T -0 . 32 is o (1 /T ) . As our algorithm always returns x T with bounded norms, the simple regret contributed by this exceptional case is negligible. Hence, we can focus on instances where || ̂ A -A || F ≤ T -0 . 32 .

Under such scenario, any non-zero ̂ λ k is close to a non-zero eigenvalue of A . Specifically, we have | ̂ λ k -λ k | ≤ T -0 . 32 for all k , where each λ k is the k th largest eigenvalue of A . Recall the definition of ̂ A 0 . It implies that for sufficiently large T , k ∗ equals the rank of A , and all non-zero eigenvalues of ̂ A 0 are bounded away from zero. Consequently, || ̂ A -1 0 || F = || A -1 || F (1 + o (1)) , which does not scale w.r.t. T , where M -1 denotes the pseudo-inverse for any symmetric M . We also have ( Tr ( ̂ A -1 2 0 )) 2 = O ( ( Tr ( A -1 2 )) 2 ) .

The closeness between ̂ A and A also implies information on the eigenvectors of ̂ A 0 . Recall that ̂ A 0 is obtained by removing the diagonal entries of ̂ A (in their eigenbasis) that are below a threshold T -0 . 2 . For sufficiently large T , the removed entries are associated with the d -k ∗ smallest eigenvalues, which are no greater than T -0 . 32 . Hence, as a rough estimate, we have || ̂ A 0 -A || F = o ( T -0 . 3 ) .

These conditions can be used to characterize the distribution of ̂ x . As mentioned earlier, from the analysis of the truncation method, each α k concentrates around ̂ e k ̂ A -1 0 A x 0 . Hence, ˜ x concentrates near ̂ A -1 0 A x 0 , and the truncation method ensures that

Note that the L 2 -norm of ̂ A -1 0 A x 0 is no greater than 1 + o (1) under the condition of || ̂ A 0 -A || F = o ( T -0 . 3 ) . Formally, by triangle inequality

We can apply inequality (16) to show that the L 2 -norm of ˜ x is no greater than 1 + o (1) with 1 -o (1 /T ) probability. Under such high probability cases, the projection of ˜ x to the hyperball of radius 1 . 5 remains identical. Hence, by the PSD property of ̂ A 0 , which is due to the convergence of its eigenvalues, we can replace all ˜ x in inequality (15) with ̂ x and obtain that

The same analysis can also be performed for each β k , which concentrates near ̂ e k ̂ A -1 0 A (2 x 0 -̂ x ) . Formally, let ˜ x T ≜ ∑ k β k ̂ e k , we have

The above results for the two descent steps can be combined, using triangle inequality and Proposition 4.2 below, to obtain the following inequality (See Appendix D for their proofs).

where we denote z ≜ ( 2 ̂ A -1 0 A -( ̂ A -1 0 A ) 2 ) x 0 for brevity.

Proposition 4.2. Let ̂ A 0 , Z, y be variables dependent on a parameter T ∈ N , where ̂ A 0 , Z are PSD matrices and y belongs to the column space of ˜ A 0 . If lim sup T →∞ || ̂ A -1 0 || F < ∞ and lim T →∞ || Z -˜ A 0 || F = 0 , then

Assume the correctness of inequality (19), the remainder of the proof consists of two parts. First, we show that the vector z can be viewed as a Taylor approximation for a projection of x 0 onto the column space of ̂ A 0 , hence, ˜ x T could achieve the needed simple regret. Then, we show that the projection of ˜ x T to the unit hypersphere to obtain x T induces negligible cost, so that the validity constraint can be satisfied the same time.

For the first part, let P 1 denote the projective map onto the column space of ̂ A 0 , i.e., P 1 = ̂ A 0 ̂ A -1 0 (here and in the following, ̂ A -1 0 denotes the pseudo inverse of ̂ A 0 ). We consider the Hessian of f on this restricted subspace, which is given by A 1 ≜ P 1 AP 1 . Note that the definition of ̂ A 0 implies ̂ A 0 = P 1 ̂ AP 1 . We have || ̂ A 0 -A 1 || F = || P 1 ( ̂ A -A ) P 1 || F ≤ || ̂ A -A || F ≤ T -0 . 32 . Then recall that the non-zero eigenvalues of ̂ A 0 are bounded away from 0 , i.e., || ̂ A -1 0 || F = o ( T 0 . 32 ) . We have that A 1 is invertible within the column space of ̂ A 0 (i.e., A 1 A -1 1 = P 1 ) for sufficiently large T .

The pseudo inversion of A 1 provides a point of equivalence to x 0 within the column space of ̂ A 0 . Particularly, let z 0 ≜ A -1 1 A x 0 , we have P 1 z = z and f ( x ) = ( x -z 0 ) ⊺ A ( x -z 0 ) for any x ∈ R d and sufficiently large T . The second equality is due to A ( x 0 -z 0 ) = ( A -AA -1 1 A ) x 0 = 0 for sufficiently large T , which is implied by the following proposition (see Appendix D.4 for a proof).

Proposition 4.3. For any symmetric matrix A and any symmetric projection map P 1 (i.e., P 2 1 = P 1 = P ⊺ 1 ), if A 1 ≜ P 1 AP 1 and A has the same rank, then the pseudo inverse A -1 1 satisfies A = AA -1 1 A .

We show that the error z -z 0 is bounded by o ( T -0 . 5 ) . Observe that P 1 ̂ A -1 0 = ̂ A -1 0 P 1 = ̂ A -1 0 and A -1 1 P 1 = A -1 1 , we have

From || ̂ A 0 -A 1 || F ≤ T -0 . 32 , || ̂ A -1 0 || F = || ̂ A -1 || F · O (1) and the fact that A 1 is restricted to the column space of ̂ A 0 , we can derive that || A -1 1 || F = || ̂ A -1 || F · O (1) , which also does not scale w.r.t. T . Therefore, the above equality implies that || z -z 0 || 2 = o ( T -0 . 5 ) . As a consequence, we have following inequalities due to inequality (19) and triangle inequality.

For the second part, we first note that the L2 norm of z 0 is bounded by the spectrum norm of A -1 1 A . Specifically,

where || · || denotes the spectrum norm. Recall the definition of A 1 , we have A -1 1 AA 1 = A -1 1 P 1 AP 1 A 1 = A -1 1 A 1 A 1 . Hence,

Note that || A -A 1 || F ≤ || ̂ A 0 -A 1 || F + || ̂ A 0 -A 1 || F ≤ o ( T -0 . 3 ) . By the fact that || A -1 1 || F = || ̂ A -1 || F · O (1) , we have || A -1 1 A || 2 ≤ 1 + o ( T -0 . 6 ) , and || z 0 || 2 ≤ 1 + o ( T -0 . 6 ) .

Therefore, we have min x ∈B ( z 0 -x ) ⊺ ̂ A 0 ( z 0 -x ) = o ( T -1 . 2 ) . The projection of ˜ x T to the unit hypersphere guarantees the following bound (see Appendix A for a proof).

Note that both x T and z 0 belong to the column space of ̂ A 0 . From Proposition 4.2, we can substitute ̂ A 0 by A for the above inequality and obtain

Then the theorem is proved from the fact that f ( x T ) = ( x T -z 0 ) ⊺ A ( x T -z 0 ) .

References

A. Agarwal, O. Dekel, and L. Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In Colt , pages 28-40. Citeseer, 2010.

D. P. Bertsekas and J. N. Tsitsiklis. Introduction to probability vol. 1. 2002.

S. Bubeck, Y. T. Lee, and R. Eldan. Kernel-based methods for bandit convex optimization. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing , pages 72-85, 2017.

D. Garber, E. Hazan, and T. Ma. Online learning of eigenvectors. In International Conference on Machine Learning , pages 560-568. PMLR, 2015.

A. Gopalan, O.-A. Maillard, and M. Zaki. Low-rank bandits with latent mixtures. arXiv preprint arXiv:1609.01508 , 2016.

E. Hazan and K. Y. Levy. Bandit convex optimization: Towards tight bounds. In MIPS , 2014.

B. Huang, K. Huang, S. M. Kakade, J. D. Lee, Q. Lei, R. Wang, and J. Yang. Optimal gradientbased algorithms for non-concave bandit optimization. arXiv preprint arXiv:2107.04518 , 2021.

K. G. Jamieson, R. Nowak, and B. Recht. Query complexity of derivative-free optimization. Advances in Neural Information Processing Systems (NeurIPS) , 2012.

N. Johnson, V. Sivakumar, and A. Banerjee. Structured stochastic linear bandits. arXiv preprint arXiv:1606.05693 , 2016.

K.-S. Jun, R. Willett, S. Wright, and R. Nowak. Bilinear bandits with low-rank structure. In International Conference on Machine Learning , pages 3163-3172. PMLR, 2019.

S. Katariya, B. Kveton, C. Szepesvari, C. Vernade, and Z. Wen. Stochastic rank-1 bandits. In Artificial Intelligence and Statistics , pages 392-401. PMLR, 2017.

B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. A. Sallab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2022.

R. F. Kizilcec, J. Reich, M. Yeomans, C. Dann, E. Brunskill, G. Lopez, S. Turkay, J. J. Williams, and D. Tingley. Scaling up behavioral science interventions in online education. Proceedings of the National Academy of Sciences , 117(26):14900-14905, 2020.

J. Kober, A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. Springer Tracts in Advanced Robotics , 97:9-67, 2014.

W. Kotłowski and G. Neu. Bandit principal component analysis. In Conference On Learning Theory , pages 1994-2024. PMLR, 2019.

S. Lale, K. Azizzadenesheli, A. Anandkumar, and B. Hassibi. Stochastic linear bandits with hidden low rank structure. arXiv preprint arXiv:1901.09490 , 2019.

J. Larson, M. Menickelly, and S. M. Wild. Derivative-free optimization methods. Acta Numerica , 28:287-404, 2019.

T. Lattimore and B. Hao. Bandit phase retrieval. arXiv preprint arXiv:2106.01660 , 2021.

Y. Lu, A. Meisami, and A. Tewari. Low-rank generalized linear bandit problems. In International Conference on Artificial Intelligence and Statistics , pages 460-468. PMLR, 2021.

E. Moulines and F. Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in neural information processing systems , 24, 2011.

O. Shamir. On the complexity of bandit and derivative-free stochastic convex optimization. In Conference on Learning Theory , pages 3-24. PMLR, 2013.

S. Yildirim and S. B. Stene. A survey on the need and use of AI in game agents . InTech, 2010.

C. Yu, J. Liu, and S. Nemati. Reinforcement learning in healthcare: A survey. In arXiv preprint arXiv:1908.08796 , 2020.

X. Yu, I. King, M. R. Lyu, and T. Yang. A generic approach for accelerating stochastic zeroth-order convex optimization. IJCAI , 2018.

L. Zhang, T. Yang, R. Jin, and Z.-H. Zhou. Online bandit learning for a special class of non-convex losses. In Twenty-Ninth AAAI Conference on Artificial Intelligence , 2015.

P. Zhao and L. Lai. Optimal stochastic nonconvex optimization with bandit feedback. arXiv preprint arXiv:2103.16082 , 2021.

A Projection Lemma

Proposition A.1. For any PSD matrix A with dimension d , any closed convex set B in the Euclidian space R d , and ̂ x ∈ R d , let where

then

More generally,

Proof. We first provide a proof for x 0 ∈ B . For any α ∈ [0 , 1] , let

By convexity, we have x α ∈ B for any α . Note that g ( ̂ x , x α ) is differentiable. By the definition of x ∗ , we have

Therefore,

where the last inequality uses the PSD property of A .

Now we consider the more general case and let x be any vector in B . Following the same steps in the earlier case, we have

Hence,

Note that the above inequality holds for any x ∈ B . The proposition is proved by taking the minimum over x .

B Proof of Proposition 3.3

Proof. Wefirst prove for the case where Z is deterministic. Let µ Z denote the conditional expectation of θ . By Cauchy's inequality,

The quantity on the RHS above can be bounded as follows.

where the last inequality uses the integrability of f Z , which implies

Then we evaluate the second factor on the LHS of inequality (22). Recall that ∂ 2 ∂θ 2 ln f Z ( θ ) is integrable, the following limit exists.

Then by positivity, we also have

If we focus the non-trivial case where the first limit is not -∞ , the above two equation implies the existence of the following limit.

The result of the above equation has to be zero, because the limit points of ∂ ∂θ f Z ( θ ) must contain zero on both ends of the real line, which is implied by the integrability of f Z . Consequently, we have

Then, the special case of Proposition 3.3 with fixed Z is implied by inequality (22).

When Z is variable, we simply have

Then the proposition is implied by equation (23).

C Proof of Theorem 2.2

We first investigate the lower bounds. Observe that the proof provided in Section 3.1 only fails when the constructed hard instances have || x 0 || 2 > 1 . Hence, we have already covered the T ≥ ( ∑ d k =1 λ -1 2 k )( ∑ d k =1 λ -3 2 k ) case, i.e., when k ∗ = dim A = d . It remains to consider the other scenarios, where k ∗ < d is satisfied.

By the assumption that T ≥ ( ∑ k ∗ k =1 λ -1 2 k )( ∑ k ∗ k =1 λ -3 2 k ) , one can instead set the entries of x 0 in the earlier proof with indices greater than k ∗ to be zero, so that || x 0 || 2 ≤ 1 is satisfied. Formally, let the hard-instance functions be constructed by the following set.

Then by the identical proof steps, we have R ( T ; A ) = Ω ( ( ∑ k ∗ k =1 λ -1 2 k ) 2 / T ) .

̸

Next, we show that R ( T ; A ) = Ω ( λ k ∗ +1 ) . We assume the non-trivial case where λ k ∗ +1 = 0 . Note that R ( T ; A ) is non-increasing w.r.t. T . We can lower bound R ( T ; A ) through the above steps but by replacing T with any larger quantity. Specifically, recall that k ∗ is largest integer satisfying T ≥ ( ∑ k ∗ k =1 λ -1 2 k )( ∑ k ∗ k =1 λ -3 2 k ) , which implies T ≤ ( ∑ k ∗ +1 k =1 λ -1 2 k )( ∑ k ∗ +1 k =1 λ -3 2 k ) . We have,

Notice that this change of sampling time allows us to apply the earlier lower bound with k ∗ incremented by 1 .

To conclude,

which completes the proof of the lower bounds.

The needed upper bounds can be obtained by only estimating the first k ∗ entries of x 0 .

Remark C.1. The requirement of T > 3 dim A in the Theorem statement is simply due to the integer constraints for the achievability bounds. Indeed, when λ dim A is large, it requires at least Ω( dim A ) samples to achieve O (1) expected simple regret.

D Proof Details for Theorem 2.4

D.1 Truncation Method and Its Applications

The truncation method is based on the following facts.

Proposition D.1. For any sequence of independent random variables X 1 , X 2 , ..., X n and any fixed parameter m satisfying m> max k | E [ X k ] | . Let Z k = max { min { X k , m } , -m } for any k ∈ [ n ] , we have

Moreover, for any z > 0 , we have

Proof. The first inequality is proved by expressing the LHS with piecewise linear functions. Note that by the definition of Z k , we have

We apply the following inequalities, which holds for any m ≥ | E [ X k ] | .

Therefore,

The second inequality is due to the following elementary facts,

where the inequality step is implied by the definition of Z k and the condition m> max k | E [ X k ] | .

To prove the third inequality, we first investigate the following upper bound, which is due to Markov's inequality.

The equality step above is by the fact that Z k 's are jointly independent. For each k , using the fact that Z k is bounded, particularly, Z k -E [ X k ] ≤ m + | E [ X k ] | , we have the following inequality

For brevity, let θ ≜ | E [ X k ] | m . We combine the above bound with inequality (24) and (25) to obtain that

Recall that θ < 1 as assumed in the proposion. From elementary calculus, we have

Therefore, recall inequality (27), we have

By symmetry, one can also prove the following bound through the same steps.

Hence, the needed inequality is obtained by adding the two inequalities above.

Now equation (14) for the 1D case is immediately implied by Proposition D.1. Recall the construction of ̂ A in the proof, for any sufficiently large T 0 , we have

Remark D.2. Instead of projecting to a bounded interval, the same achievability result can be obtained if the we average over any functions that map the samples to [ -T 0 . 5 0 , T 0 . 5 0 ] while imposing an additional error of o ( T -α ) everywhere. This includes Θ(ln T ) -bit uniform quantizers, which naturally appear in digital systems, over which exact computation can be performed to eliminate numerical errors. We present this simple generalization in the following corollary.

Corollary D.3. Consider the setting in Proposition D.1. Let Y 1 , ..., Y n be variables that satisfy | Y k -Z k | ≤ b for all k with probability 1 . We have

D.2 Proof of Proposition 4.2

Proof.

D.3 Proof of inequality (19)

We apply Proposition 4.2 to inequality (17) and let Z = A ̂ A -1 0 A . Note that

which satisfies the condition of Proposition 4.2. Using the fact that ̂ A -1 0 ̂ A 0 ̂ A -1 0 = ̂ A -1 0 , we have

Then by the triangle inequality for the PSD matrix ̂ A 0 , the combination of the above inequality and inequality (18) gives

D.4 Proof of Proposition 4.3

Proof. When A 1 and A has the same rank, the map P 1 is invertible over the column space of A . Under such condition, there exists a matrix X such that A = XP 1 A . Note that A 1 A -1 1 = P 1 . We have XP 1 = XP 1 A 1 A -1 1 = AA -1 1 . Therefore, the needed A = AA -1 1 A is obtained by multiplying A on the right-hand sides in the above identity.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of Quadratic Hessian bandit and establishes both lower and upper bounds on algorithm performance to provide a characterization of simple regret in this setting. The authors also propose an algorithm that is agnostic to the knowledge of the Hessian.\n\nSTRENGTHS:\nThe paper considers a new problem on quadratic bandits and provides a complete characterization of simple regret in the setting. The completeness and tightness of the results paints a clear picture of the results about this new problem.\n\nWEAKNESSES:\nOne thing that the paper needs to improve upon is the writing. The exposition lacks a flow and proofs of the theorems seem to be all over the place.  Some points that I realized can be improved:\n1. The proof of Theorem 2.1 can be improved. It took me quite a bit of effort to understand the steps mainly because some intermediate steps were missing. The missing steps were non-trivial in the sense if they are missing, it is not easy for the reader to make the connection unless they have several results handy. I would suggest to please ensure that the proof is largely self-contained and it can be understood by the reader without actually having them to do intermediate steps. If it cannot be accommodated in the main text, you can move some intermediate result to Appendix or atleast provide a citation of some the results being used in the intermediate steps e.g. MMSE estimator. Also I think you are missing an \"$L_k$\" in equation 7.\n2. The statement of Theorem 2.1 is not clear. From the text leading upto it and the proof, it seems that it is a statement about the lower bound. But there is no lower bound anywhere in the statement of the theorem (apart from the one that is implicit in the limit for Gaussian noise). I would suggest to please make the statement of theorem clear and precise.\n3. Similarly, the way Theorem 2.4 is phrased, it is (almost) immediate from the current statement of Theorem 2.1. I think in addition to universal optimality you also need to mention that this algorithm does not need the prior information about the Hessian. This not only makes the theorem clearer but also stronger.\n\nIn addition to the above comments, I encourage the authors to please check the exposition for consistency and flow.\n\nQUESTIONS:\nThe choice of $x_k$ and the lower bound on $T$ in Theorem 2.2. (line 67) are rather interesting. Do you have an intuitive/physical explanation of why that particular expression comes up or what that represents? I haven't come across that combination of $\\lambda_k^{-3/2}$ and $\\lambda^{-1/2}$ anywhere else and it might have something fundamentally insightful.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
This papers studies quadratic stochastic zeroth-order optimization and provide lower and upper bounds on the minimax simple regret. First, the authors provide an asymptotically tight upper bound of $\frac{1}{2}({\rm Tr}(A^{-1/2}))^2$ where $A$ is the Hessian of the quadratic. Then, the authors provide asymptotic lower bounds which take two distinct expressions depending on properties of $A$. Next, the authors show the existence of a universally optimal algorithm, meaning that it asymptotically achieves the Hessian-dependent bounds without accessing the Hessian information.

STRENGTHS:
Clear problem statement and result presentation.

WEAKNESSES:
See **Questions**.

QUESTIONS:
- "Energy allocation" seems to be the same as "energy spectrum" defined in Section 3.1. Please formally define energy allocation or unify the terminology.
- Line 207: serve --> served.
- Line 195: remove "an".
- Please formally define Truncated Diff (projecting the diff onto [-t, t]) and other simple operations in the algorithms (Simple Project, etc.).
- Regarding Section 4.1, are there potential ways to do better than applying the 1D estimator poly(d) times to obtain an estimate for $A$? Are there more efficient join estimators that make use of $A$ being p.s.d.? If so, it would be helpful to point out (without incorporating them in this paper's analysis as it does not seem necessary). It might still be useful in future finite-sample analysis.

LIMITATIONS:
The authors defined the scope of this paper to be within specific problem setups. This work is largely theoretical and does not have potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
The paper considers the black-box optimization problem of quadratic functions specified with PSD matrices. The paper shows matching upper and lower sample complexity bounds depending on the matrices. For the upper bound, the paper proposes algorithms.

STRENGTHS:
The paper gives tight upper and lower sample complexity bounds for the black-box optimization of quadratic functions. The strength of the result is that the bounds depend on the PSD matrix, which means the bounds are instance-dependent. These results recovers previous results as well.

WEAKNESSES:
I feel that the focus of the paper is a bit narrow. So far, I do not know any motivating situation where we need to solve some black-box optimization of quadratic functions. Although the results are technically solid, their motivation and practical usefulness are not strong enough. 
The paper would be better evaluated for more theory/optimization-oriented conferences/journals. 

Or, if the proposed algorithms were more practical, say, it works for general convex functions but works better when the function is quadratic (best-of-both-world type algorithms), I would evaluate the paper higher.

QUESTIONS:
Do the proposed algorithm work for general convex functions?


I read other reviews and rebuttal comments. The rebuttal answers my question sufficiently. But I would keep my score the same since my concern about the motivation is not resolved.

LIMITATIONS:
As raised above, the technical results are solid, but practical usefulness is limited.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies zero-th order quadratic optimization with bandit feedback. It provides a comprehensive analysis of the optimal sample complexity, which depends on the Hessian of the objective function. There are two contributions. Firstly, it establishes lower bounds on Hessian-dependent complexities using the concept of energy allocation, capturing the interaction between the search algorithm and the geometry of the problem. The matching upper bound is achieved through optimizing the energy spectrum. Secondly, an algorithm is presented that achieves asymptotically optimal sample complexities for all Hessian instances, independent of the Hessian. These optimal sample complexities remain valid even for heavy-tailed additive noise distributions, enabled by a truncation method.\n\nSTRENGTHS:\n1. If the result is correct, it provides matching upper and lower bounds for quadratic bandits, which is pretty nice\n\n2. The results hold even for heavy-tailed noise with the help of truncation.\n\nWEAKNESSES:\n1. The definition of the effective dimension $k^*$ does not look correct\n\n2. The proposed algorithm needs to know a lot of problem-related quantities, which is not practical\n\n3. There is no experimental evaluation\n\nQUESTIONS:\nWhat is $dimA$? It is not defined. Is it just $d$?\n\nIn Line 67, in the definition of the effective dimension $k^*$, do you miss a fractional number notion? i.e., $/$ Otherwise, it would be an ill-defined dimension.\n\nThe construction of the lower bound is a bit strange. When you construct x_0 in this way, it will naturally lead to the definition of the effective dim $k^*$. I’m wondering if the current lower bound is truly essential, or just an artifact?\n\nIn my opinion, the non-asymptotic lower bound (Theorem 2.2) is more interesting, why don’t you present it instead of proving a weaker (and asymptotic) lower bound in the main paper?\n\nLIMITATIONS:\nThe zero-th order quadratic optimization problem is a bit simple. But it is a good starting point\n\nThe lower bound proof technique lacks generality. It might be simplified by the generalized le cam’s method. See the textbook “Bandit algorithms”", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}