Boosting One-Point Derivative-Free Online Optimization via Residual Feedback

1 INTRODUCTION . Zeroth-order optimization ( ZO ) algorithms have been widely used to solve online optimization problems where first or second order information ( i.e. , gradient or Hessian information ) is unavailable at each time instant . Such problems arise , e.g. , in online learning and involve adversarial training Chen et al . ( 2017 ) and reinforcement learning Fazel et al . ( 2018 ) ; Malik et al . ( 2018 ) among others . The goal in online optimization is to minimize a sequence of time-varying objective functions { ft ( x ) } t=1 : T , where the value ft ( xt ) is revealed to the agent after an action xt is selected and is used to adapt the agent ’ s future strategy . Since the future objective functions are not known a priori , the performance of the online decision process can be measured using notions of regret , generally defined as the difference between the total cost incurred by the decision selected by the agent online and the cost of the fixed or varying optimal decision that a clairvoyant agent could select . Perhaps the most popular zeroth-order gradient estimator is the two-point estimator that has been extensively studied in Agarwal et al . ( 2010 ) ; Ghadimi & Lan ( 2013 ) ; Duchi et al . ( 2015 ) ; Ghadimi et al . ( 2016 ) ; Bach & Perchet ( 2016 ) ; Nesterov & Spokoiny ( 2017 ) ; Gao et al . ( 2018 ) ; Roy et al . ( 2019 ) . Specifically , the two-point estimator queries the function value ft ( x ) for twice , for two different realizations of the decision variables , and uses the difference in these function values to estimate the desired gradient , as illustrated by the equation ( Two-point feedback ) : eg ( 2 ) t ( x ) = u ⇣ ft ( x+ u ) ft ( x ) ⌘ , ( 1 ) where > 0 is a parameter and u ⇠ N ( 0 , I ) . However , the two-point gradient estimator can not be used for the solution of non-stationary online optimization problems that arise frequently , e.g. , in online learning . The reason is that in these non-stationary online optimization problems , the objective function being queried is time-varying , and hence only a single function value can be sampled at a given time instant . In this case , the following one-point feedback can be used ( One-point feedback ) : eg ( 1 ) t ( x ) = u ft ( x+ u ) , ( 2 ) which queries the objective function ft ( x ) only once at each time instant . One-point feedback was first proposed and analyzed in Flaxman et al . ( 2005 ) for the solution of online convex optimization problems . Saha & Tewari ( 2011 ) ; Hazan & Levy ( 2014 ) ; Dekel et al . ( 2015 ) showed that the regret of convex online optimization methods using one-point gradient estimation can be improved assuming smoothness or strong convexity of the objective functions and using self-concordant regularization . More recently , Gasnikov et al . ( 2017 ) developed such regret bounds for stochastic convex problems . On the other hand , Hazan et al . ( 2016 ) characterized the convergence of one-point zeroth-order methods for static stochastic non-convex optimization problems . However , as shown in these studies , a limitation of one-point feedback is that the resulting gradient estimator has large variance and , therefore , induces large regret . In addition , the regret analysis for ZO with one-point feedback usually requires the strong assumption that the function value is uniformly upper bounded over time , so this method can not be used for practical non-stationary optimization problems . Contributions : In this paper , we propose a novel one-point gradient estimator for zeroth-order online optimization and develop new regret bounds to study its performance . Specifically , our contributions are as follows . We propose a new one-point feedback scheme which requires a single function evaluation at each time instant . This feedback scheme estimates the gradient using the residual between two consecutive feedback points and we refer to it as residual feedback . We show that our residual feedback induces a smaller gradient estimation variance than the conventional one-point feedback scheme in Flaxman et al . ( 2005 ) ; Gasnikov et al . ( 2017 ) . Furthermore , we provide regret bounds for online convex optimization with our proposed residual feedback estimator . Our analysis relies on a weaker assumption than the one needed in the case of the conventional one-point estimator , and our proposed regret bounds are tighter especially when the value of the objective function is large . In addition , we provide regret bounds for online non-convex optimization with residual feedback . Finally , we present numerical experiments that demonstrate that the proposed residual-feedback estimator significantly outperforms the conventional one-point method in its ability to track the time-varying optimizers of online learning problems . To the best of our knowledge , this is the first time a one-point zeroth-order method is theoretically studied for online non-convex optimization problems . It is also the first time that a one-point gradient estimator demonstrates comparable empirical performance to that of the two-point method . We note that two-point estimators can only be used to solve online non-stationary learning problems in simulations , where the system can be hard coded to be fixed during two queries of the objective function values at two different decision variables . Related work : Zeroth-order methods have been used to solve many different types of optimization problems . For example , Balasubramanian & Ghadimi ( 2018 ) apply ZO to solve a set-constrained optimization problem where the projection onto the constraint set is non-trivial . Gorbunov et al . ( 2018 ) ; Ji et al . ( 2019 ) apply a variance-reduced technique and acceleration schemes to achieve better convergence speed in ZO . Wang et al . ( 2018 ) improve the dependence of the iteration complexity on the dimension of the problem under an additional sparsity assumption on the gradient of the objective function . And Hajinezhad & Zavlanos ( 2018 ) ; Tang & Li ( 2019 ) apply zeroth-order oracles to distributed optimization problems when only bandit feedbacks are available at each local agents . Our proposed residual feedback oracle can be used to solve such online optimization problems as well . Also related is work by Zhang et al . ( 2015 ) that considers non-convex online bandit optimization problems with a single query at each time step . However , this method employs the exploration and exploitation bandit learning framework and the proposed analysis is restricted to a special class of non-convex objective functions . Finally , Agarwal et al . ( 2011 ) ; Hazan & Li ( 2016 ) ; Bubeck et al . ( 2017 ) study online bandit algorithms using ellipsoid methods . In particular , these methods induce heavy computation per step and achieve regret bounds that have bad dependence on the problem dimension . As a comparison , our one-point method is computation light and achieves regret bounds that have better dependence on the problem dimension . 2 PRELIMINARIES AND RESIDUAL FEEDBACK . We first introduce the classes of Lipschitz and smooth functions . Definition 2.1 ( Lipschitz functions ) . The class of Lipschtiz-continuous functions C0,0 satisfies : for any f 2 C 0,0 , |f ( x ) f ( y ) | L0kx yk , 8x , y 2 Rd , where L0 > 0 is the Lipschitz parameter . The class of smooth functions C 1,1 satisfies : for any f 2 C 1,1 , krf ( x ) rf ( y ) k L1kx yk , 8x , y 2 Rd , where L1 > 0 is the smoothness parameter . In ZO , the objective is to estimate the first-order gradient of a function using zeroth-order oracles . Necessarily , we need to perturb the function around the current point along all the directions uniformly in order to estimate the gradient . This motivates us to consider the Gaussian-smoothed version of the function f as introduced in Nesterov & Spokoiny ( 2017 ) , f ( x ) : = Eu⇠N ( 0,1 ) [ f ( x+ u ) ] , where the coordinates of the vector u are i.i.d standard Gaussian random variables . The following bounds on the approximation error of the function f ( x ) have been developed in Nesterov & Spokoiny ( 2017 ) . Lemma 2.2 . Consider a function f and its smoothed version f . It holds that |f ( x ) f ( x ) | ⇢ L0 p d , if f 2 C 0,0 , 2 L1d , if f 2 C 1,1 , and krf ( x ) rf ( x ) k L1 ( d+ 3 ) 3/2 , if f 2 C 1,1 . The smoothed function f ( x ) satisfies the following amenable property Nesterov & Spokoiny ( 2017 ) . Lemma 2.3 . If f 2 C0,0 is L0-Lipschitz , then f 2 C1,1 with Lipschitz constant L1 = p d 1 L0 . Consider the following online bandit optimization problem . min x2X T 1X t=0 ft ( x ) , ( P ) where X ⇢ Rd is a convex set and { ft } t is a random sequence of objective functions . In this setting , the objective functions { ft } t are unknown a priori and their derivatives are unavailable . At time t , a new objective function ft is randomly generated independent of an agent ’ s decisions , and then the agent queries the objective function value at certain perturbed points and use them to update the current policy parameters . The goal of the agent is to minimize a certain regret function . Such an online setting often occurs in non-stationary learning scenarios where either the system is time-varying on its own or a single query of the function ft changes the system state ( i.e. , ft changes to ft+1 ) . In this non-stationary setting , the conventional two-point feedback scheme is known to be impractical as it requires to evaluate ft at two different points at the same time t. Instead , it is natural to use the one-point feedback scheme ( 2 ) in Gasnikov et al . ( 2017 ) . However , the gradient estimate based on the above one-point feedback induces a large variance that leads to a large regret . In this paper , we focus on such an one-point derivative-free setting and propose the following novel one-point residual feedback scheme for estimating the gradient with reduced variance . ( Residual feedback ) : egt ( xt ) : = ut ft ( xt + ut ) ft 1 ( xt 1 + ut 1 ) , ( 3 ) where ut 1 , ut ⇠ N ( 0 , I ) are independent random vectors . To elaborate , the residual feedback in ( 3 ) queries ft at a single perturbed point xt + ut , and then subtracts it by ft 1 ( xt 1 + ut 1 ) obtained from the previous iteration . We name such a scheme as one-point residual feedback . Next , we explore some basic properties of the residual feedback . We first show that this estimator is an unbiased gradient estimate of the smoothed function f , t. Lemma 2.4 . The residual feedback satisfies E ⇥ egt ( xt ) ⇤ = rf , t ( xt ) for all xt 2 X and t. Proof . By the fact that ut has zero mean and is independent from ut 1 and xt 1 . We consider the following ZO algorithm with residual feedback ( ZO with residual feedback ) : xt+1 = ⇧X xt ⌘g̃t ( xt ) , ( 4 ) where ⌘ is the learning rate and ⇧X is the projection operator onto the set X . The update ( 4 ) can be implemented assuming that the objective function can be queried at points outside the feasible set X , similar to the methods considered in Duchi et al . ( 2015 ) ; Bach & Perchet ( 2016 ) ; Gasnikov et al . ( 2017 ) . Note that it is possible to modify the update ( 4 ) so that the iterates are guaranteed to be within the feasible set X . This modification and related analysis can be found in Section H in the supplementary material . The requirement that the objective function is evaluated at feasible points in derivative-free optimization algorithms has also been considered in Bubeck et al . ( 2017 ) ; Bilenne et al . ( 2020 ) . Specifically , Bubeck et al . ( 2017 ) develop the so called ellipsoid method , which requires computation of an ellipsoid containing the optimizer at each time step . On the other hand , almost concurrently with this work , Bilenne et al . ( 2020 ) proposed a similar oracle as in ( 3 ) for a static convex optimization problem with specific objective and constraint functions . Next , we bound the second moment of the gradient estimate based on the residual feedback . Lemma 2.5 ( Second moment ) . Assume that ft 2 C0,0 with Lipschitz constant L0 for all time t. Then , under the ZO update rule in ( 4 ) , the second moment of the residual feedback satisfies : for all t , E [ kegt ( xt ) k2 ] 4dL20⌘ 2 2 E [ kegt 1 ( xt 1 ) k2 ] +Dt , ( 5 ) where Dt : = 16L 2 0 ( d+ 4 ) 2 + 2d 2 E ⇥ ft ( xt 1 + ut 1 ) ft 1 ( xt 1 + ut 1 ) 2⇤ . The above lemma shows that the second moment of the residual feedback can be bounded by a perturbed contraction , provided that we choose ⌘ and such that the contracting rate ↵ = 4dL20⌘ 2 2 < 1 . As we show later in the analysis , such a contraction property leads to a small variance of the residual feedback that helps reduce the regret of the online ZO algorithm .

Summary: The paper considers online optimization with zero-order oracle. Motivated by nonstationarity of the objective function, impracticality is underlined for the two-point feedback approach. Instead, staying in the one-point setting, the proposed approach reuses the objective value from the previous round of observations, which is called as residual feedback. The variance of the corresponding proxy for the subgradient is estimated under more relaxed assumptions than existing in the literature. The proposed approach leads to smaller variance and better regret bounds. Regret bounds are proved for smooth/non-smooth convex/non-convex cases, the non-convex case being analyzed for the first time in the literature. Numerical experiments show that the practical performance of the proposed gradient estimator is better than that of the existing one-point feedback methods and is close to the performance of the one-point approach with two observations per round. The latter approach can be impractical for some applications.