Boosting One-Point Derivative-Free Online Optimization via Residual Feedback

1 INTRODUCTION . Zeroth-order optimization ( ZO ) algorithms have been widely used to solve online optimization problems where first or second order information ( i.e. , gradient or Hessian information ) is unavailable at each time instant . Such problems arise , e.g. , in online learning and involve adversarial training Chen et al . ( 2017 ) and reinforcement learning Fazel et al . ( 2018 ) ; Malik et al . ( 2018 ) among others . The goal in online optimization is to minimize a sequence of time-varying objective functions { ft ( x ) } t=1 : T , where the value ft ( xt ) is revealed to the agent after an action xt is selected and is used to adapt the agent ’ s future strategy . Since the future objective functions are not known a priori , the performance of the online decision process can be measured using notions of regret , generally defined as the difference between the total cost incurred by the decision selected by the agent online and the cost of the fixed or varying optimal decision that a clairvoyant agent could select . Perhaps the most popular zeroth-order gradient estimator is the two-point estimator that has been extensively studied in Agarwal et al . ( 2010 ) ; Ghadimi & Lan ( 2013 ) ; Duchi et al . ( 2015 ) ; Ghadimi et al . ( 2016 ) ; Bach & Perchet ( 2016 ) ; Nesterov & Spokoiny ( 2017 ) ; Gao et al . ( 2018 ) ; Roy et al . ( 2019 ) . Specifically , the two-point estimator queries the function value ft ( x ) for twice , for two different realizations of the decision variables , and uses the difference in these function values to estimate the desired gradient , as illustrated by the equation ( Two-point feedback ) : eg ( 2 ) t ( x ) = u ⇣ ft ( x+ u ) ft ( x ) ⌘ , ( 1 ) where > 0 is a parameter and u ⇠ N ( 0 , I ) . However , the two-point gradient estimator can not be used for the solution of non-stationary online optimization problems that arise frequently , e.g. , in online learning . The reason is that in these non-stationary online optimization problems , the objective function being queried is time-varying , and hence only a single function value can be sampled at a given time instant . In this case , the following one-point feedback can be used ( One-point feedback ) : eg ( 1 ) t ( x ) = u ft ( x+ u ) , ( 2 ) which queries the objective function ft ( x ) only once at each time instant . One-point feedback was first proposed and analyzed in Flaxman et al . ( 2005 ) for the solution of online convex optimization problems . Saha & Tewari ( 2011 ) ; Hazan & Levy ( 2014 ) ; Dekel et al . ( 2015 ) showed that the regret of convex online optimization methods using one-point gradient estimation can be improved assuming smoothness or strong convexity of the objective functions and using self-concordant regularization . More recently , Gasnikov et al . ( 2017 ) developed such regret bounds for stochastic convex problems . On the other hand , Hazan et al . ( 2016 ) characterized the convergence of one-point zeroth-order methods for static stochastic non-convex optimization problems . However , as shown in these studies , a limitation of one-point feedback is that the resulting gradient estimator has large variance and , therefore , induces large regret . In addition , the regret analysis for ZO with one-point feedback usually requires the strong assumption that the function value is uniformly upper bounded over time , so this method can not be used for practical non-stationary optimization problems . Contributions : In this paper , we propose a novel one-point gradient estimator for zeroth-order online optimization and develop new regret bounds to study its performance . Specifically , our contributions are as follows . We propose a new one-point feedback scheme which requires a single function evaluation at each time instant . This feedback scheme estimates the gradient using the residual between two consecutive feedback points and we refer to it as residual feedback . We show that our residual feedback induces a smaller gradient estimation variance than the conventional one-point feedback scheme in Flaxman et al . ( 2005 ) ; Gasnikov et al . ( 2017 ) . Furthermore , we provide regret bounds for online convex optimization with our proposed residual feedback estimator . Our analysis relies on a weaker assumption than the one needed in the case of the conventional one-point estimator , and our proposed regret bounds are tighter especially when the value of the objective function is large . In addition , we provide regret bounds for online non-convex optimization with residual feedback . Finally , we present numerical experiments that demonstrate that the proposed residual-feedback estimator significantly outperforms the conventional one-point method in its ability to track the time-varying optimizers of online learning problems . To the best of our knowledge , this is the first time a one-point zeroth-order method is theoretically studied for online non-convex optimization problems . It is also the first time that a one-point gradient estimator demonstrates comparable empirical performance to that of the two-point method . We note that two-point estimators can only be used to solve online non-stationary learning problems in simulations , where the system can be hard coded to be fixed during two queries of the objective function values at two different decision variables . Related work : Zeroth-order methods have been used to solve many different types of optimization problems . For example , Balasubramanian & Ghadimi ( 2018 ) apply ZO to solve a set-constrained optimization problem where the projection onto the constraint set is non-trivial . Gorbunov et al . ( 2018 ) ; Ji et al . ( 2019 ) apply a variance-reduced technique and acceleration schemes to achieve better convergence speed in ZO . Wang et al . ( 2018 ) improve the dependence of the iteration complexity on the dimension of the problem under an additional sparsity assumption on the gradient of the objective function . And Hajinezhad & Zavlanos ( 2018 ) ; Tang & Li ( 2019 ) apply zeroth-order oracles to distributed optimization problems when only bandit feedbacks are available at each local agents . Our proposed residual feedback oracle can be used to solve such online optimization problems as well . Also related is work by Zhang et al . ( 2015 ) that considers non-convex online bandit optimization problems with a single query at each time step . However , this method employs the exploration and exploitation bandit learning framework and the proposed analysis is restricted to a special class of non-convex objective functions . Finally , Agarwal et al . ( 2011 ) ; Hazan & Li ( 2016 ) ; Bubeck et al . ( 2017 ) study online bandit algorithms using ellipsoid methods . In particular , these methods induce heavy computation per step and achieve regret bounds that have bad dependence on the problem dimension . As a comparison , our one-point method is computation light and achieves regret bounds that have better dependence on the problem dimension . 2 PRELIMINARIES AND RESIDUAL FEEDBACK . We first introduce the classes of Lipschitz and smooth functions . Definition 2.1 ( Lipschitz functions ) . The class of Lipschtiz-continuous functions C0,0 satisfies : for any f 2 C 0,0 , |f ( x ) f ( y ) | L0kx yk , 8x , y 2 Rd , where L0 > 0 is the Lipschitz parameter . The class of smooth functions C 1,1 satisfies : for any f 2 C 1,1 , krf ( x ) rf ( y ) k L1kx yk , 8x , y 2 Rd , where L1 > 0 is the smoothness parameter . In ZO , the objective is to estimate the first-order gradient of a function using zeroth-order oracles . Necessarily , we need to perturb the function around the current point along all the directions uniformly in order to estimate the gradient . This motivates us to consider the Gaussian-smoothed version of the function f as introduced in Nesterov & Spokoiny ( 2017 ) , f ( x ) : = Eu⇠N ( 0,1 ) [ f ( x+ u ) ] , where the coordinates of the vector u are i.i.d standard Gaussian random variables . The following bounds on the approximation error of the function f ( x ) have been developed in Nesterov & Spokoiny ( 2017 ) . Lemma 2.2 . Consider a function f and its smoothed version f . It holds that |f ( x ) f ( x ) | ⇢ L0 p d , if f 2 C 0,0 , 2 L1d , if f 2 C 1,1 , and krf ( x ) rf ( x ) k L1 ( d+ 3 ) 3/2 , if f 2 C 1,1 . The smoothed function f ( x ) satisfies the following amenable property Nesterov & Spokoiny ( 2017 ) . Lemma 2.3 . If f 2 C0,0 is L0-Lipschitz , then f 2 C1,1 with Lipschitz constant L1 = p d 1 L0 . Consider the following online bandit optimization problem . min x2X T 1X t=0 ft ( x ) , ( P ) where X ⇢ Rd is a convex set and { ft } t is a random sequence of objective functions . In this setting , the objective functions { ft } t are unknown a priori and their derivatives are unavailable . At time t , a new objective function ft is randomly generated independent of an agent ’ s decisions , and then the agent queries the objective function value at certain perturbed points and use them to update the current policy parameters . The goal of the agent is to minimize a certain regret function . Such an online setting often occurs in non-stationary learning scenarios where either the system is time-varying on its own or a single query of the function ft changes the system state ( i.e. , ft changes to ft+1 ) . In this non-stationary setting , the conventional two-point feedback scheme is known to be impractical as it requires to evaluate ft at two different points at the same time t. Instead , it is natural to use the one-point feedback scheme ( 2 ) in Gasnikov et al . ( 2017 ) . However , the gradient estimate based on the above one-point feedback induces a large variance that leads to a large regret . In this paper , we focus on such an one-point derivative-free setting and propose the following novel one-point residual feedback scheme for estimating the gradient with reduced variance . ( Residual feedback ) : egt ( xt ) : = ut ft ( xt + ut ) ft 1 ( xt 1 + ut 1 ) , ( 3 ) where ut 1 , ut ⇠ N ( 0 , I ) are independent random vectors . To elaborate , the residual feedback in ( 3 ) queries ft at a single perturbed point xt + ut , and then subtracts it by ft 1 ( xt 1 + ut 1 ) obtained from the previous iteration . We name such a scheme as one-point residual feedback . Next , we explore some basic properties of the residual feedback . We first show that this estimator is an unbiased gradient estimate of the smoothed function f , t. Lemma 2.4 . The residual feedback satisfies E ⇥ egt ( xt ) ⇤ = rf , t ( xt ) for all xt 2 X and t. Proof . By the fact that ut has zero mean and is independent from ut 1 and xt 1 . We consider the following ZO algorithm with residual feedback ( ZO with residual feedback ) : xt+1 = ⇧X xt ⌘g̃t ( xt ) , ( 4 ) where ⌘ is the learning rate and ⇧X is the projection operator onto the set X . The update ( 4 ) can be implemented assuming that the objective function can be queried at points outside the feasible set X , similar to the methods considered in Duchi et al . ( 2015 ) ; Bach & Perchet ( 2016 ) ; Gasnikov et al . ( 2017 ) . Note that it is possible to modify the update ( 4 ) so that the iterates are guaranteed to be within the feasible set X . This modification and related analysis can be found in Section H in the supplementary material . The requirement that the objective function is evaluated at feasible points in derivative-free optimization algorithms has also been considered in Bubeck et al . ( 2017 ) ; Bilenne et al . ( 2020 ) . Specifically , Bubeck et al . ( 2017 ) develop the so called ellipsoid method , which requires computation of an ellipsoid containing the optimizer at each time step . On the other hand , almost concurrently with this work , Bilenne et al . ( 2020 ) proposed a similar oracle as in ( 3 ) for a static convex optimization problem with specific objective and constraint functions . Next , we bound the second moment of the gradient estimate based on the residual feedback . Lemma 2.5 ( Second moment ) . Assume that ft 2 C0,0 with Lipschitz constant L0 for all time t. Then , under the ZO update rule in ( 4 ) , the second moment of the residual feedback satisfies : for all t , E [ kegt ( xt ) k2 ] 4dL20⌘ 2 2 E [ kegt 1 ( xt 1 ) k2 ] +Dt , ( 5 ) where Dt : = 16L 2 0 ( d+ 4 ) 2 + 2d 2 E ⇥ ft ( xt 1 + ut 1 ) ft 1 ( xt 1 + ut 1 ) 2⇤ . The above lemma shows that the second moment of the residual feedback can be bounded by a perturbed contraction , provided that we choose ⌘ and such that the contracting rate ↵ = 4dL20⌘ 2 2 < 1 . As we show later in the analysis , such a contraction property leads to a small variance of the residual feedback that helps reduce the regret of the online ZO algorithm .

Summary: This manuscript considers online zeroth order optimization and it develops a gradient estimator based on one query per function. In particular, the proposed method mimics two-point estimators by evaluating two consecutive functions at perturbations of an iterate, as shown in equation (3). Although one-point gradient estimates are possible, they have impractically large variances. Given this limitation and the wide need of zeroth order optimization (in particular in RL), the study of two-point estimators is important.