Learning to Learn by Zeroth-Order Oracle

1 INTRODUCTION . Learning to learn ( L2L ) is a recently proposed meta-learning framework where we leverage deep neural networks to learn optimization algorithms automatically . The most common choice for the learned optimizer is recurrent neural network ( RNN ) since it can capture long-term dependencies and propose parameter updates based on knowledge of previous iterations . By training RNN optimizers on predefined optimization problems , the optimizers are capable of learning to explore the loss landscape and adaptively choose descent directions and steps ( Lv et al. , 2017 ) . Recent works ( Andrychowicz et al. , 2016 ; Wichrowska et al. , 2017 ; Lv et al. , 2017 ) have shown promising results that these learned optimizers can often outperform widely used hand-designed algorithms such as SGD , RMSProp , ADAM , etc . Despite great prospects in this field , almost all previous learned optimizers are gradient-based , which can not be applied to solve optimization problems where explicit gradients are difficult or infeasible to obtain . Such problems mentioned above are called zeroth-order ( ZO ) optimization problems , where the optimizer is only provided with function values ( zeroth-order information ) rather than explicit gradients ( first-order information ) . They are attracting increasing attention for solving ML problems in the black-box setting or when computing gradients is too expensive ( Liu et al. , 2018a ) . Recently , one of the most important applications of ZO optimization is the black-box adversarial attack to welltrained deep neural networks , since in practice only input-output correspondence of targeted models rather than internal model information is accessible ( Papernot et al. , 2017 ; Chen et al. , 2017a ) . Although ZO optimization is popular for solving ML problems , the performance of existing algorithms is barely satisfactory . The basic idea behind these algorithms is to approximate gradients via ZO oracle ( Nesterov & Spokoiny , 2017 ; Ghadimi & Lan , 2013 ) . Given the loss function f with its parameter θ to be optimized ( called the optimizee ) , we can obtain its ZO gradient estimator by : ∇̂f ( θ ) = ( 1/µq ) ∑q i=1 [ f ( θ + µui ) − f ( θ ) ] ui ( 1 ) 1Our code is available at https : //github.com/RYoungJ/ZO-L2L where µ is the smoothing parameter , { ui } are random query directions drawn from standard Gaussian distribution ( Nesterov & Spokoiny , 2017 ) and q is the number of sampled query directions . However , the high variance of ZO gradient estimator which results from both random query directions and random samples ( in stochastic setting ) hampers the convergence rate of current ZO algorithms . Typically , as problem dimension d increases , these ZO algorithms suffer from increasing iteration complexity by a small polynomial of d to explore the higher dimensional query space . In this paper , we propose to learn a zeroth-order optimizer . Instead of designing variance reduced and faster converging algorithms by hand as in Liu et al . ( 2018a ; b ) , we replace parameter update rule as well as guided sampling rule for query directions with learned recurrent neural networks ( RNNs ) . The main contributions of this paper are summarized as follows : • We extend the L2L framework to ZO optimization setting and propose to use RNN to learn ZO update rules automatically . Our learned optimizer contributes to faster convergence and lower final loss compared with hand-designed ZO algorithms . • Instead of using standard Gaussian sampling for random query directions as in traditional ZO algorithms , we propose to learn the Gaussian sampling rule and adaptively modify the search distribution . We use another RNN to adapt the variance of random Gaussian sampling . This new technique helps the optimizer to automatically sample on a more important search space and thus results in a more accurate gradient estimator at each iteration . • Our learned optimizer leads to significant improvement on some ZO optimization tasks ( especially the black-box adversarial attack task ) . We also conduct extensive experiments to analyze the effectiveness of our learned optimizer . 2 RELATED WORK . Learning to learn ( L2L ) In the L2L framework , the design of optimization algorithms is cast as a learning problem and deep neural network is used to learn the update rule automatically . In Cotter & Conwell ( 1990 ) , early attempts were made to model adaptive learning algorithms as recurrent neural network ( RNN ) and were further developed in Younger et al . ( 2001 ) where RNN was trained to optimize simple convex functions . Recently , Andrychowicz et al . ( 2016 ) proposed a coordinatewise LSTM optimizer model to learn the parameter update rule tailored to a particular class of optimization problems and showed the learned optimizer could be applied to train deep neural networks . In Wichrowska et al . ( 2017 ) and Lv et al . ( 2017 ) , several elaborate designs were proposed to improve the generalization and scalability of learned optimizers . Li & Malik ( 2016 ) and Li & Malik ( 2017 ) took a reinforcement learning ( RL ) perspective and used policy search to learn the optimization algorithms ( viewed as RL policies ) . However , most previous learned optimizers rely on first-order information and use explicit gradients to produce parameter updates , which is not applicable when explicit gradients are not available . In this paper , we aim to learn an optimizer for ZO optimization problems . The most relevant work to ours is Chen et al . ( 2017b ) . In this work , the authors proposed to learn a global black-box ( zeroth-order ) optimizer which takes as inputs current query point and function value and outputs the next query point . Although the learned optimizer achieves comparable performance with traditional Bayesian optimization algorithms on some black-box optimization tasks , it has several crucial drawbacks . As is pointed out in their paper , the learned optimizer scales poorly with long training steps and is specialized to a fixed problem dimension . Furthermore , it is not suitable for solving black-box optimization problems of high dimensions . Zeroth-order ( ZO ) optimization The most common method of ZO optimization is to approximate gradient by ZO gradient estimator . Existing ZO optimization algorithms include ZO-SGD ( Ghadimi & Lan , 2013 ) , ZO-SCD ( Lian et al. , 2016 ) , ZO-signSGD ( Liu et al. , 2019 ) , ZO-ADAM ( Chen et al. , 2017a ) , etc . These algorithms suffer from high variance of ZO gradient estimator and typically increase the iteration complexity of their first-order counterparts by a small-degree polynomial of problem dimension d. To tackle this problem , several variance reduced and faster converging algorithms were proposed . ZO-SVRG ( Liu et al. , 2018b ) reduced the variance of random samples by dividing optimization steps into several epochs and maintaining a snapshot point at each epoch whose gradient was estimated using a larger or the full batch . And the snapshot point served as a reference in building a modified stochastic gradient estimate at each inner iteration . ZO-SZVR-G ( Liu et al. , 2018a ) adopted a similar strategy and extended to reduce the variance of both random samples and random query directions . But these methods reduce the variance at the cost of higher query complexity . In this paper , we avoid laborious hand design of these algorithms and aim to learn ZO optimization algorithms automatically . 3 METHOD . 3.1 MODEL ARCHITECTURE . Our proposed RNN optimizer consists of three main parts : UpdateRNN , Guided ZO Oracle , and QueryRNN , as shown in Figure 1 . UpdateRNN The function of the UpdateRNN is to learn the parameter update rule of ZO optimization . Following the idea in Andrychowicz et al . ( 2016 ) , we use coordinatewise LSTM to model the UpdateRNN . Each coordinate of the optimizee shares the same network but maintains its own separate hidden state , which means that different parameters are optimized using the same update rule based on their own knowledge of previous iterations . Different from previous design in the first-order setting , UpdateRNN takes as input ZO gradient estimator in equation 1 rather than exact gradient and outputs parameter update for each coordinate . Thus the parameter update rule is : θt = θt−1 + UpdateRNN ( ∇̂f ( θt ) ) ( 2 ) where θt is the optimizee parameter at iteration t. Besides learning to adaptively compute parameter updates by exploring the loss landscape , the UpdateRNN can also reduce negative effects caused by the high variance of ZO gradient estimator due to its long-term dependency . Guided ZO Oracle In current ZO optimization approaches , ZO gradient estimator is computed by finite difference along the query direction which is randomly sampled from multivariate standard Gaussian distribution . But this estimate suffers from high variance and leads to poor convergence rate when applied to optimize problems of high dimensions ( Duchi et al. , 2015 ) . To tackle this problem , we propose to use some prior knowledge learned from previous iterates during optimization to guide the random query direction search and adaptively modify the search distribution . Specifically , at iteration t , we useN ( 0 , Σt ) to sample query directions ( Σt is produced by the QueryRNN which is introduced later ) and then obtain ZO gradient estimator along sampled directions via ZO Oracle ( equation 1 ) . The learned adaptive sampling strategy will automatically identify important sampling space which leads to a more accurate gradient estimator under a fixed query budget , thus further increases the convergence rate in ZO optimization tasks . For example , in the black-box adversarial attack task , there is usually a clear important subspace for successful attack , and sampling directions from that subspace will lead to much faster convergence . This idea is similar to that of search distribution augmentation techniques for evolutionary strategies ( ES ) such as Natural ES ( Wierstra et al. , 2008 ) , CMA-ES ( Hansen , 2016 ) and Guided ES ( Maheswaranathan et al. , 2018 ) . However , these methods explicitly define the sampling rule whereas we propose to learn the Gaussian sampling rule ( i.e. , the adaption of covariance matrix Σt ) in an automatic manner . QueryRNN We propose to use another LSTM network called QueryRNN to learn the Gaussian sampling rule and dynamically predict the covariance matrix Σt . We assume Σt is diagonal so that it could be predicted in a coordinatewise manner as in the UpdateRNN and the learned QueryRNN is invariant to the dimension of optimizee parameter . It takes as input ZO gradient estimator and parameter update at last iterate ( which could be viewed as surrogate gradient information ) and outputs the sampling variance coordinatewisely , which can be written as : Σt = QueryRNN ( [ ∇̂f ( θt−1 ) , ∆θt−1 ] ) ( 3 ) The intuition is that if estimated gradients or parameter updates of previous iterates are biased toward a certain direction , then we can probably increase the sampling probability toward that direction . Using predicted covariance Σt to sample query directions increases the bias of estimated gradient and reduces the variance , which leads to a tradeoff between bias and variance . The reduction of variance contributes to faster convergence , but too much bias tends to make the learned optimizer stuck at bad local optima ( See more illustrations in Appendix C.3 ) . To balance the bias and variance , at test time we randomly use the covariance of standard Gaussian distribution Id and the predicted covariance Σt as the input of Guided ZO Oracle : Σ′t = XΣt + ( 1 − X ) Id where X ∼ Ber ( p ) is a Bernoulli random variable that trades off between bias and variance . Note that the norm of the sampling covariance ‖Σ′t‖ may not equal to that of standard Gaussian sampling covariance ‖Id‖ , which makes the expectation of the sampled query direction norm ‖u‖ change . To keep the norm of query direction invariant , we then normalize the norm of Σ′t to the norm of Id .

Summary: The paper proposes a zeroth-order optimization framework that employs an RNN to modulate the sampling used to estimate gradients and a second RNN that models the parameter update. More specifically, query directions are sampled from a Gaussian distribution with a diagonal covariance whose evolution is determined by an RNN (QueryRNN). The resulting gradient estimates are used by an RNN (UpdateRNN) that learns the parameter update rule. This framework has the stated advantage that, unlike existing work in ZO optimization, it does not rely upon hand-designed strategies to reduce the variance common to ZO gradient estimators. The paper evaluates the proposed framework on MNIST and CFAR tasks, as well as a synthetic binary classification task. The results demonstrate faster convergence compared to baseline zeroth-order optimization algorithms, while ablations indicate the contributions of the different model components.