Title: ReLIZO: Sample Reusable Linear Interpolation-based Zeroth-order Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Zero-order Optimization; Linear Interpolation; Reusing Strategy;
Abstract: Gradient estimation is critical in zeroth-order optimization methods, which aims to obtain the descent direction by sampling update directions and querying function evaluations. Extensive research has been conducted including smoothing and linear interpolation. The former methods smooth the objective function, causing a biased gradient estimation, while the latter often enjoys more accurate estimates, at the cost of large amounts of samples and queries at each iteration to update variables. This paper resorts to the linear interpolation strategy and proposes to reduce the complexity of gradient estimation by reusing queries in the prior iterations while maintaining the sample size unchanged. Specifically, we model the gradient estimation as a quadratically constrained linear program problem and manage to derive the analytical solution. It innovatively decouples the required sample size from the variable dimension without extra conditions required, making it able to leverage the queries in the prior iterations. Moreover, part of the intermediate variables that contribute to the gradient estimation can be directly indexed, significantly reducing the computation complexity. Experiments on both simulation functions and real scenarios (black-box adversarial attacks neural architecture search, and parameter-efficient fine-tuning for large language models), show its efficacy and efficiency. Our code is available at https://github.com/Thinklab-SJTU/ReLIZO.git.

ReLIZO: Sample Reusable Linear Interpolation-based Zeroth-order Optimization

Abstract

Gradient estimation is critical in zeroth-order optimization methods, which aims to obtain the descent direction by sampling update directions and querying function evaluations. Extensive research has been conducted including smoothing and linear interpolation. The former methods smooth the objective function, causing a biased gradient estimation, while the latter often enjoys more accurate estimates, at the cost of large amounts of samples and queries at each iteration to update variables. This paper resorts to the linear interpolation strategy and proposes to reduce the complexity of gradient estimation by reusing queries in the prior iterations while maintaining the sample size unchanged. Specifically, we model the gradient estimation as a quadratically constrained linear program problem and manage to derive the analytical solution. It innovatively decouples the required sample size from the variable dimension without extra conditions required, making it able to leverage the queries in the prior iterations. Moreover, part of the intermediate variables that contribute to the gradient estimation can be directly indexed, significantly reducing the computation complexity. Experiments on both simulation functions and real scenarios (black-box adversarial attacks neural architecture search, and parameterefficient fine-tuning for large language models), show its efficacy and efficiency. Our code is available at https://github.com/Thinklab-SJTU/ReLIZO.git .

1 Introduction

Zeroth-order optimization (ZO) aims to estimate the gradients by only function evaluations and solve optimization problems by the descent method. It has been successfully applied to many fields, including science [41, 25], finance [35], and artificial intelligence [37, 45, 47], where the optimization functions are usually black-box without available gradients w.r.t. the variables. Typical ZO methods iteratively perform three major steps [35]: 1) estimate the gradient by sampling directions to update the variable and querying the function evaluation, 2) rectify the descent direction through momentum, element-wise sign operation, stochastic variance reduced algorithms, etc., and 3) update the variables according to the descent direction. Wherein, the first step, i.e. gradient estimation, is critical since it provides the essential direction to update variables, which have been explored by many recent works [5, 30].

Gradient estimation methods in ZO can be roughly categorized into two groups [5]: linear interpolation technique [4, 30] and smoothing strategy [39, 12]. The former proposes to sample a set of linearly independent directions to form an invertible matrix and compute the gradient through linear interpolation. Though such methods can get better descent directions, the sample size of directions should be the same as the dimension of variables, leading to an intractable complexity of building an invertible matrix and computing the matrix inversion as the dimension increases. A recent work [30]

makes the sample size in linear interpolation strategy independent of the dimension of variables yet involves an extra orthogonal requirement of the sampled vectors. The latter group proposes to estimate the gradient by a sum of directional derivatives along random directions e.g. Gaussian and uniform on a unit sphere. Such methods are simple to implement and the sample size directions are independent of the variable dimension. However, it provides a gradient estimation of a smoothed version of the objective function [35], which differs from the original one with a bound w.r.t. a smoothing parameter [39]. Additionally, the prior work [5] shows that smoothing methods require significantly more samples than linear interpolation strategies to obtain an accurate estimation The work [28] leverages information theory showing that under the Lipschitz continuous gradient assumption, ZO methods require sample complexities growing polynomially with the variable dimension.

Overall, the prior gradient estimation methods require large amounts of samples at each iteration to update variables. Nevertheless, in many real scenarios, one query of function evaluation can require large amounts of resources such as tasks in AutoML and reinforcement learning, which significantly restricts the acceptable sample size at each iteration and thus slow down the convergence of ZO methods. Consequently, it has been urgently demanded to reduce the number of queries while ensuring the convergence of ZO methods.

To this end, we delve into the ZO pipeline and observe that the magnitudes of steps to update variables are limited, making it possible to reuse the queries in the prior iterations. It is hoped that such a reusing strategy can be explored to significantly reduce the number of queries of function evaluation while maintaining the sample size unchanged at each iteration. However, it is hard to be directly applied to the prior gradient estimation methods. On one hand, for smoothing strategies, it is hard to guarantee the reused samples obey the demanded distribution, which will further increase the gradient estimation bias. On the other hand, methods based on linear interpolation techniques require a large computation complexity since building an invertible matrix needs to compute the null space of reused samples. Additionally, another work [30] requires orthogonal sampled vectors, which can hardly be satisfied by the prior samples, making it impossible to reuse the samples.

This work refers to Taylor's expansion and proposes to estimate gradients through a linear combination of limited direction samples and their queried function evaluations. We then model the gradient estimation as a quadratically constrained linear program (QCLP) and derive the analytical solution. Our method can reduce the required sample size (decoupled from the variable dimension) for linear interpolation strategies without the orthogonal condition required. Consequently, it supports to reuse the queries in the prior iterations. Moreover, part of the intermediate variables that contribute to the gradient estimation can be directly indexed from the computation in the prior iterations, significantly reducing the computation complexity during the ZO process. Theoretical analysis is performed to show that our method can be regarded as a general version of the traditional linear interpolation strategy and has a similar convergence rate. In summary, our contributions lie in:

1) We introduce to reuse the prior queries to speed up the ZO procedure. Unlike recent works that cannot reuse the queries due to the requirements of sampled vectors to be orthogonal in linear interpolation strategies or obey a specific distribution in smoothing techniques, we introduce to estimate the gradient with an arbitrary number of sampled vectors without orthogonal condition, making it possible to leverage the queries in the prior iterations. To the best of our knowledge, this work is the first that attempts to reuse queries to speed up the ZO procedure.

2) We model the gradient estimation in ZO as a QCLP and derive the analytical solution by Lagrange multiplier. Theoretical analysis also proves that our method has a similar convergence rate as conventional linear interpolation techniques. Moreover, after combining our gradient estimator based on QCLP with the reusing strategy, part of the intermediate variables that contribute to the analytical solution can also be directly indexed from the computation in the prior iterations, significantly reducing the complexity when the sample size is much lower than the variable dimension. Overall, our method is more efficient than linear interpolation-based methods with a comparable accurate estimation of the gradient.

3) We conduct extensive experiments to show the efficacy and efficiency of our method. We first compare our method with recent ZO methods on the CUTEst, showing that it has faster convergence speed and better solutions. Ablation studies show that the performance drop is negligible even with more than 50% reuse rate. We then conduct experiments on the black-box adversarial attack task, showing that it has lower attack loss and 5% better attack success rate compared to peer ZO solvers

with similar final ℓ 2 distortion. Our method is further applied to the Neural Architecture Search (NAS) on NAS-Bench-201, outperforming other ZO solvers with 39% less number of queries.

2 Related Work

Zeroth-order optimization. There are typically two types of gradient estimation in ZO algorithms: one-point estimation [38, 13, 54] and multi-point estimation [1, 39, 22, 19]. Since multi-point estimation is more accurate, some works [33, 52] made further research on ZO convex problems. As for the non-convex setting, the prior works [39, 22] proposed ZO-GD and its stochastic counterpart ZO-SGD. ZO algorithms have been widely employed in various scenarios, such as black-box adversarial attacks [11, 27], AutoML [47] and transfer learning [44]. It is worth noting that the gradient estimation is the most critical part of ZO solvers, which is the focus of the above methods.

Gradient estimation via smoothing strategy. Methods based on the smoothing strategy [3, 36] estimate gradients by averaging over several directional derivatives by sampling the direction from either a Gaussian distribution [39] or a uniform distribution [20] on a unit sphere. Referring to the prior smoothing strategy based ZO algorithms [39, 22], ZO-signSGD [34] yields faster convergence but only guarantees to converge to a neighborhood of a solution, and ZO-AdaMM [12] uses adaptive momentum to enhance the optimization.

Gradient estimation via linear interpolation. Methods based on linear interpolation [15, 4] estimate gradient by solving linear programming. Conn et al. [15] derived the error bounds between the true function value and interpolating polynomials, including linear and quadratic approximation models. Berahas et al.[4] showed that linear interpolation gives better gradients than Gaussian smoothing in derivative-free optimization. The prior work [5] derived bounds on the sample size and the sampling radius, showing that smoothing strategy requires significantly more samples to obtain an accurate estimation than linear interpolation. Kozak et al.[30] approximated the gradient by finite differences computed in a set of orthogonal random directions, which is a special case of linear interpolation. This work also focuses on the scope of linear interpolation due to its more accurate estimation of gradient than smoothing techniques.

Remarks. Our approach has few constraints on the sample size, similar to the smoothing techniques. Meanwhile, we estimate the gradients by solving a linear program problem, similar to the linear interpolation strategies. Overall, our approach enjoys the efficiency of smoothing techniques while maintaining estimation accuracy.

3 The Proposed ReLIZO Method

Similar to the prior ZO methods [22, 12], we consider the problem of finding a point x ∗ ∈ R d such that f ( x ∗ ) = min x f ( x ) . The function f : R d → R is black-box and satisfies the following assumptions:

Assumption 3.1. 0 < f ( x 0 ) -f ( x ∗ ) < ∞ , where x 0 is the initial point.

Assumption 3.2. The function f is continuously differentiable, and ∇ f is L -Lipschitz continuous for ∀ x ∈ R d .

3.1 Rethinking Gradient Estimation as QCLP

Consider the Taylor's expansion of function f :

The remainder can be omitted in many scenarios. For large-scale applications it can be hard to compute the number of function evaluations required to construct quadratic models.

Given n random vectors as candidate updates to the variable ∆ x ∈ R n × d , we denote the i-th row of it as ∆ i x . The corresponding function evaluation { f ( x +∆ i x ) } n i =1 can be queried and compose ∆ f ∈ R n , whose entries ∆ i f = f ( x +∆ i x ) -f ( x ) , ∀ i = 1 , ..., n . According to Eq. 1, we can have the equations as follows:

In this work, we mainly focus on the case where n < d , hoping to obtain approximately accurate gradients with fewer queries. To this end, we estimate the direction of gradient ∇ f ( x ) as g ( x ) by linearly combining the n sampled vectors { ∆ i x } n i =1 , yielding g ( x ) = ∆ ⊤ x α , where α ∈ R n represents the coefficients of the linear combination. The inner product of normalized g ( x ) and ∇ f ( x ) increases when their directions are getting closed, which can be formulated as:

Substitute Eq. 2 into Eq. 3 and define A x = ∆ x ∆ ⊤ x ∈ R n × n , We can model the estimation of gradient direction as a quadratically constrained linear program (QCLP):

We employ the Lagrange multiplier method to address the QCLP problem (4). The Lagrangian function is defined as: L ( α, λ ) = α ⊤ ∆ f -λ ( α ⊤ A x α -1) , where λ is the Lagrange multiplier. Taking partial derivatives w.r.t. α and setting it to zero, we have ∂L ∂α = ∆ f -2 λA x α = 0 and A x α = 1 2 λ ∆ f , yielding the direction of α ∝ A -1 x ∆ f . Consequently, we obtain the estimated direction of the gradient:

Finally, we initialize the magnitude of the descent direction σ k with learning rate η and adopt the line search strategy to update σ k to satisfy the Armijo-Goldstein condition [2, 23].

3.2 Reusing the Queries in the Prior Iterations

Unlike recent works unable to reuse the queries due to the requirements of sampled vectors to be orthogonal in linear interpolation strategies [30] or to obey a specific distribution in smoothing techniques [39], our method has few constraints on the sampled vectors ∆ x , making it possible to reuse the queries in the prior iterations.

Suppose x k -1 is the variable and ∆ x k -1 is the n random vectors at ( k -1) -th iteration. We have the queried function values at points { x k -1 +∆ i x k -1 } n i =1 . The estimated descent direction is g ( x k -1 ) with step σ k -1 . After updating the variable at ( k -1) -th iteration, we have x k = x k -1 -σ k -1 g ( x k -1 ) . Since f ( x k -1 ) has also been queried, we have n +1 queried points and can collect n +1 random vectors for x k with known queries at k -th iteration as:

Distances between x k and the prior n +1 queried points are ∥ x k -x k -1 ∥ = ∥ σ k -1 g ( x k -1 ) ∥ and ∥ x k -( x k -1 +∆ i x k -1 ) ∥ = ∥ σ k -1 g ( x k -1 ) + ∆ i x k -1 ∥ , ∀ i = { 1 , ..., n } .

According to Taylor's expansion in Eq. 1, the remainder o ( y -x ) will be non-negligible as ∥ y -x ∥ increases. Therefore, we introduce a reusable distance bound b to filter out the samples far away from the current point x k , i.e., the reused vectors are obtained by:

After getting the reusable queries, we then randomly sample another n -| ∆ r x k | vectors { p j } to build ∆ x k ∈ R n × d for gradient estimation at k -th iteration:

Overall, in the k -th iteration, only n -| ∆ r x k | new queries should be conducted. The ablation study in Sec. 4.2 shows that as the sample size increases, even reusing 80% prior queries can also find good solutions.

Discussion on the reusable distance bound b : ZO methods estimate the directional derivative by difference and restrict the distances between the samples and current point to a small value, which has the same order as the stepsize (initial learning rate) η ∼ O ( 1 d ) as previous works (Table 1 in [29]). In our method, the reusable distance bound b restricts the distances between the reusable samples and current point, which should has the same order of magnitude as the stepsize. Then we choose b ∼ O ( 1 d ) . Ablation studies in Fig. 2 show that b = 2 η works pretty well in different optimization tasks with different sample sizes N , thus we set b = 2 η by default.

Algorithm 1 ReLIZO

Input: A Black-box function f ( x ) ; Sample size at each iteration n ; Reusable distance bound b ; Number of iterations K ; Learning rate η .

Output: The best solution x ∗ that minimizes f ( x ) .

1: ˜ ∆ x 0 = ∅ ;

2: for k = 0 to K -1 do

3: Collect the reusable vectors from the last iteration and get ∆ r x k by Eq. 6 and Eq. 7;

4: Sample another n -| ∆ r x k | vectors P and build ∆ x k ∈ R n × d by Eq. 8;

5: Query the function evaluation for newly sampled vectors f ( x k + p j ) , ∀ p j ∈ P ;

6: Compute A x k by Eq. 11;

7: Obtain the estimated gradient direction g ( x k ) ∝ ∆ ⊤ x k A -1 x k ∆ f by Eq. 5;

8: Initialize the magnitude of descent direction σ k as η and update σ k by the backtracking line search;

9: Update the variable: x k +1 ← x k -σ k g ( x k ) .

10: end for

11: Return the solution x K and the best solution x ∗ satisfying f ( x ∗ ) ≥ f ( x k ) , ∀ k ∈ { 0 , ..., K } .

3.3 Reducing the Computation Complexity by Indexing

Eq. 5 shows that the estimated direction is related to the inverse of A x = ∆ x ∆ ⊤ x . When the sample size is much smaller than the dimension of variables, i.e., n << d , the computation complexity of A x is O ( n 2 d ) dominating the complexity of matrix inversion O ( n 3 ) . Nevertheless, the complexity can be significantly reduced when a large number of rows of ∆ x is reused.

We denote A x k -1 = ∆ x k -1 ∆ ⊤ x k -1 , which has been computed in the ( k -1) -th iteration. In the k -th iteration, ∆ r x k in Eq. 7 is the reused samples. In the following, we introduce an efficient strategy to compute ∆ r x k ∆ r ⊤ x k .

First, we define ¯ ∆ x k -1 ∈ R ( n +1) × d to denote the vector differences between x k -1 and all n + 1 reusable points including { x k -1 +∆ i x k -1 } n i =1 ∪ { x k -1 } . Hence, we get ¯ A x k -1 = ¯ ∆ x k -1 ¯ ∆ ⊤ x k -1 as:

Without losing generality, we suppose ∆ r x k has n k rows and the i -th row of it reuses the r i -th row vector in ¯ ∆ x k -1 , and R = [ r 1 , ...r n k ] . Since g ( x k -1 ) = ∆ ⊤ x k -1 α , we have:

where ¯ ∆ R x k -1 = [ ¯ ∆ r 1 x k -1 ; ... ; ¯ ∆ r n k x k -1 ] ∈ R n k × d , and the subtraction should be broadcast among the first dimension. Then, we can derive ∆ r x k ∆ r ⊤ x k as follows:

where the addition and subtractions should broadcast among the corresponding dimensions. Note that ¯ ∆ R x k -1 ¯ ∆ R ⊤ x k -1 and ¯ ∆ R x k -1 ∆ ⊤ x k -1 can be directly indexed from ¯ A x k -1 (Eq. 9), that has been computed in the prior iterations. Hence, we can simplify Eq. 11 as:

where R = [ r 1 , ...r n k ] is the index of reusing vectors in ¯ ∆ x k -1 , ¯ A ( R,R ) x k -1 ∈ R n k × n k denotes to index R rows and R columns from ¯ A x k -1 , ¯ A R x k -1 ∈ R n k × n denotes to index R rows from ¯ A x k -1 , and α ∈ R n × 1 is the solution to QCLP problem (4). Hence, by reusing n k vectors ∆ r x k and newly sampled vectors P ∈ R ( n -n k ) × d , A x k = ∆ x k ∆ ⊤ x k can be obtained by:

Consequently, the complexity of computing A x k can be reduced to O ( n k n +( n -n k ) nd ) , which decreases as the reuse rate n k n increases. The overall computation complexity is analyzed and compared with other ZO methods in Appendix B. The algorithm flow is shown in Alg. 1.

3.4 Convergence Analysis

We start the analysis with a lemma that estimates the distance between the estimated gradient in our method and the projected exact gradient of the function.

Lemma 3.1. Let f satisfies Assumption 3.1 and 3.2. For every k ∈ N , ∆ x k ∈ R n × d and ∥ ∆ i x k ∥ < b :

Then we can derive the distance bound between the estimated gradient and the exact gradient of the function.

Lemma 3.2. Let the SVD of ∆ x k = U Λ V ⊤ , A -1 x k = (∆ x k ∆ ⊤ x k ) -1 , ρ k denotes the spectral radius of A -1 x k . Based on Lemma 3.1, we have:

where r = ∑ d j = n v 2 j ∥∇ f ( x k ) ∥ 2 < 1 , v = V ⊤ ∇ f ( x k ) .

Consequently, we can derive the following proposition:

Proposition 3.3. Suppose f is convex and ∃ θ ∈ [0 , 1) satisfying ρ k ≤ 4( θ 2 -r ) ∥∇ f ( x ) ∥ 2 L 2 b 2 n , we have:

which yields a sublinear convergence. Proof can be found in Appendix A. Furthermore, our experimental results demonstrate that our method can converge in many scenarios, including simulation benchmarks and real-world applications.

3.5 Relationship with Linear Interpolation Methods

We show that our method can be regarded as a general version of the linear interpolation strategy.

First, consider the situation where n = d and the matrix ∆ x is nonsingular, the estimated gradient by our method in Eq. 5 and be simplified as g ( x ) ∝ ∆ ⊤ x A -1 x ∆ f = ∆ -1 x ∆ f , which has a similar formulation to the gradient estimated by linear interpolation strategy [5].

Second, consider a more general situation where n < d , the matrix ∆ x can be singular, and its singular value decomposition (SVD) is denoted as ∆ x = U Λ V ⊤ . Then A x = ∆ x ∆ ⊤ x = U ΛΛ ⊤ U ⊤ , and the estimated gradient by our method in Eq. 5 can be transformed as:

Since U and V are orthonormal, we have ∆ x ( V Λ ⊤ (ΛΛ ⊤ ) -1 U ⊤ )∆ x = ∆ x , showing that V Λ ⊤ (ΛΛ ⊤ ) -1 U ⊤ is the pseudo-inverse of ∆ x . Consequently, our method can be regarded as a general version of the linear interpolation strategy.

Moreover, in the spacial case where ∆ x is orthonormal, ΛΛ ⊤ = E is an identity matrix, and Eq. 15 can be simplified as g ( x ) ∝ V Λ ⊤ U -1 ∆ f = ∆ ⊤ x ∆ f , consistent with the gradient estimation in [30].

4 Experiment

4.1 Protocols

To demonstrate the efficacy and efficiency of our method, we conduct experiments on both simulation benchmarks and real-world scenarios. First, we test on the CUTEst [24], one of the most popular

0

200

400

Iteration

0

200

400

600

800

1000

1200

Function Value F(x)

ZO-SGD

ZO-signSGD

ZO-AdaMM

(a) BROYDNBDLS(d=50)

0

200

400

Iteration

0

2500

5000

7500

10000

12500

15000

Function Value F(x)

ZO-SGD

ZO-signSGD

ZO-AdaMM

(b) COATING(d=134)

0

200

400

Iteration

0

1000

2000

3000

4000

5000

Function Value F(x)

ZO-SGD

ZO-signSGD

ZO-AdaMM

(c) SROSENBR(d=500)

0

200

400

Iteration

0

2000

4000

6000

8000

Function Value F(x)

ZO-SGD

ZO-signSGD

ZO-AdaMM

(d) BOXPOWER(d=1000)

benchmarks for derivative-free optimization containing over 1,500 problems, many of which are parametrized to allow for variable dimensions through user-selectable parameters. Specifically, we utilize the PyCUTEst [21], a Python interface, to import the optimization problems from the CUTEst and solve them by multiple ZO solvers implemented ourselves on the PyTorch platform. Second, we conduct experiments on the black-box adversarial attack task, a real-world application for ZO solvers. The task attempts to search additive noises for each pixel of inputs to confuse a pretrained neural network, which is one of the most popular scenarios for ZO solvers. Third, we apply our method to the Neural Architecture Search (NAS) task, aiming to search for the best neural architecture. Since querying once in NAS task consumes large amounts of resources, we conduct experiments on NAS-Bench-201 [18], which can directly query the performance of 15,625 architectures on three datasets including CIFAR10, CIFAR100, and ImageNet-16-120. Finally, we conduct experiment on large-scale neural network training by fine-tuning large language models (LLMs), demonstrating the applicability of our proposed ReLIZO.

4.2 Experiments on the Simulation Benchmark

We compare the performance of our method with baselines on four problems from the CUTEst optimization benchmark, containing an ill-conditioned or singular type problem (BOXPOWER), a nonlinear optimization test problem (SROSENBR), and a problem coming from application scenarios (COATING). The dimension of variables ranges from 50 to 1000 to evaluate the effectiveness of our method on both low-dimension and high-dimension problems. The baselines include ZO-SGD [22], ZO-signSGD [34] and ZO-AdaMM [12] implemented by ourselves on the PyTorch platform. Each solver updates the variables 500 times and samples 8 random directions at each iteration to update the variables. We also utilize grid search to obtain the best learning rate for each problem. The candidate learning rate η ranges from {0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5}. As for our method, the total sample size at each iteration is set as 8, and the reusable distance bound b is set as 2 η , where η is the learning rate obtained by the grid search. The variables are initialized as the default values in CUTEst for all solvers. Experiments at each setting are repeated three times, and the results are shown in Fig. 1. We observe that our method has a faster convergence speed and achieves better solutions to different problems. Moreover, the standard deviation of our method among three replications (shadows in Fig. 1) is much lower than other methods, showing that our method can stably find good solutions.

Ablation Study about sample size N and reusable distance bound b . To evaluate the impact of sample size at each iteration and reusable distance bound, we run our method for 500 iterations on different sample sizes ranging from {6, 8, 10, 20, 50, 100, 200} and reusable distance bounds ranging from {0, η , 2 η , 5 η }, where η is the learning rate at each iteration. Note that we set the sample size smaller than the dimension of variables. Results are shown in Fig. 2. The x-axis indicates the sample size, the y-axis shows the optimal value found by our solver after 500 iterations, and colors indicate different reusable distance bounds. For each setting, we conduct three replication experiments and plot the average performance and standard deviation as the nodes and lines. We also illustrate the reusing rates = # reused queries total sample size in Fig. 2 as the floats in the boxes beside the scatters. According to the

6

8

10

20

50

Number of Samples Per Step, N

0

2

4

6

8

10

12

14

F(x*)

0.00

0.39

0.65

0.76

0.00

0.61

0.74

0.84

0.00

0.72

0.76

0.79

0.00

0.72

0.83

0.91

0.00

0.84

0.89

0.96

b=0

b=

b=2

b=5

(a) BROYDNBDLS (d=50)

6

8

10

20

50

100

200

Number of Samples Per Step, N

0

2

4

6

8

10

12

14

16

F(x*)

0.00

0.12

0.26

0.42

0.00

0.28

0.38

0.48

0.00

0.42

0.47

0.60

0.00

0.68

0.72

0.78

0.00

0.86

0.87

0.90

0.00

0.93

0.94

0.94

0.00

0.96

0.97

0.97

b=0

b=

b=2

b=5

(b) SROSENBR (d=500)

results, we have three observations: 1) A larger reusable distance bound b has a larger reusable rate, and a larger sample size will also lead to a larger reusable rate. 2) As the sample size increases, large reusable distance bound can also find good solutions, demonstrating the rationality of reusing queries. 3) the standard deviation of performance is pretty small when N > 6 and b < 5 η , showing that it can stably find good solutions. Experiments and analysis on extreme conditions (e.g. small N and large d ) in Appendix C.1 also demonstrate the robustness and scalability of ReLIZO.

4.3 Experiments on Black-Box Adversarial Attacks

ZO methods are popular solutions to black-box adversarial attacks, and we conduct experiments to show the effectiveness of our ReLIZO. We focus on universal adversarial perturbation against images [27, 14] and attack a well-trained DNN model on the CIFAR-10 dataset. Supported by [8] and [12], we consider the optimization problem as: arg min δ λ M ∑ M i =1 f ( x i + denotes the black-box attack loss for imis a regularization hyperparameter bedistor-

̸

δ ) + ∥ δ ∥ 2 2 s.t. ( x i + δ ) ∈ [ -0 . 5 , 0 . 5] d , ∀ i, where f ( x i + δ ) age i , λ > 0 tween minimizing the attack loss and the ℓ 2 tion. We normalize the pixel values to [ -0 . 5 , 0 . 5] d , and we specify the loss function for untargeted attack as f ( x ′ ) = max j = t { Z ( x ′ ) t -max j = t Z ( x ′ ) j -κ } , where Z ( x ′ ) k denotes the prediction score of class k given the input x ′ , and κ > 0 governs the gap between

̸

the confidence of the predicted label and the true label t . In experiments, we set κ = 0 , and the attack loss f reaches the minimum value 0 as the perturbation succeeds to fool the network.

We conduct experiments on the CIFAR10 test dataset. Similar to the settings in ZO-AdaMM [12], we randomly select 100 images, ensuring that these images are initially correctly classified by the model. We conduct adversarial attacks on these selected images and compare the performance of our proposed ReLIZO method with 4 existing methods: ZO-SGD [22], ZO-signSGD [34], ZO-SMD [19], and ZOAdaMM [12]. We conduct a linear search for the learning rate of each approach to maintain consistency in the ℓ 2 distortion when employing each method for the at-

0

5000

10000

15000

20000

Iteration

4

6

8

10

12

14

16

18

Loss

ZO-AdaMM

ZO-SMD

ZO-SGD

ZO-signSGD

tacks, and compared the loss and the attack success rate on this basis. Fig. 3 illustrates the overall loss curves, while Table 1 presents the success rates of the adversarial attacks for each method over T = 20000 iterations. As we can see, our ReLIZO method exhibits the fastest rate of loss reduction compared to other ZO methods. Simultaneously, it achieves the highest attack success rate (more than 5% improvement) under condition where the ℓ 2 distortion remains relatively consistent. This result fully demonstrates the effectiveness of our method.

4.4 Experiments on Neural Architecture Search

AutoML, including neural architecture search [32, 40, 51, 50, 49, 46, 53] and hyper-parameter optimization tasks [7, 43, 26, 48], has attracted wide attention due to its efficacy in reducing labor costs. Zero-order optimization is an effective algorithm for solving AutoML tasks [47], which can be modeled as bi-level optimization.

Table 3: Number of queries and the search cost (seconds) during the NAS procedure on the search space of NAS-Bench201.

Table 2 reports the performance of our method on the NAS-Bench-201, a popular benchmark in the NAS task [32, 17, 51]. The first block shows the performance of gradient-based methods which is obtained from the paper [18]. The second block shows the performance of various ZO methods, including Zo-AdaMM, ZO-SGD, ZO-signSGD and ours. Specifically, similar to DARTS, we iteratively train the operation weights by the SGD optimizer and train the architecture parameters by the ZO optimizer for 50 epochs on the CIFAR10 dataset. To query the performance of one sampled direction, we update the architecture parameters by the direction and return the loss after training the operation weights for 10 iterations. The accuracy of discovered architectures on three datasets is directly indexed from the benchmark. For each ZO optimizer, we conduct three replication experiments and report the averaged accuracy and standard deviation in Table 2. We observe that our method surpasses other ZO methods in performance. We also report the number of queries of each ZO optimizer during the whole search process and the search time in Table 3. Our method is much faster than other ZO optimizers with fewer queries. Notice that the queries in Table 3 are used to train the architecture parameters, while the cost consists of training operation weights and architecture parameters, making the reduction of cost lower than the reduction of queries.

4.5 Experiments on Parameter Efficient Fine-tuning

Compared to gradient-based methods, ZO methods are well-suited for black-box optimization problems where gradients with respect to the variables are unavailable. Additionally, ZO methods are more memory-efficient than gradient-based methods since they do not require a backward pass,

making them applicable to network training, as demonstrated by recent works [10, 55]. Specifically, DeepZero [10] introduces a principled ZO framework for deep learning that is computational-graphfree and can scale to deep neural network training with performance comparable to first-order methods. The work [55] also applies ZO methods to large language model (LLM) fine-tuning, highlighting their memory efficiency and introducing a ZO-LLM benchmark.

To illustrate the applicability of ReLIZO in network training, we adopt it for fine-tuning an OPT-1.3b model (with 1.3 billion parameters) on the Stanford Sentiment Treebank v2 (SST2) task with a batch size 128, following the methodology of ZO-LLM [55]. The results, shown in Table 4 indicate that ReLIZO outperforms other ZO methods across various fine-tuning schemes, including full parameter fine-tuning (FT), LoRA, Prefix-tuning, and Prompt-tuning. Notably, ReLIZO even surpasses SGD in the FT scheme while requiring significantly less memory, demonstrating its promising potential.

5 Conclusion, Limitations and Future Work

This work handles gradient estimation in zeroth-order optimization by modeling it as a quadratically constrained linear program problem, with analytical solution derived via the Lagrange multiplier method. Our method ReLIZO, decouples the required sample size from the variable dimension without the requirement of orthogonal condition in the recent linear interpolation works, making it possible to leverage the queries in the prior iterations. We also show that part of the intermediate variables contributing to the gradient estimation can be directly indexed, significantly reducing the computation complexity. We further perform a theoretical analysis of the convergence of ReLIZO and show the relationship with the prior linear interpolation methods. The efficacy and efficiency have been shown in extensive experiments, including simulated benchmarks, NAS, and black-box adversarial attack tasks.

Limitations and Future work. Compared to the gradient-based methods, ZO methods require multiple queries of function evaluation at each iteration, which restricts their applications in realworld scenarios such as neural network training. Our ReLIZO can reduce the required number of queries by the reusing strategy, potentially broadening the scope of applications of ZO, which can be explored in future work. However, in cases where the consumption of one query is negligible, the profit due to the reusing technique decreases, causing a relative increase in the cost of solving the QCLP. We leave these problems in future work.

Impact Statements

This paper presents work whose goal is to advance the field of Machine Learning. Note that zerothorder optimization is an effective method for black-box adversarial attacks as we have shown in Section 4.3. Its purpose is to evaluate the robustness of machine learning models, but we should also be aware of the impact this may have on AI safety. There are many potential societal consequences of our work, yet none of which we think must be specifically highlighted here.

References

Alekh Agarwal, Ofer Dekel, and Lin Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In Colt , pages 28-40. Citeseer, 2010.

Larry Armijo. Minimization of functions having lipschitz continuous first partial derivatives. Pacific Journal of mathematics , 16(1):1-3, 1966.

Anastasia Bayandina, Alexander Gasnikov, Fariman Guliev, and Anastasia Lagunovskaya. Gradient-free two-points optimal method for non smooth stochastic convex optimization problem with additional small noise. arXiv preprint arXiv:1701.03821 , 2017.

Albert S Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. Linear interpolation gives better gradients than gaussian smoothing in derivative-free optimization. arXiv preprint arXiv:1905.13043 , 2019.

Albert S. Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Found. Comput. Math. , 22(2):507-560, 2022.

Albert S Berahas, Liyuan Cao, and Katya Scheinberg. Global convergence rate analysis of a generic line search algorithm with noise. SIAM Journal on Optimization , 31(2):1489-1518, 2021.

James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal Of Machine Learning Research , 2012.

Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) , pages 39-57. Ieee, 2017.

Richard G Carter. On the global convergence of trust region algorithms using inexact gradient information. SIAM Journal on Numerical Analysis , 28(1):251-265, 1991.

Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Konstantinos Parasyris, Jiancheng Liu, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. In ICLR , 2024.

Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In AISEC , pages 15-26, 2017.

Xiangyi Chen, Sijia Liu, Kaidi Xu, Xingguo Li, Xue Lin, Mingyi Hong, and David Cox. Zoadamm: Zeroth-order adaptive momentum method for black-box optimization. NeurIPS , 32, 2019.

Xin Chen, Yujie Tang, and Na Li. Improve single-point zeroth-order optimization using high-pass and low-pass filters. In ICML , volume 162, pages 3603-3620, 2022.

Minhao Cheng, Thong Le, Pin-Yu Chen, Jinfeng Yi, Huan Zhang, and Cho-Jui Hsieh. Query-efficient hard-label black-box attack: An optimization-based approach. arXiv preprint arXiv:1807.04457 , 2018.

Andrew R Conn, Katya Scheinberg, and Luís N Vicente. Geometry of interpolation sets in derivative free optimization. Mathematical programming , 111:141-172, 2008.

Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In ICCV , pages 3680-3689. IEEE, 2019.

Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four GPU hours. In CVPR , pages 1761-1770. Computer Vision Foundation / IEEE, 2019.

Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In ICLR . OpenReview.net, 2020.

John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory , 61(5):2788-2806, 2015.

Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. arXiv preprint cs/0408007 , 2004.

Jaroslav Fowkes, Lindon Roberts, and Árpád Bürmen. Pycutest: an open source python package of optimization test problems. J. Open Source Softw. , 7(78):4377, 2022.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Allen A Goldstein. On steepest descent. Journal of the Society for Industrial and Applied Mathematics, Series A: Control , 3(1):147-151, 1965.

Nicholas I. M. Gould, Dominique Orban, and Philippe L. Toint. Cutest: a constrained and unconstrained testing environment with safe threads for mathematical optimization. Comput. Optim. Appl. , 60(3):545-557, 2015.

Genetha Anne Gray, Tamara G. Kolda, Ken Sale, and Malin M. Young. Optimizing an empirical scoring function for transmembrane protein structure determination. INFORMS J. Comput. , 16(4):406-418, 2004.

Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In LION , 2011.

Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning , pages 2137-2146. PMLR, 2018.

Kevin G. Jamieson, Robert D. Nowak, and Benjamin Recht. Query complexity of derivative-free optimization. In NeurIPS , pages 2681-2689, 2012.

Kaiyi Ji, Zhe Wang, Yi Zhou, and Yingbin Liang. Improved zeroth-order variance reduced algorithms and analysis for nonconvex optimization. In ICML , pages 3100-3109. PMLR, 2019.

David Kozak, Cesare Molinari, Lorenzo Rosasco, Luis Tenorio, and Silvia Villa. Zeroth-order optimization with orthogonal random directions. Math. Program. , 199(1):1179-1219, 2023.

Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019 , volume 115 of Proceedings of Machine Learning Research , pages 367-377. AUAI Press, 2019.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Sijia Liu, Jie Chen, Pin-Yu Chen, and Alfred Hero. Zeroth-order online alternating direction method of multipliers: Convergence analysis and applications. In International Conference on Artificial Intelligence and Statistics , pages 288-297. PMLR, 2018.

Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong. signsgd via zeroth-order oracle. In International conference on learning representations . International Conference on Learning Representations, ICLR, 2019.

Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O. Hero III, and Pramod K. Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Process. Mag. , 37(5):43-54, 2020.

Alvaro Maggiar, Andreas Wachter, Irina S Dolinskaya, and Jeremy Staum. A derivative-free trust-region algorithm for the optimization of functions smoothed via gaussian convolution using adaptive multiple importance sampling. SIAM Journal on Optimization , 28(2):1478-1507, 2018.

Dhruv Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter L. Bartlett, and Martin J. Wainwright. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems. J. Mach. Learn. Res. , 21:21:1-21:51, 2020.

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. SIAM J. OPTIM , 1983.

Yurii Nesterov et al. Random gradient-free minimization of convex functions. Technical report, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE), 2011.

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In ICML , pages 4095-4104. PMLR, 2018.

Luis Miguel Rios and Nikolaos V. Sahinidis. Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. , 56(3):1247-1293, 2013.

Hao-Jun M Shi, Yuchen Xie, Richard Byrd, and Jorge Nocedal. A noise-tolerant quasi-newton algorithm for unconstrained optimization. SIAM Journal on Optimization , 32(1):29-55, 2022.

Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In NeurIPS , 2012.

Yun-Yun Tsai, Pin-Yu Chen, and Tsung-Yi Ho. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In International Conference on Machine Learning , pages 9614-9624. PMLR, 2020.

Anirudh Vemula, Wen Sun, and J. Andrew Bagnell. Contrasting exploration in parameter and action space: A zeroth-order optimization perspective. In AISTATS , volume 89 of Proceedings of Machine Learning Research , pages 2926-2935. PMLR, 2019.

Xiaoxing Wang, Xiangxiang Chu, Yuda Fan, Zhexi Zhang, Bo Zhang, Xiaokang Yang, and Junchi Yan. ROME: robustifying memory-efficient NAS via topology disentanglement and gradient accumulation. In ICCV , pages 5916-5926. IEEE, 2023.

Xiaoxing Wang, Wenxuan Guo, Jianlin Su, Xiaokang Yang, and Junchi Yan. Zarts: On zeroorder optimization for neural architecture search. Advances in Neural Information Processing Systems , 35:12868-12880, 2022.

Xiaoxing Wang, Jiaxing Li, Chao Xue, Wei Liu, Weifeng Liu, Xiaokang Yang, Junchi Yan, and Dacheng Tao. Poisson process for bayesian optimization. In AutoML , volume 224 of Proceedings of Machine Learning Research , pages 3/1-20, 2023.

Xiaoxing Wang, Zhirui Lian, Jiale Lin, Chao Xue, and Junchi Yan. DIY your easynas for vision: Convolution operation merging, map channel reducing, and search space to supernet conversion tooling. IEEE Trans. Pattern Anal. Mach. Intell. , 45(11):13974-13990, 2023.

Xiaoxing Wang, Jiale Lin, Juanping Zhao, Xiaokang Yang, and Junchi Yan. Eautodet: Efficient architecture search for object detection. In ECCV , volume 13680 of Lecture Notes in Computer Science , pages 668-684, 2022.

Xiaoxing Wang, Chao Xue, Junchi Yan, Xiaokang Yang, Yonggang Hu, and Kewei Sun. Mergenas: Merge operations into one for differentiable architecture search. In IJCAI , pages 3065-3072, 2020.

Yining Wang, Simon Du, Sivaraman Balakrishnan, and Aarti Singh. Stochastic zeroth-order optimization in high dimensions. In International conference on artificial intelligence and statistics , pages 1356-1365. PMLR, 2018.

Beichen Zhang, Xiaoxing Wang, Xiaohan Qin, and Junchi Yan. Boosting order-preserving and transferability for neural architecture search: a joint architecture refined search and finetuning approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5662-5671, 2024.

Yan Zhang, Yi Zhou, Kaiyi Ji, and Michael M. Zavlanos. A new one-point residual-feedback oracle for black-box learning and control. Autom. , 136:110006, 2022.

Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In ICML , 2024.

A Convergence Analysis

A.1 Proof of Lemma 3.1

Lemma 3.1 Let f be a function satisfying Assumption 3.1 and 3.2. For every k ∈ N , ∆ x k ∈ R n × d and ∥ ∆ i x k ∥ < b , we have:

Proof. We start with the Taylor's expansion of function f defined in Eq. 1

thus

With the condition ∥ ∆ i x k ∥ < b , we then derive that

which completes the proof.

A.2 Proof of Lemma 3.2

Lemma 3.2 Let the SVD of ∆ x k = U Λ V ⊤ , A -1 x k = (∆ x k ∆ ⊤ x k ) -1 , ρ k denotes the spectral radius of A -1 x k . Based on Lemma 3.1, we have:

where

Proof. Without losing generality, we omit the subscript k in our proof. g ( x ) = ∆ ⊤ x A -1 ∆ f and ∆ f = ∆ x g ( x ) = ∆ x [ g ( x ) -∇ f ( x )] + ∆ x ∇ f ( x ) , we define

then we have:

where I d × d denotes an identity matrix with d rows and columns. Suppose the SVD of ∆ x = U Λ V ⊤ , substitute to ∆ ⊤ x A -1 ∆ x , we have:

Then, we can simplify ∥ g ( x ) -∇ f ( x ) ∥ as:

where ⟨ x, y ⟩ denotes the inner product of vectors x and y . According to the definition of inner product and the SVD of ∆ x = U Λ V T , we have:

Moreover, we denote v = V ⊤ ∇ f ( x ) , then ∥ V ˜ I d -n V ⊤ ∇ f ( x ) ∥ 2 = ∑ d j = n v 2 j , we define r as

Therefore,

where ρ is the spectral radius of A -1 . According to Lemma 3.1, we have:

A.3 Proof of Proposition 3.3

Proposition 3.3 Suppose f is a convex function and ∃ θ ∈ [0 , 1) satisfying ρ k ≤ 4( θ 2 -r ) ∥∇ f ( x ) ∥ 2 L 2 b 2 n , we have:

Proof. Based on Lemma 3.2, we substitute Eq. 13 with ρ k ≤ 4( θ 2 -r ) ∥∇ f ( x ) ∥ 2 L 2 b 2 n , then we have

This condition is also referred to as norm condition and is introduced and studied in [9] in the context of trust-region methods with inaccurate gradients. This then yelids

by using Cauchy-Schwarz inequality. Note that our iteration also satisfies the Armijo condition

where c 1 ∈ (0 , 1) is a constant, and σ k is the magnitude of descent direction found by using Armijo backtracking line search. By (16), we have

Assumption A.1. (Convexity and bounded level sets of f ) The function f is convex and has bounded level sets, i.e.,

where x ∗ is a global minimizer of f . Let f ∗ = f ( x ∗ ) .

By Assumption A.1, we have

Let z k = f ( x k ) -f ∗ , thus

which implies

thus

Moreover, in line with the assumption in [30], we can set σ as the minimum step size in the algorithm. In fact, the previous works [6] and [42] have shown that in the convex case, the number of failed line searches can be bounded w.r.t σ . Thus we mainly focus on the case when (17) holds. In this case, define C = D 2 c 1 σ , then (18) implies

which shows a sublinear convergence rate. Similar to the prior ZO works (Table 1 in [29]), we set the minimum step size σ ∼ O ( 1 d ) to guarantee the correctness of the assumptions, thus the convergence rate of our method is:

B Comparison of Computation Complexity

The algorithm flow is shown in Alg. 1. Suppose n is the sample size, d is the dimension of variables, and n < d . For the k -th iteration, suppose n k vectors are reused from the prior iterations and n -n k vectors are newly sampled. The computation complexity includes two parts: querying function evaluations for newly sampled vectors, and computing gradient estimation by Eq. 5. For the former, we denote the complexity of querying function evaluation once as O ( c ( d )) , where c ( · ) could be linear, polynomial, etc.. Then querying n -n k times requires O (( n -n k ) c ( d )) computation resources. As for the latter, Eq. 5 requires to compute A x k with complexity O (( n -n k ) nd ) , the inverse of matrix A x k with complexity O ( n 3 ) , and the matrix multiplication of ∆ ⊤ x k A -1 x k ∆ f with complexity O ( n 2 + nd ) . Since n < d , the overall computation complexity for the k -th iteration is O ( n 3 +( n -n k ) nd +( n -n k ) c ( d ) ) . Table 5 compares the computation complexity in one iteration of our method with prior smoothing and linear interpolation methods. Except for our methods, others do not support the reusing strategy.

C Supplementary Experiments

C.1 Experiments on Extreme Conditions

Experiments on small sample size N : We set N = 2 and compare ReLIZO and other methods on six PyCUTEst optimization tasks. The results are reported in Table 6, showing that ReLIZO also surpasses other baselines. We analyze that the methods based on smoothing strategy utilize Monte Carlo sampling to estimate the expectation of various directional derivatives and a small N will lead to a biased estimation as shown in [5]. In contrast, our ReLIZO computes the projection of gradient on a space generated by arbitrary numbers of sampled directions by solving the QCLP, and thus can also work pretty well under a small N .

Experiments on large dimension d : We conduct experiments on two PyCUTEst optimization tasks with an adjustable variable dimension. Each optimizer updates the variables 500 times and samples 8 random directions at each iteration to update the variables. We also utilize grid search to obtain the best learning rate for each optimizer for different tasks. The best solution obtained by each optimizer are reported in Table 7, and the values in the brackets report the gap between each baseline and ReLIZO. The results show that the improvement of ReLIZO is increasing as d increases for each optimization task. We analyze that the sample size N becomes relatively small compared to the dimension d as d increases, our method computes the projection of gradient on a space generated by N sampled directions by solving the QCLP, and thus can also work pretty well even if N << d .

C.2 Discussion on the Line Search

We adopt line search to adaptively adjust the stepsize to ensure a fast convergence rate. Our experimental results show that there are few queries from line search during the whole optimization process in most cases. Specifically, there are 0 queries from line search in NAS task and several optimization tasks in PyCUTEst, including CHNROSNB, SROSENBR, and BOXPOWER. We report the number of queries in each task in Table 8, where '#LS' denotes the number of queries from line search, '#Queries' denotes the number of actual queries including #LS during the optimization procedure, '#Reused' denotes the number of reused queries, 'N x T' denotes the total queries required by other methods without reusing strategy ( N is the sample size per iteration and T is the number of iterations).

C.3 Ablation Studies on Other Problems

Results of ablation studies on another two problems MANCINO and ARGTRIGLS are shown in Fig. 4.

6

8

10

20

50

100

Number of Samples Per Step, N

0

10

20

30

40

F(x*)

0.00

0.00

0.00

0.01

0.00

0.11

0.22

0.24

0.00

0.27

0.35

0.41

0.00

0.58

0.65

0.69

0.00

0.82

0.87

0.88

0.00

0.63

0.81

0.79

b=0

b=

b=2

b=5

(a) MANCINO (d=100)

6

8

10

20

50

100

200

Number of Samples Per Step, N

0

10

20

30

40

50

F(x*)

0.00

0.87

0.94

0.98

0.00

0.92

0.96

0.98

0.00

0.93

0.97

0.99

0.00

0.95

0.96

1.00

0.00

0.96

0.98

0.99

0.00

0.96

0.98

0.98

0.00

0.98

0.99

1.00

b=0

b=

b=2

b=5

(b) ARGTRIGLS (d=200)

We also report the optimization procedure of our method at different settings of reusable distance bound b in Fig. 5 and sample size n in Fig. 6.

C.4 Improved Sampling Strategy with Momentum

The ability of ReLIZO to sample new vectors from an arbitrary distribution can be considered one of its strengths compared to other ZO methods. In contrast, smoothing-based ZO methods have to rely on Gaussian or uniform distributions with a mean of zero. We introduce an effective sampling strategy inspired by SGD with momentum. Specifically, we propose a sampling momentum strategy where the sampling probability in the current iteration follows a distribution defined as

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

b=0

b=

b=2

b=5

b=10

(a) BROYDNBDLS (d=50)

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

b=0

b=

b=2

b=5

b=10

(b) MANCINO (d=100)

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

b=0

b=

b=2

b=5

b=10

(c) COATING (d=134)

0

100

200

300

400

500

Iter

0

10

20

30

40

50

60

Function Object

b=0

b=

b=2

b=5

b=10

(d) ARGTRIGLS (d=200)

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

b=0

b=

b=2

b=5

b=10

(e) SROSENBR (d=500)

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

b=0

b=

b=2

b=5

b=10

(f) BOXPOWER (d=1000)

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

N=6

N=8

N=10

N=50

(a) BROYDNBDLS (d=50)

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

N=6

N=8

N=10

N=50

N=100

(b) MANCINO (d=100)

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

N=6

N=8

N=10

N=50

N=100

(c) COATING (d=134)

0

100

200

300

400

500

Iter

0

10

20

30

40

50

60

Function Object

N=6

N=8

N=10

N=50

N=100

N=200

(d) ARGTRIGLS (d=200)

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

N=6

N=8

N=10

N=50

N=100

N=200

(e) SROSENBR (d=500)

0

100

200

300

400

500

Iter

0

100

200

300

400

500

600

700

800

900

Function Object

N=6

N=8

N=10

N=50

N=100

N=200

(f) BOXPOWER (d=1000)

p ( x t ) = αN ( g t -1 , σ ) + (1 -α ) N (0 , σ ) , with N () denoting a Gaussian distribution, g t -1 is the estimated gradient in the last iteration, α as the momentum parameter, and σ as the standard deviation. The results of this approach, labeled as ReLIZO-m, are presented in Table 9.

C.5 Additional Experimental Results on the Reusable Distance Bound b

In Fig. 5, we observe a relative performance drop (slightly slower convergence) when the reusable distance bound b ≥ η . However, this does not imply that the reuse strategy is ineffective in this case. The ARGTRIGLS problem is characterized by a rugged landscape with numerous local minima. In this case, as shown in Fig. 4(b), even with b = η and a sample size n = 8 , the reuse rate of ARGTRIGLS exceeds 90%, whereas reuse rates for other problems are generally below 50% (e.g. MANCINO with d=100 in Fig. 4(a), SROSENBR with d=500 in Fig. 2(b)). This high reuse rate indicates that the number of new queries during optimization is relatively small, contributing to the observed decrease in convergence speed. To address this issue, we can employ simple strategies such as reducing the reusable distance bound b . Table 10 illustrates that performance improves as b decreases.

Moreover, setting an upper bound for the reuse rate in each iteration can also help. Table 11 reports the performance when the maximum reuse rate in each iteration is restricted to 50%. We observe that with b ≤ 0 . 05 η and 2000 iterations, there is minimal performance degradation, even with a reuse rate of approximately 30%.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims are clearly presented in the abstract and introduction, and are further elaborated and substantiated throughout the subsequent sections of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Refer to Sec.5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In our paper, we provide detailed descriptions of each step of the algorithm, the pseudocode, and all settings required to replicate the experiments. Additionally, we commit to making the code publicly available by the time of the camera-ready submission.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Open source upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Refer to Sec.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Refer to Sec.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Refer to Sec.4. and Appendix C.1 (e.g. the training epochs and the clock time of experiments are provided.)

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Refer to Sec.5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators and original owners of all assets used in the paper are properly credited, and the licenses and terms of use are explicitly mentioned and fully respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
From my understanding, this paper give a zero-th order algorithm with application to popular vision tasks neural architecture search and black-box adversarial attacks. The authors derive a closed-form solution after modeling the gradient estimation as a quadratically constrained linear program problem. The key idea is to try to decouple the required sample size from the variable dimension without extra conditions required, making it able to leverage the queries in the prior iterations. The speedup is further technically achieved by directly indexing some of the intermediate variables that contribute to the gradient estimation. The theoretical studies are given for its convergence speed and its cost-effectiveness is verified on benchmarks.

STRENGTHS:
1. Clear motivation with clearly derived approach, and it is a new zero-th order algorithm indeed and the authors also contextualize well the proposed method with related work discussion.
2. Strong empirical performance on representative vision tasks with rich testbeds and settings.
3.  The approach by its design, could enjoy the efficiency of smoothing techniques while maintaining estimation accuracy. Table 4 in the appendix is informative.
4.  The paper gives comprehensive results and technical details in both main paper and appendix.

WEAKNESSES:
1. As remarked by the authors, it has few constraints on the sample size, similar to the smoothing techniques; and it requires the estimation of the gradients which involves solving a linear program problem.
2. As a zero-th order algorithm, it may still not be suited for large-scale application e.g. network training.

QUESTIONS:
I wonder if the proposed method could really facilitate the community of NAS? as zero-order optimization is not common in NAS.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ReLIZO, a novel zeroth-order optimization method leveraging linear interpolation to estimate gradients efficiently. It reduces the complexity of gradient estimation by reusing prior queries without additional conditions on sample size, decoupling it from variable dimension constraints. ReLIZO models gradient estimation as a quadratically constrained linear program, solving it analytically to reduce computation complexity. Experimental results demonstrate ReLIZO's efficacy in various scenarios, including black-box adversarial attacks and neural architecture search, showcasing faster convergence and better solutions compared to existing methods.\n\nSTRENGTHS:\n* The paper is well-written, with clear and easy-to-follow explanations.\n* The paper introduces a method for estimating gradients using arbitrarily sampled vectors without requiring orthogonal conditions or adherence to a specific distribution, enabling the reuse of queries to accelerate the zeroth-order (ZO) optimization process.\n* Extensive experiments on simulation benchmarks and real-world applications validate the method’s performance.\n* The paper highlights that ReLIZO can be viewed as a generalized version of traditional linear interpolation methods, capable of handling both equal and smaller sample sizes compared to variable dimensions. This demonstrates ReLIZO's theoretical soundness and enhanced flexibility in gradient estimation.\n\nWEAKNESSES:\n* The effectiveness of reusing queries depends on the choice of the reusable distance bound, which might require fine-tuning for different applications, adding complexity to its implementation.\n* While the method reduces the number of function queries, the process of solving the quadratically constrained linear program might introduce additional computational overhead for large $n$.\n\nQUESTIONS:\nIn Figure 5, the results for the ARGTRIGLS problem indicate that any reusable distance bound leads to a performance drop. What is the specific structure of this problem? Does this suggest that the reuse strategy may be ineffective in certain special cases?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
This study introduces a novel gradient estimation algorithm that operates solely on forward function evaluations. The method employs a Quadratically Constrained Linear Program (QCLP) to determine the optimal linear approximation of sample vectors. The authors present performance enhancement strategies, including sample reuse and efficient inverse matrix computation within the QCLP framework. Empirical evaluations conducted on black-box adversarial attacks and neural architecture search demonstrate the proposed algorithm's superiority over existing zeroth-order methods.

STRENGTHS:
1. The proposed method is natural. Approximating the gradient using linear combinations of samples and formulating as the QCLP is an intuitive idea, and the auxiliary techniques employed in this study are both judicious and pertinent to the research objectives.

2. The paper is well-written and easy to follow.

WEAKNESSES:
1. Zeroth-order gradient estimation has a relatively limited impact. While the proposed zeroth-order gradient estimation method demonstrates superiority over existing algorithms in its class, its overall impact on solving underlying optimization problems may be constrained. This limitation is exemplified in the NAS evaluation, where ReLIZO does not consistently achieve optimal performance.

2. According to my interpretation, in ReLIZO algorithm, obtaining new samples from the input space in each iteration is random and arbitrary. I feel there might be more effective strategies to sample new vectors based on known information. Could the authors comment on this?

QUESTIONS:
See weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13