SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section we profile the memory usage of zero-shot, ICL, FT, FT (prefix), and MeZO. We test OPT models of various sizes with Nvidia A100 GPUs (80GB memory) on MultiRC (average  $\# \mathrm{tokens} = 400$ ), and report the peak GPU memory consumption (details in Appendix E.7).
As shown in Figure 3 (refer to Appendix F.5 for detailed numbers), MeZO exhibits the same memory consumption as zero-shot while offering memory savings of up to 12 times compared to standard FT and 6 times compared to FT (prefix). This advantage enables training larger models within a fixed hardware budget, as illustrated in Figure 4. Specifically, using a single A100 GPU, MeZO allows for tuning a model that is 11 times larger than what is feasible with FT.
In Appendix F.6, we compare the wall-clock time efficiencies of our implementations of MeZO and Adam fine-tuning. MeZO achieves  $7.74 \times$  per-step speedup and requires  $8 \times$  fewer GPUs with a 30B model, but takes more steps to converge. Overall, MeZO only requires half as many GPU-hours to fine-tune a 30B model compared to full-parameter fine-tuning. The wall-clock benefit of MeZO is not inherent to the algorithm and is highly dependent on the implementation. We primarily provide this information as a demonstration that MeZO does not take a prohibitively long time to run.
The above measurements are dependent on the computing infrastructure. In Appendix C, we compare the theoretical time-memory tradeoff of MeZO and backpropagation and find that MeZO is always more memory-efficient than backpropagation and is often more time-efficient. The above analyses also do not consider recent advances (e.g., gradient checkpointing [18], FlashAttention [23], and quantization [27]). We leave investigating the how MeZO works with these methods to future work.
Our theoretical analysis highlights why MeZO can optimize large LMs, although a number of classical results [69, 47, 79, 3, 70] suggest that optimization should be catastrophically slow when training so many parameters. The inclusion of a simple prompt is crucial for MeZO to succeed (Appendix A). Past works [83, 67] have suggested that including such a prompt ensures that the fine-tuning objective is closely related to the pre-training one. As such, here, we make the assumption that the model has already been trained for many steps on the fine-tuning objective, which implies that the loss landscape exhibits favorable conditions (Assumption 1). Then, we derive a convergence rate independent of the number of parameters. We show that the loss decreases per step at a rate independent of the parameter dimension  $d$  (Theorem 1), and that, under stronger conditions, the algorithm converges in time independent of  $d$  (Lemma 3). Together, these results imply that MeZO is not catastrophically slower than SGD when fine-tuning.<sup>6</sup> For ease of illustration, we assume that  $z$  is sampled from a sphere with radius  $\sqrt{d}$ , and in Appendix G.2, we derive the rate for a general Gaussian  $z$ , which was used in the experiments.
We follow classical analyses of SGD and replace the minibatch gradient estimate with SPSA (Definition 1). Consider the minibatch SGD update  $\theta_{t + 1}\gets \theta_t - \eta \nabla \mathcal{L}(\pmb {\theta};\mathcal{B}_t)$  where  $\mathcal{B}_t$  is a minibatch drawn uniformly from  $\mathcal{D}^B$ . Crucially, the SGD minibatch gradient estimate is unbiased.
Definition 3 (Unbiased Gradient Estimate). Any minibatch gradient estimate  $g(\pmb{\theta}, \mathcal{B})$  is said to be unbiased if  $\mathbb{E}[g(\pmb{\theta}, \mathcal{B})] = \nabla \mathcal{L}(\pmb{\theta})$ .
4.1 Per-step analysis
The classical descent lemma uses a Taylor expansion to study how SGD reduces the loss at each optimization step. It highlights that when the gradient covariance is large, the maximum possible decrease in loss at each optimization step is small, thereby resulting in slower optimization.
Lemma 1 (Descent Lemma). Let  $\mathcal{L}(\pmb{\theta})$  be  $\ell$ -smooth. For any unbiased gradient estimate  $g(\pmb{\theta},\mathcal{B})$ ,
$$
\mathbb {E} \left[ \mathcal {L} \left(\boldsymbol {\theta} _ {t + 1}\right) \mid \boldsymbol {\theta} _ {t} \right] - \mathcal {L} \left(\boldsymbol {\theta} _ {t}\right) \leq - \eta \| \nabla \mathcal {L} \left(\boldsymbol {\theta} _ {t}\right) \| ^ {2} + \frac {1}{2} \eta^ {2} \ell \cdot \mathbb {E} \left[ \| \boldsymbol {g} \left(\boldsymbol {\theta}, \mathcal {B} _ {t}\right) \| ^ {2} \right]. \tag {2}
$$
The descent lemma highlights the importance of the gradient norm, which we derive for MeZO below.
Lemma 2. Let  $\mathcal{B}$  be a random minibatch of size  $B$ . Then, the gradient norm of MeZO is
$$
\mathbb {E} _ {x} \left[ \left\| \widehat {\nabla} \mathcal {L} (\boldsymbol {\theta}; \mathcal {B}) \right\| ^ {2} \right] = \frac {d + n - 1}{n} \mathbb {E} \left[ \| \nabla \mathcal {L} (\boldsymbol {\theta}; \mathcal {B}) \| ^ {2} \right].
$$
where  $n$  is the number of  $z$  sampled in  $n$ -SPSA (Definition 1) and  $d$  is the number of parameters.
Thus, in the usual case where  $n \ll d$ , MeZO has a much larger gradient norm than SGD. The descent lemma also shows that to guarantee loss decrease, one needs to choose the learning rate as
$$
\eta \leq \frac {2 \| \nabla \mathcal {L} (\boldsymbol {\theta} _ {t}) \| ^ {2}}{\ell \cdot \mathbb {E} [ \| \boldsymbol {g} (\boldsymbol {\theta} , \mathcal {B}) \| ^ {2} ]} \quad \stackrel {\text {L e m m a 2}} {\Longrightarrow} \quad \eta_ {\mathrm {Z O}} = \frac {n}{d + n - 1} \eta_ {\mathrm {S G D}} \tag {3}
$$
where  $\eta_{\mathrm{ZO}}$  and  $\eta_{\mathrm{SGD}}$  are the maximum permissible learning rates for MeZO and SGD respectively. Thus we see that without any further assumptions, MeZO can slow optimization by decreasing the largest permissible learning rate by a factor of  $d$ . Moreover, MeZO reduces the loss decrease that can be obtained at each step and, as a consequence, slows convergence by a factor of  $d$  as well.
Surprisingly, our experiments show that MeZO can quickly optimize pre-trained models with billions of parameters, and reducing the number of tuned parameters via PEFT techniques does not substantially accelerate optimization (Appendix F.3). We attribute these phenomena to the Hessian of the loss exhibiting small local effective rank. It is prohibitively expensive to directly measure the effective rank of the Hessian of a large LM on a reasonably sized dataset. However, many previous works have shown that the Hessian of the loss for deep neural networks trained by SGD has remarkably low
<sup>6</sup>Section 3 uses the standard choice of Adam for FT; we provide SGD experiments in Appendix F.1.
This is satisfied for the standard cross-entropy objective.
All of our experiments use  $n = 1$ .
effective rank [74, 75, 36, 107, 105, 82]. In particular, the bulk of the spectrum concentrates around 0 with only a small number of outliers, and the number of these outliers is an upper bound on the effective rank. In addition, prior works [4, 56] have demonstrated that LM fine-tuning can occur in a very low dimensional subspace ( $< 200$  parameters), which further supports the below assumption. We formalize the assumption on the effective rank below. In particular, we require an upper bound on the Hessian in a neighborhood around the current iterate to have effective rank at most  $r$ .
Assumption 1 (Local  $r$ -effective rank). Let  $G(\pmb{\theta}_t) = \max_{(\pmb{x},\pmb{y})\in \mathcal{D}}\|\nabla\mathcal{L}(\pmb{\theta}_t;\{(\pmb{x},\pmb{y})\})\|$ . There exists a matrix  $\pmb{H}(\pmb{\theta}_t)\preceq \ell \cdot \pmb{I}_d$  such that:
1. For all  $\pmb{\theta}$  such that  $\| \pmb {\theta} - \pmb {\theta}_t\| \leq \eta dG(\pmb {\theta}_t)$ , we have  $\nabla^2\mathcal{L}(\pmb {\theta})\preceq \pmb {H}(\pmb {\theta}_t)$ .
2. The effective rank of  $\mathbf{H}(\pmb{\theta}_t)$ , i.e.  $\mathrm{tr}(\pmb{H}(\pmb{\theta}_t)) / \| \pmb{H}(\pmb{\theta}_t) \|_{op}$ , is at most  $r$ .
Under this assumption, we show that the convergence rate of ZO-SGD does not depend on the number of parameters. Instead, the slowdown factor only depends on the effective rank of the Hessian.
Theorem 1 (Dimension-Free Rate). Assume the loss exhibits local  $r$ -effective rank (Assumption 1). If  $\pmb{\theta}_{t+1} = \pmb{\theta}_t - \eta_{ZO}\widehat{\nabla}\mathcal{L}(\pmb{\theta}_t;\mathcal{B})$  is a single step of ZO-SGD using the  $n$ -SPSA estimate with a minibatch of size  $B$ , then there exists a  $\gamma = \Theta(r/n)$  such that the expected loss decrease can be bounded as
$$
\mathbb {E} \left[ \mathcal {L} \left(\boldsymbol {\theta} _ {t + 1}\right) \mid \boldsymbol {\theta} _ {t} \right] - \mathcal {L} \left(\boldsymbol {\theta} _ {t}\right) \leq - \eta_ {Z O} \| \nabla \mathcal {L} \left(\boldsymbol {\theta} _ {t}\right) \| ^ {2} + \frac {1}{2} \eta_ {Z O} ^ {2} \ell \cdot \gamma \cdot \mathbb {E} \left[ \| \nabla \mathcal {L} (\boldsymbol {\theta}; \mathcal {B}) \| ^ {2} \right] \tag {4}
$$
By applying Equation (3), we can directly compare to the SGD descent lemma.
Corollary 1. Choosing the learning rate  $\eta_{ZO} = \gamma^{-1} \cdot \eta_{SGD}$ ,  $ZO$ -SGD obtains a loss decrease of
$$
\mathbb {E} \left[ \mathcal {L} \left(\boldsymbol {\theta} _ {t + 1}\right) \mid \boldsymbol {\theta} _ {t} \right] - \mathcal {L} \left(\boldsymbol {\theta} _ {t}\right) \leq \frac {1}{\gamma} \cdot \left[ - \eta_ {S G D} \| \nabla \mathcal {L} \left(\boldsymbol {\theta} _ {t}\right) \| ^ {2} + \frac {1}{2} \eta_ {S G D} ^ {2} \ell \cdot \mathbb {E} [ \| \nabla \mathcal {L} (\boldsymbol {\theta}; \mathcal {B}) \| ^ {2} ] \right]. \tag {5}
$$
Here we see that comparing to SGD, the slowdown factor of ZO-SGD scales with the local effective rank  $r$ , which we argue is much smaller than the number of parameters  $d$ . The above analysis focuses on how much ZO-SGD and SGD decrease the loss at each step. Below, we show that under stronger assumptions about the loss landscape, we can obtain rates for how quickly the ZO-SGD algorithm converges to an optimal value.
4.2 Global convergence analysis
We show that the global convergence rate also slows by a factor proportional to the local effective rank under stronger assumptions about the loss landscape. We assume that the landscape obeys the classical PL inequality: the gradient norm grows quadratically with the suboptimality of the iterate.
Definition 4 (PL Inequality). Let  $\mathcal{L}^* = \min_{\theta} \mathcal{L}(\theta)$ . The loss  $\mathcal{L}$  is  $\mu$ -PL if, for all  $\theta$ ,  $\frac{1}{2} \| \nabla \mathcal{L}(\theta) \|^2 \geq \mu (\mathcal{L}(\theta) - \mathcal{L}^*)$ .
The PL inequality is not as strong as assuming that optimization exhibits kernel-like dynamics, but it ensures that the landscape is amenable to analysis [50]. In addition to the PL inequality, we assume the trace of the gradient covariance is bounded, so noise does not disrupt the trajectory too drastically.
Definition 5 (Gradient Covariance). The SGD gradient estimate on a minibatch of size  $B$  has covariance  $\boldsymbol{\Sigma}(\boldsymbol{\theta}) = B(\mathbb{E}\left[\nabla \mathcal{L}(\boldsymbol{\theta};\mathcal{B})\nabla \mathcal{L}(\boldsymbol{\theta};\mathcal{B})^\top\right] - \nabla \mathcal{L}(\boldsymbol{\theta})\nabla \mathcal{L}(\boldsymbol{\theta})^\top)$ .
As we show in Appendix G.1, this assumption holds for common loss functions such as square loss or binary cross entropy for several settings (e.g., kernel behavior [67]). With these two assumptions, we show that ZO-SGD has a slowdown proportional to the effective rank  $r$ , not the parameter dimension.
Lemma 3 (Global Convergence of ZO-SGD). Let  $\mathcal{L}(\pmb{\theta})$  be  $\mu$ -PL and let there exist  $\alpha$  such that  $\operatorname{tr}(\pmb{\Sigma}(\pmb{\theta})) \leq \alpha (\mathcal{L}(\pmb{\theta}) - \mathcal{L}^{*})$  for all  $\pmb{\theta}$ . Then after
$$
t = \mathcal {O} \left(\left(\frac {r}{n} + 1\right) \cdot \underbrace {\left(\frac {\ell}{\mu} + \frac {\ell \alpha}{\mu^ {2} B}\right) \log \frac {\mathcal {L} \left(\boldsymbol {\theta} _ {0}\right) - \mathcal {L} ^ {*}}{\epsilon}} _ {S G D r a t e (L e m m a 4)}\right)
$$
iterations of ZO-SGD we have  $\mathbb{E}[\mathcal{L}(\pmb{\theta}_t)] \leq \mathcal{L}^* + \epsilon$ .
Zeroth-order optimization Many classical lower bounds have been derived for ZO-SGD in the strongly convex and convex settings [47, 3, 79, 32, 85, 69] as well as non-convex [101]. These bounds generally depend on the number of parameters  $d$ . More recently, [100, 6, 15] showed that if the gradient has low-dimensional structure, then the query complexity scales linearly with the intrinsic dimension and logarithmically with the number of parameters, though the estimation has at least  $\Omega (sd\log d)$  memory cost. Additional tricks such as sampling schedules [11] and other variance reduction methods [48, 62] can be added to ZO-SGD. ZO has inspired distributed methods [93, 43] and black-box adversarial example generation [14, 63, 17, 64] in deep learning. Ye et al. [108], Balasubramanian and Ghadimi [7] estimate the Hessian to perform ZO optimization along important directions. There are also ZO methods that optimize without estimating the gradient [38, 68, 44].
Memory-efficient backpropagation Several algorithms have been proposed to efficiently approximate backpropagation by sparsifying gradients [92, 102], approximating Jacobians [1, 19], and subsampling the computational graph [71, 2]. However, these methods may accrue large approximation errors for deep networks. Gradient checkpointing [18] reduces memory cost of backpropagation at the cost of recomputing some activations. FlashAttention [23] also reduces memory cost by recomputing attention matrices. Dettmers et al. [26, 27] explore quantization of large LMs' weights and optimizer states, which leads to memory reduction in both training and inference.
Gradient-free adaptation of large language models BBT and BBTv2 [91, 90] use evolutionary algorithms to achieve gradient-free optimization; however, due to its sensitivity to high dimensionality, BBT is limited to only optimize a low-dimension projection of prefixes and they focus on RoBERTa-large size models and few-shot settings. Other works in "black-box tuning" of LMs focus on optimizing discrete prompts without updating the model, either via reinforcement learning [16, 25, 29], ensemble [45], or iterative search [78]. Concurrent work in [106] uses iterative forward passes to improve in-context learning performance.
We have shown that MeZO can effectively optimize large LMs across many tasks and scales. Further experiments suggest that MeZO can optimize non-differentiable objectives, which backpropagation usually cannot do. Our theory illustrates why MeZO is not catastrophically slow when tuning billions of parameters. As a limitation, MeZO takes many steps in order to achieve strong performance, though we show that the per-step speedup in MeZO can often make fine-tuning with MeZO run faster than a standard implementation of fine-tuning with backpropagation. We did not explore combining MeZO with other memory-efficient methods, such as FlashAttention [23] and quantization [26], though we hope to investigate this in the future.
We are excited to explore the applicability of MeZO to a number of areas, including but not limited to: pruning, distillation, saliency, interpretability, and dataset selection for fine-tuning. Non-differentiable objectives are a particularly exciting area, given recent advances in tuning large LMs to adapt to human feedback. Conducting theoretical analyses for how these efficient gradient estimates impact the performance of different applications is also of interest.
We thank Xinyi Chen, Yin Tat Lee, Kaifeng Lyu, Tengyu Ma, Abhishek Panigrahi, Nikunj Saunshi, and Mengzhou Xia for their helpful feedback. SA and SM are funded by NSR, ONR, SRC, and Simons Foundation. JDL, AD, and EN acknowledge the support of the ARO under MURI Award W911NF-11-1-0304, the Sloan Research Fellowship, NSF CCF 2002272, NSF IIS 2107304, NSF CIF 2212262, ONR Young Investigator Award, and NSF CAREER Award 2144994. TG is supported by an IBM PhD Fellowship. This work is also partially funded by the National Science Foundation (IIS-2211779).
A Algorithmic Ablations
We perform a number of ablations to select the best algorithm. As is standard in ZO literature, we consider the main computational cost to be the number of forward passes. In our case, the number of forward passes can be affected by the number of gradient steps taken, any usage of gradient accumulation, and using more noise samples to reduce the variance of the gradient estimate.
We observed that the performance of MeZO improves monotonically with the number of steps, and there does not appear to be any overfitting. Hence, when performing algorithmic ablations, we can focus on the efficiency of different algorithms without considering implicit bias. This is also reflected in our theoretical analysis. To ease the computational load, we fix the number of forward passes to 10,000 and compare many different algorithms for RoBERTa-large on a smaller set of tasks that span sentiment analysis, entailment, and topic classification: SST-2, SNLI, and TREC. We emphasize that 10,000 is a small budget and is only used as a setting to compare these ZO algorithms to each other. We find that using a linearly decreasing learning rate schedule during training, as was done for fine-tuning with backpropagation in [65], does not help or hurt MeZO. Similarly, using a learning rate warmup leads to identical results on these three tasks. For simplicity, we use a constant learning rate schedule with no warmup for all of the below experiments. We perform few-shot experiments with  $k = 16$  and average the results across 5 seeds.
Experiment	Hyperparameters	Values
MeZO	Batch size	{16,64} ×
Learning rate	{1e-5,1e-6,1e-7} ×
ε	{1e-3,1e-5} ×
Weight Decay	{0,0.1}
A.1 Prompting
We study if adding a prompt is crucial to the ability of MeZO to optimize the network. We use prompts from Gao et al. [35]. Malladi et al. [67] claimed the prompt makes the optimization trajectory well-behaved, though we note that the current paper considers RoBERTa-large and large autoregressive models while the previous work only studied RoBERTa-base. We note the similarity between kernel behavior and our theoretical setting in Section 4. MeZO succeeds on tasks that are reported to not exhibit kernel behavior in Malladi et al. [67], so we investigate whether or not the prompt is necessary.
Table 4: The hyperparameter grid used in our ablation experiments. For simplicity, we use a constant learning rate schedule.
SST-2	SNLI	TREC
Prompt	89.6 (1.2)	65.1 (6.2)	66.7 (6.2)
No Prompt	51.9 (2.9)	34.8 (2.1)	19.5 (9.0)
Table 5: Experiments using MeZO to fine-tune models with and without a prompt.
Both experiments followed the grid in Table 4, but we also expanded the grid to include a learning rate of  $1\mathrm{e} - 4$  for the no prompt case. As a result of these experiments, we fix the setting to prompt-based fine-tuning for all of the below experiments.
A.2 Sample schedules
One can sample  $n_t$  noise vectors at the  $t$ th step and use  $n_t$ -SPSA to compute the gradient estimate. Similar ideas were proposed in Bollapragada et al. [11], Cai et al. [15]. We study the effect of linearly increasing and constant sampling schedules in the ablation setting. The intuition for the linearly increasing schedule is that the optimizer may need a higher fidelity gradient as it approaches the minimum. Increasing the number of  $z$ s can speed up optimization by reducing the gradient variance, but doing so also increases the number of forward passes required for each optimization step, so there is a trade-off to study. We note that increasing the number of  $z$ s should be accompanied by
a proportional scaling of the learning rate, analogous to the linear scaling rule proposed in [39] (theoretical justification can follow the SDE technique [58]). Table 6 shows no consistent benefit in one schedule over the other, and it demonstrates that increasing the  $n$  in  $n$ -SPSA while fixing the number of forward passes allowed results in only marginal gains at best.
n	Schedule	SST-2	SNLI	TREC
n = 1	Constant	89.6 (1.2)	65.1 (6.2)	66.7 (6.2)
n = 4	Constant	89.5 (1.1)	68.6 (3.2)	62.3 (5.6)
n = 4	Linear	89.6 (1.4)	65.3 (6.4)	66.1 (5.5)
n = 16	Constant	90.4 (0.7)	67.0 (3.4)	62.8 (6.3)
n = 16	Linear	88.9 (1.2)	62.8 (5.9)	64.2 (5.3)
B MeZO Variants
There is a rich history of transferring ideas from first order optimization to enhance ZO algorithms. Below, we highlight several variants of MeZO that did not achieve as high performance as the algorithm presented in Algorithm 1.
B.1 Memory-efficient n-SPSA
We highlight how MeZO can perform  $n$ -SPSA (Definition 1) efficiently for  $n > 1$  in Algorithm 2. In particular, if sampling  $n$ $z$  vectors and averaging the projected gradients, we require storing  $2n$  additional scalars: the random seeds and the projected gradients. The same caveat about perturbing individual weights versus entire weight matrices still applies here (see Section 2).
B.2 Augmenting MeZO with Gradient History
The  $n$ -SPSA algorithm merely provides a gradient estimate that can subsequently be used in place of the gradient in any gradient-based optimizer. Many popular optimizers, such as Adam and SGD with momentum, require storing some historical information about gradients (e.g., a moving average). This requirement causes such algorithms to require  $2 \times$  or  $3 \times$  the memory that is needed for SGD.
However, one advantage of MeZO is that the gradient history can be recomputed at each step without requiring much additional memory. In reference to Algorithm 1, note that the gradient only needs projected_grad and the random seed  $s$  used to compute the perturbation  $z$ , so we need to only store 2 scalars per step to reproduce the gradient history (i.e., up to  $2T$  scalars during training). This is a substantial reduction in added memory overhead that is usually needed for using Adam or momentum instead of vanilla SGD.
Table 18 illustrates that MeZO-Adam can sometimes improve the performance of MeZO, though each gradient step requires additional computation (but no additional forward passes). We leave it to future work to investigate when MeZO-Adam may be more useful than MeZO.
Table 6: Experiments using MeZO with different schedules for  $n$ . We scale the learning rate proportionally to the number of  $z$ 's sampled.
Experiment	Hyperparameters	Values
MeZO-Adam	Batch size	64
Learning rate	{1e-6, 1e-5, 1e-4, 5e-4, 1e-3}
ε	1e-3
Weight Decay	0
Table 7: The hyperparameter grid used for MeZO-Adam. For simplicity, we use a constant learning rate schedule.
Require: parameters  $\pmb{\theta} \in \mathbb{R}^d$ , loss  $\mathcal{L}: \mathbb{R}^d \to \mathbb{R}$ , step budget  $T$ , perturbation scale  $\epsilon$ , batch size  $B$ , learning rate schedule  $\{\eta_t\}$ ,  $n$  for  $n$ -SPSA estimate (Definition 1)
Algorithm 2: MeZO with  $n > 1$
for  $t = 1,\dots,T$  do seeds, projected_grads  $\leftarrow []$  Will each contain  $n$  scalars for  $j = 1,\ldots ,n$  do Sample batch  $\mathcal{B}\subset \mathcal{D}^{\mathcal{B}}$  and random seed s  $\theta \gets$  PerturbParameters  $(\theta ,\epsilon ,s)$ $\ell_{+}\gets \mathcal{L}(\theta ;\mathcal{B})$ $\theta \gets$  PerturbParameters  $(\theta , - 2\epsilon ,s)$ $\ell_{-}\gets \mathcal{L}(\theta ;\mathcal{B})$ $\theta \gets$  PerturbParameters  $(\theta ,\epsilon ,s)$  Reset parameters projected_grad  $\leftarrow (\ell_{+} - \ell_{-}) / (2\epsilon)$  projected_grads[j]  $\leftarrow$  projected_grad seeds [j]  $\leftarrow s$    
end   
for  $j = 1,\dots,n$  do Reset random number generator with seed seeds [j] for  $\theta_i\in \theta$  do  $z\sim \mathcal{N}(0,1)$ $\theta_{i}\gets \theta_{i} - (\eta_{t} / n)*$  projected_grads[j]\*z Avg grad for  $z_{1},\ldots ,z_{n}$  end   
end
Subroutine PerturbParameters  $(\theta ,\epsilon ,s)$  Reset random number generator with seed  $s$  For sampling z for  $\theta_{i}\in \theta$  do  $z\sim \mathcal{N}(0,1)$  0  $\theta_i\gets \theta_i + \epsilon z$  Modify parameters in place end return  $\pmb{\theta}$
B.3 Modifying the Variance of MeZO