Stepping Forward on the Last Mile

Authors: chen  feng
shaojie  zhuo
xiaopeng  zhang
ramchalam  kinattinkara
zhaocong  yuan
andrew zou li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:47:59

Abstract: Continuously adapting pre-trained models to local data on resource constrained edge devices is the \emph{last mile} for model deployment. However, as models increase in size and depth, backpropagation requires a large amount of memory, which becomes prohibitive for edge devices. In addition, most existing low power neural processing engines (e.g., NPUs, DSPs, MCUs, etc.) are designed as fixed-point inference accelerators, without training capabilities. Forward gradients, solely based on directional derivatives computed from two forward calls, have been recently used for model training, with substantial savings in computation and memory. However, the performance of quantized training with fixed-point forward gradients remains unclear. In this paper, we investigate the feasibility of on-device training using fixed-point forward gradients, by conducting comprehensive experiments across a variety of deep learning benchmark tasks in both vision and audio domains. We propose a series of algorithm enhancements that further reduce the memory footprint, and the accuracy gap compared to backpropagation. An empirical study on how training with forward gradients navigates in the loss landscape is further explored. Our results demonstrate that on the last mile of model customization on edge devices, training with fixed-point forward gradients is a feasible and practical approach.

Main paper: 
 introduction: 
 On-device training allows pre-trained models to be continuously adapted to newly collected personal data after deployment. Moving model training from the cloud to local devices is essential for model customization and protecting users' privacy (Moon et al. [2024] ). However, the constraint on power and memory makes training on edge devices extremely challenging (Dhar et al. [2019] ). Traditional backpropagation involves a forward step, which computes activations given an input, and a backward step which computes the gradients. Intermediate activation values must be stored in memory prior to Figure 1 : An overview of fixed-point forward gradient learning. The pipeline includes quantized weights perturbation, quantized forward gradient calculation through two forward calls with perturbed weights, and quantized weights update. Each process is explained in details in section 3.3.the gradient of a certain layer is computed (Baldi and Sadowski [2016] ). As models increase in size and depth, this process requires a prohibitive amount of memory for most existing edge devices.To avoid large memory consumption, recent studies have re-examined the procedure of computing Forward Gradients as an alternative to standard backpropagation (Fournier et al. [2023] ). As introduced by Baydin et al. [2022] , a forward gradient, computed through a random, isotropic directional derivative, is an unbiased approximation of a weight gradient. Forward gradients can be further estimated solely with two forward calls of a neural network (Liu et al. [2020] ), which saves computation and memory substantially. The work of MeZO (Malladi et al. [2023] ) applies forward gradients on fine-tuning Large Lanugage Models (LLMs), and shows a success on diverse downstream tasks, with the same memory footprint as inference.Despite the aforementioned benefits, forward gradients may encounter the curse of dimensionality as the size of trainable parameters increases. Gradient approaximations from two forward calls may be noisy and with large variance (Ren et al. [2023] ), resulting in less effective training of large networks. Moreover, most existing low power neural processing engines (e.g., NPUs, DSPs, MCUs, etc.) are designed as efficient fixed-point inference accelerators. The feasibility of utilizing fixed-point forward gradients for quantized training remains uncertain. Our goal is to gain deeper insights into whether training with fixed-point forward gradients can still result in competitive models while preserving the memory and computation benefits. To answer the question, we conduct comprehensive experiments across a variety of deep learning benchmark tasks in both vision and audio domains. A series of algorithm enhancements are proposed to further reduce the memory footprint, and accuracy gap compared to backpropagation. We believe our study to be of high interest in making model personalization happen locally on edge devices.We formulate the computation of forward gradients in the quantized space. Weight perturbations and gradient calculations are all in fixed-point precision during model training or adaptation (see Figure 1 and Section 3). (b) We demonstrate the feasibility of on-device training with fixed-point forward gradients, through comprehensive experiments across a variety of deep learning benchmark tasks in both vision and audio domains. Although the method is model architecture agnostic, the experiments cover most typical model types (e.g., CNN, RNN, ViT-based) and parameter sizes (100K to 80M). (c) We propose a series of algorithm enhancements that further reduce the memory footprint and accuracy gap compared to backpropagation, leading to a practical solution for model adaptation on edge devices. (d) Finally, we visualize the neural loss landscape and trajectories of training with forward gradients, and show its dynamics and characteristics.2 Related Work 
memory efficient training through backpropagation: 
 With an increasing number of applications using large neural networks on device, there is a demand of moving model training from the cloud to local devices. However, the key bottleneck for efficient ondevice training is the limitation of memory resources. For example, training a simple Convolutional Recurrent model (CRNN, Keren and Schuller [2017] ) with a parameter size of 250kB, requires 11.5MB (46×) memory to store activations. Training memory is primarily attributed to activations rather than parameters. Studies on algorithms to reduce resource consumption during training have been published, with a trade-off between memory usage and model accuracy. Parameter-efficient fine-tuning techniques such as LoRA (Hu et al. [2021] ) and prefix tuning (Li and Liang [2021] ) are proposed to train a model with reduced parameters. Dynamic sparse representation (Mostafa and Wang [2019] ) is proposed to reduce memory requirements by making the weight and activation values sparse during training. Low precision training (Micikevicius et al. [2018] ) reduces model sizes and computation requirements by adopting 16-bit float precision instead of 32-bit. The work of Lin et al. [2022] pushes conventional convolutional neural network training on devices with only 256kB by pruning the training graph during compilation time. These methods mainly focus on reducing the trainable parameters or activation sizes, thus reduce the peak memory required for training a neural network. However, due to the inherent nature of backpropagation, intermediate activations across all layers must be retained until loss is backpropagated and gradients are calculated. Therefore, as models increase in size and depth, parameter-efficient techniques do not fundamentally resolve the training memory problem.Forward gradient has been recently brought to attention by Baydin et al. [2022] and Silver et al. [2022] , which showed that gradients can be computed solely based on the directional derivatives using the forward mode of auto-differentiation only. The forward gradients can be estimated via two forward calls using zeroth-order optimization (Liu et al. [2020] ) by incorporating random perturbations on weights, entirely eliminating the need for backpropagation in gradient descent. The work of Ren et al. [2023] shows that it is possible to substantially reduce the variance of the forward gradient estimation by applying perturbations to activations rather than weights. Considering the memory required for storage of intermediate activations, only weight-perturbed forward gradient estimator can be deployed on low resource constrained devices. While research by Belouze [2022] claimed shortcomings of forward gradients in high dimensions, the work of MeZO (Malladi et al. [2023] ) proposes a contradictory perspective by showing the lower bounds of such zeroth-order optimization is conditioned on loss landscape instead of number of trainable parameters. MeZO further applies forward gradients on fine-tuning LLMs, and shows a success on diverse downstream tasks.There is limited literature on gradient computation in the quantized space. Quantization-aware training (QAT Nagel et al. [2021] ) has been widely used to simulate the potential quantization loss in the training stage. However, most existing low power neural processors (e.g., NPUs, DSPs, MCUs, etc.) are designed and optimized for fixed-point inference. Direct training in the quantized space will fundamentally bridge the gap between training and inference, thus being essential for model adaptation on edge devices. However, the work of Lin et al. [2022] observed that the quantization process distorts backward gradients, resulting in significantly lower accuracy in model training through backpropagation. Quantization-aware scaling (QAS) is proposed to address this problem. It remains uncertain whether training with quantized forward gradients through zeroth-order optimization can still lead to competitive models on device, while preserving the memory and computation benefits. 
quantized forward gradient learning: 
 Forward gradients utilize directional derivatives to bypass backpropagation, while retaining unbiased estimations of true gradients. In the following, we first review the technique of forward-mode autodifferentiation (AD Baydin et al. [2022] ), alongside a practical implementation known as Simultaneous Perturbation Stochastic Approximation (SPSA) for zeroth-order gradient estimation (Spall [1992] ). We then propose sign-m-SPSA, a variant of SPSA to alleviate the noisy component of forward gradients estimated by SPSA, which leads to stable performance in many use cases. Once the gradients are estimated, optimizers such as SGD, Adam etc. can be applied to update the weights. Finally, we formulate the Quantized Zeroth-order Forward Gradient (QZO-FF) estimator, mapping the processes of weights perturbation, gradients estimation and weights update in the fixed-point space. An overview of the QZO-FF algorithm is illustrated in Algorithm 1.Definition 1 (Forward Gradients). Consider a machine learning function f (w) : R n → R, where w ∈ R n is the trainable parameters that the gradients are evaluated. Forward gradients g : R n → R n is defined as: g(w) = (∇f (w) • z)z (1) where z ∈ R n is a perturbation vector taken as multivariate random variable z ∼ p(z) such that z ′ s scalar components z i are independent and have zero-mean and unit variance for all i. ∇f (w) • z ∈ R, the Jacobian matrix-vector product, defines the directional derivative of f at point w in direction z.In order to have runtime advantage over backpropagation, a classical zeroth-order estimator, SPSA can be used to estimate the forward gradients by evaluating f in forward path m times, where m ≪ n.Definition 2 (SPSA). Given a model f with parameters w ∈ R n and a loss function L(w), SPSA estimates the gradient as:EQUATIONwhere z ∼ N(0, I n ) is a weighted vector over all parameter dimensions, randomly sampled from normal distribution with zero-mean and standard deviation. The perturbation scale ϵ is a small constant value (e.g., 1e -3). For each sampled z, SPSA only requires two forward calls through the model, with positive and negative perturbed weights respectively, to estimate the gradients.Gradient maganitude defined in (2) is determined by loss difference of two forward calls based on a random perturbation applied on weights, which easily becomes noisy. Inspired by many popular optimizers, such as sign-SGD and RMSProp (Bernstein et al. [2018] ), updating weights through a sign-based method achieves good practical performance for many gradient compression use cases.In order to mitigate the noisy component of forward gradients estimated by SPSA, we propose sign-m-SPSA by only taking the direction of loss difference under a certain perturbation, while disregarding the magnitude component. The estimation can be improved by averaging ĝ(w) over m randomly sampled z (m ≪ n), with an increased number of training iterations.Definition 3 (Sign-m-SPSA).EQUATIONThe intuition behind sign-m-SPSA is that during the training, the estimator samples a random perturbation direction z i , i ∈ {1, .., m}, and tests how it aligns with the true gradient by examining the loss change, and then multiplies the alignment direction with the perturbation direction. Weights will be updated along the sampled direction that leads to a decrease in loss. This design is also quantization-friendly, constraining the range of gradient values to be the same as perturbation for static quantization. Our later experiments show that 8-bit quantization of perturbation and forward gradient is sufficient for preserving the model accuracy across many use cases.Definition 4 (Sign-m-SPSA-SGD). With ĝ(w) as the forward gradients estimated through sign-m-SPSA, similar to backpropagation, an optimizer such as SGD with learning rate η can be used to update model parameters:EQUATIONSign-m-SPSA in (3) estimates forward gradients through a minimum of two forward calls, with positive and negative perturbed weights in float precision, respectively. For low power devices with fixed-point computation engines, model weights are quantized in low bit precision. Therefore, the random perturbation needs to be quantized prior to apply on weights.For a given model, consider w as the floating point weights of a certain layer. Assume model is per-tensor quantized with symmetric quantization in b-bit, the quantized weights can be represented by:EQUATIONwhere ∆ w , denoted as the quantization scaling factor, is calculated by∆ w = w max /(2 b-1 -1), where w max is the maximum absolute value in w found by a quantization method (i.e., TF, MSE, AdaRound, etc., Nagel et al. [2021] ). ⌊•⌉ represents for the rounding operation.Quantized Perturbation. With the given quantization method in 5, the quantized weights perturbation can be defined and calculated as:w ± ϵz = w • 1.0 ± ϵz ≈ ∆ w w q • ∆ z 1 q ± ∆ w ϵ q • ∆ z z q = ∆ w ∆ z (w q • 1 q ± ϵ q • z q ) re-quant =⇒ ∆ w • w q ± (6)Since weights w and perturbation z have different quantization scaling factors and possibly different bit-width used, we quantize 1.0 with the scaling factor of z, and quantize ϵ with the scaling factor of w, prior to direct adding the quantized values in accumulator. 1 q = ⌊ 1.0 ∆z ⌉, represents for the quantized value of floating point 1.0 with ∆ z as its scaling factor. Similarly, ϵ q = ⌊ ϵ ∆w ⌉, represents for the quantized value of ϵ with ∆ w as its scaling factor.The random perturbation vector z is sampled from normal distribution with zero-mean and standard deviation N(0, I n ), we can use static quantization with a pre-determined z max to pre-calculate ∆ z . For example, in the case of z max = 3.5, with 8-bit symmetric quantization, ∆ z = 0.0276, and 1 q = 36. Similarly, ϵ q can be pre-calculated, if a pre-trained model with w max is given. It is noted that ϵ is a very small value (e.g., 1e -3). Therefore, we require 16-bit to be used for weight quantization, such that ϵ can be properly represented by the minimum representation power of ∆ w without clipping loss, and small perturbation can be reflected on the weights change in the quantized space.In (6) , two values with 16-bit and 8-bit are multiplied, and then fed to a quantized add/subtract operation. In hardware, a 32-bit accumulator is used to hold the result. The result is then re-quantized to 16-bit by a multiply and a shift operation through a post-processing block (Appendix A), using the original weight scaling factor ∆ w . The quantized perturbed weights are denoted as (∆ w , w q + ) and (∆ w , w q -). The above formulation is derived under per-tensor quantization, however, per-channel quantization can be similarly derived with finer granularity.Quantized Forward Gradients. Based on the quantization method in (5), quantized forward gradients, estimated from sign-m-SPSA, can be calculated as:EQUATIONwhere g q represents for the quantized gradients, and it is using the same quantization scaling factor and bit-width as perturbation vector z.Quantized Weights Update. We can further quantize the learning rate η to a quantized value of 1, using quantization scaling factor of ∆ η = η. Finally, quantized weights update can be computed by:EQUATIONwhere wq represents for the change of weights in the quantized space, with ∆ w as the re-quantized scaling factor (Appendix A). ∆ η can be pre-calculated. In our experiments, we find that it is important to keep weights in 16-bit, while the perturbation z and gradient g can be in 8-bit representations.Momentum Guided Sampling. Besides naive SGD, quantized forward gradient learning can also be combined with other optimizers such as Adam or SGD with momentum, with slight overhead to Algorithm 1 QZO-FF: Quantized Zero-order Forward Gradient Learning(quantized, fp16)Require: quantized model parameters w q ∈ I n , loss L : • Given a pre-defined z max of perturbation z, calculate ∆ z = z max /(2 b-1 -1) with b-bit.I n → R,• Quantize 1.0 to 1 q with ∆ z .• Get the quantization scaling factor, ∆ w i , of quantized weights of each layer. Generate perturbation vector z ∼ N(0, I n ), and quantize the values to (∆ z , z q ), z q ∈ I n 6:w q + ← P erturbP arameters(w q , z q , ϵ q ) ▷ Perturb in positive direction 7:l + ← L(w q + ; B) 8:w -← P erturbP arameters(w q , z q , -2ϵ q ) ▷ Perturb in negative direction 9:l -← L(w q -; B) 10:g a q += sign(l + -l -) • z q ▷ Quantized gradient accumulation 11:w q ← P erturbP arameters(w q , z q , ϵ q ) ▷ Reset weights to original position 12:end for 13:g q = g a q /M ▷ Quantized gradient averaging 14:for w i q ∈ w q do ▷ Update weights of each layer 15:wi q = ⌊ ∆η∆z ∆ w i g q ⌉ ▷ Re-quantization (see Append.A for fixed-point approximation)16:w i q ← w i q -wi q 17:end for 18: end for 19: 20: Subroutine: P erturbP arameters(w q , z q , ϵ q ) 21: for w i q ∈ w q do 22:w i q ← ⌊∆ z (w i q • 1 q + ϵ q • z q )⌉, where ϵ q = ⌊ϵ/∆ w i ⌉ ▷ per-tensor ∆ w i 23: end for store the gradient history. Similarly, by allocating additional memory to store the perturbation history, momentum can be used to guide the sampling process. Instead of sampling solely from a zero-centered Gaussian distribution, perturbations are computed from a combination of a momentum-centered and a zero-centered Gaussian distribution. Mathematically,z 1 ∼ N(0, I n * √ α), z 2 ∼ N(z t , I n * √ 1 -α), and z t+1 = β * z 1 + (1 -β) * z 2 .Here, β is a smoothing parameter; α and β can be adaptively adjusted during training. For example, during the initial training stage, random perturbations are applied with β = 1. As training progresses, a history of the momentum z t is incorporated to guide the new sampling process.Sharpness-aware Perturbation. Motivated by the connection between sharpness of the loss landscape and model generalization, we can perturb parameter values from its neighborhood location. This is done by performing an additional step of directional gradient ascent through parameter perturbation and loss evaluation, prior to QZO-FF, as illustrated in Figure 2 . This process helps to prevent the model from converging to a sharp minimum. Sparse Update. To further reduce memory consumption, the forward gradient learning can be combined with a sparsity algorithm such that only a subset of the weights are selected from the network for updating. Examples of sparsity algorithm may include pruning by top-k magnitude, randomized pruning, pruning values beyond a specified threshold, to determine the importance of the weights. Our experiments show that incorporating sparsity with forward gradient learning allows for a 90% reduction in the size of trainable parameters, with only minor decrease in accuracy, as well as slight improvement in convergence speed.Kernel-wise Normalization. In (3), forward gradients are estimated through sign-m-SPSA. In addition, we can also apply a kernel-wise normalization to scale the gradient adaptively. z is normalized by the norm of w in each layer.ĝ(w i ) = sign(L(w + ϵz) -L(w -ϵz))z i /∥z i ∥ • ∥w i ∥ (9)4 Experiments 
few-shot learning: 
 We first apply forward gradient learning in the setting of few-shot learning, targeting to adapt a base-learner to a new task for which only a few labeled samples are available. Experiments across a variety of challenging few-shot learning benchmarks in both vision and audio domains are explored. Models are trained for each dataset individually and then evaluated with the corresponding test split.To address whether forward gradient learning (FF) could match the performance of backpropagation (BP), we explore classification tasks on training models with full fine-tuning (FT) and linear probing (LP), utilizing float16 (fp16) precision. Training accuracy with quantized FF (16-bit weights and 8-bit activations, 16w8a) is also evaluated and compared with that of fp16 precision. Details and analysis on memory usage during training are reported in Appendix B -E. Quantization bit-width. Experiments show that 8-bit weights quantization (8w8a) does not lead to model convergence. Therefore, 16-bit weights quantization is necessary to capture the small perturbation, while the perturbation z and gradients can use 8-bit.Perturbation sampling. The random perturbation z in Equation ( 2) is sampled from a normal distribution with zero-mean and standard deviation N(0, I n ). Other distibutions, such as Binomial distribution, also works well for forward gradient learning.QZO-FF enhancement. FF can be extended with sharpness-aware scheme, where a perturbation is performed at a neighborhood location through an extra step of gradient ascent. Together with kernel-wise normalization, this technique results in the closest performance to BP in both training methods. Although obtaining the norm of weights involves a trade-off between computation and accuracy, efficient implementations using gemm and sqrt operations can minimize the overhead on hardware.Loss landscape. It is believed that the convergence and generalization property of perturbation-based learning, such as forward gradient learning, depends on the loss landscape instead of number of parameters. Visualization of loss landscape has the potential to help us answer several important questions about how a neural network is trained, and why do the resulting minima generalize under certain training approach. Utilizing the tool provided in Li et al. [2018] , we show the 2D contours of loss landscape of ViT tiny network under the task of cross-domain adaptation, together with the loss trajectory during training, providing an empirical characterization of neural loss functions, and exploring how training with forward gradients navigates in the loss landscape (See Appendix E).On-device model adaptation often involves fine-tuning on data that is out-of-distribution (OOD). To evaluate the performance of FF, we pretrain a ViT tiny backbone on Cifar10, and fine-tune the decoder on Cifar10-C (Hendrycks and Dietterich [2019] ), where 15 types of corruptions, such as Gaussian noise or pixelation, of varying severity are applied. We take the lowest (easy), middle (medium), and highest (hard) corruption severity from the dataset as separate benchmarks for fine-tuning. Finetuning techniques include LP with 1 linear decoder layer, LP with 3 linear decoder layers, and D-VPT (Jia et al. [2022] ). Additionally, we explore the impact of sparsity by pruning 90% of the trainable parameters using a zero-order method (Chen et al. [2024] ). Table 5 shows a comparison of accuracy on the test set between BP, FF, quantized FF and Sparsed FF, alongside different fine-tuning methods.Detailed training hyper-parameters are listed in Appendix D.As the number of trainable parameters increases, forward gradient learning improves the model accuracy on OOD dataset. Even with a sparsity level of 90%, FF can still achieve comparable accuracy levels to those of BP. The largest accuracy disparity between the two is 6.98%, observed on the Cifar10-C (hard) category using the LP method for 3 decoder layers. As corruption intensifies, the loss surface becomes less smooth, potentially causing FF to be impacted more from the noisy gradient estimation. 
conclusion: 
 Continuously updating pre-trained models to local data on the edge is the last mile for model adaptation and customization. To overcome the memory limitation of most existing low power devices, forward gradients are used for model adaptation. We have formulated the forward gradient learning in the quantized space, where weight perturbations and gradient calculations are all in fixed-point during model training. To investigate the feasibility of on-device training with fixed-point forward gradients, we have extensively conducted experiments across a variety of deep learning benchmark tasks in both vision and audio domains. Model adaptation to cross-domain dataset and in-domain OOD datasets are further evaluated and analyzed.We further explore 2D contours of loss landscape, together with loss trajectory during training, providing an empirical explanation on how the model is trained. We have shown that quantized forward gradient learning with 16w8a can effectively adapt most typical model architectures (e.g., Resnet, ViT-tiny, CRNN, AST) and scales. With minimum accuracy reduction, fixed-point forward gradients allows model adaptation using the same memory footprint and operation support as inference, as opposed to backpropagation. Therefore, it has the potential to enable model fine-tuning on existing edge devices with limited memory and backpropagation support, without requiring additional hardware adaptation. 
 
 Appendix: The process of quantized perturbation (Equation 6) and gradient calculation (Equation 8) involves a re-quantization process. In fixed-point engines, this is approximated by a multiply and a shift operation through a post-processing block.EQUATIONwhere m and k are integer numbers, and m 2 k ≈ ∆ z .In our experiments, the number of forward-forward calls performed (m) for averaging gradients is 3 unless specified. All our experiments are running on single Nvidia Tesla V100 GPU. It is noted that our experiments do not aim to beat the benchmark state-of-the-art (SOTA) performance, but to compare the performance gap between forward and backward gradient learning across datasets and tasks. Due to the limited tuning performed, it is possible to obtain a specific result marginally better than those presented. However, this does not undermine the comparision investigated in this work.In vision benchmark, five common few-shot learning datasets are explored: CUB ( [41] ), Omniglot () and tieredImageNet ( [31] ). Each dataset is split into three parts based on different non-overlapping sets of classes, for model training, validation, and testing. All recognition tasks across datasets are using 5-way 5-shot setting.Table 6 : The hyper-parameters used in our few-shot learning experiments for vision tasks. For fair comparisons, FF and BP are using the same hyper-parameters. Model architectures of Resnet18, modified Resnet12 and ViT tiny are based on [14] , [43] , and [39] . Pre-trained models used for zero-shot evaluation can be found at [33] , [34] and [38] . Different learning rate grids are explored, and the best accuracy is reported. Figure 3 shows the memory usage of BP and FF during the training. The total memory usage during training is composed of two parts, a scratch buffer used for input and output activation tensors for gradient calculation and storage, and allocated memory for weights storage. Without storing the activation tensors, forward gradient learning has a significant reduction on the scratch memory usage. For example, in the case of full fine-tuning on ViT Tiny network, under the same precision of fp16, FF reduces the scratch memory from 31.64MB to 11.43MB (2.8×). When sparse update and fixed-point training are enabled, only 0.40MB of scratch memory is needed for model fine-tuning.The extent of memory saving with FF depends on the number of layers being fine-tuned, and their positions within the network. When applied to methods such as full fine-tuning, LoRA ( [17] ) and other parameter-efficient fine-tuning approaches, FF shows significant memory reduction because it eliminates the need to store intermediate activations. In the case of LP, where only the last few layers are updated, the difference of memory usage between BP and FF will get smaller. As the number of trainable layers increases, FF benefits more in memory savings. These promising results indicate that FF can perform comparably to BP with only a slight degradation in accuracy, while significantly reducing the memory cost. With the same memory footprint as inference, model training with FF is feasible on low memory devices where BP cannot be afforded.In audio use cases, two few-shot audio classification benchmark datasets are selected: ESC-50 ( [30] ) and FSDKaggle18 ( [11] ). Prior to adaptation, publicly available pretrained models based on AudioSet are adopted ( [1] ). The averaged accuracy after 200 epochs over 10, 000 tasks drawn from the test set is reported.Table 7 : The hyper-parameters used in our few-shot learning experiments for audio tasks. Both datasets are using 5-way 1-shot setting. For fair comparisons, FF and BP are using the same hyperparameters except that FF uses a smaller learning rate. Model architectures of CRNN and AST are based on [15] and [13] . Pre-trained models used for zero-shot evaluation can be found at [15] and [1] . Different learning rate grids are explored, and the best accuracy is reported. Cross-domain adaptation is performed on VWW dataset. Cifar10-C provides 5 levels of corruption severity, from which we take the lowest (easy), middle (medium), and highest (hard) corruption severity as separate benchmarks for fine-tuning, randomly partitioning each section into a 90%-10% train-test split.The convergence and generalization property of perturbation-based learning, such as forward gradient learning, depends on the loss landscape instead of number of parameters. Visualization of loss landscape has the potential to help us answer several important questions about how a neural network is trained, and why do the resulting minima generalize under certain training approaches. In our experiments, it is also observed that 8-bit quantization of weights does not lead to model convergence. This is because the small perturbation of ϵ is quantized using the scaling factor of weights (∆ w ). It requires higher bits to be properly represented without clipping loss, thus the weights change can be reflected in the quantized space. In future, techniques for ultra low bit (i.e., 8-bit, 4-bit) forward gradient learning can be explored. In addition, experiments beyond classification and across multiple modalities can be conducted for further evaluations. 1.9 0 1.9 5(a) BP, LP, test error 12.13%. (g) FF Quant (8w8a), LP, test error 51.35%, not converged.  


Decision: We received author rebuttals, all reviewers except Reviewer wcC5 acknowledged the rebuttals and left follow-up comments.
They also participated in AC-reviewer discussions and shared their final thoughts. 

In summary, a key concern from the reviewers is limited novelty e.g.("added quantization to ZO-FF" by Reviewer oWX1), which I partially agree to. However, I found that their claimed novelty is of the proposed method. While novelty of the method is often discussed, I believe that novelty of work is more important. This work not only introduces a new proposed algorithm but also performs comprehensive experiments, exploring experimental settings, data modalities and architectures. I appreciate this effort as it should be critical to cover various datasets / modalities (vision and audio) / models in experiments to support the authors' claims.  Reviewer uc7y values the contribution as well.

Reviewers oWX1 and wcC5 suggested lack of (L)LM evaluations as a weakness of this work, but I disagree to the assessment. Since this work is focused on on-device training, training even part of (L)LMs may be technically impossible for relatively weak devices with respect to cloud instances.

Overall, this paper introduces an interesting, practical approach and demonstrates the potential of the approach through various experiments. I recommend accepting this paper for NeurIPS'24.
Last but not least, Reviewer vPS7 pointed out as a weakness that model size in this work is relatively small (100K - 80M) for "pretrained models", which I find fair and not well addressed by the general author rebuttal. The weakness point itself seems not critical enough to reject this paper, but I think it is an important discussio for future work n as the model size has been increasing.

Reviews: review: paper_summary: paper_summary: The paper proposes a quantization approach for fine-tuning pretrained data to new local data on resource-constrained devices. In particular, the weights perturbation, gradients estimation, and weights updates are quantized to either 8-bit or 16-bit. This quantization approach is combined with Momentum Guided Sampling, Sharpness-aware Perturbation, Sparse Update, and Kernel-wise Normalization to enhance fine-tuning performance. The proposed approaches are evaluated on various AI benchmarks. The results of this study indicate that quantized forward gradients are a good candidate for a fine-tuning approach that can be deployed on edge devices.
strength_weakness: strength_weakness: 1- The author are recommended to discuss the accuracy degradation of quantized forward gradients compared to the backpropagation algorithm. In some cases, the accuracy degradation is high (more than 5%). A comparison of performance versus hardware complexity (FLOPs or another metric) is recommended, as seen in [1].

2- Evaluating the efficacy of quantized forward gradients on fine-tuning LLM models such as LLaMA-3 is recommended. 

[1] Carmichael, Zachariah, et al. "Performance-efficiency trade-off of low-precision numerical formats in deep neural networks." Proceedings of the conference for next generation arithmetic 2019. 2019.
questions: questions: Why is the random perturbation vector z sampled from a normal distribution with zero mean and standard deviation? Is it possible to sample from a log-normal distribution since activation gradients are shown to be distributed near log-normal [1]?

[1] Chmiel, Brian, et al. "Neural gradients are near-lognormal: improved quantized and sparse training." arXiv preprint arXiv:2006.08173 (2020).
limitations: limitations: The author addresses the limitations of this study by mentioning the initialization requirements for forward gradients and 16-bit weight quantization.

1- It would be suggested to discuss various initialization approaches that might be used instead of a pretrained network.
2- It would be suggested to discuss other numerical formats, such as 8-bit floating point or Posit, to solve the 16-bit weight quantization problem.
paper_summary: paper_summary: The authors investigate fixed-point forward gradients for quantized training. They conduct experiments across various deep learning tasks in vision and audio to assess if this method yields competitive models while conserving memory and computational resources.
They introduce algorithm enhancements to reduce memory usage and accuracy gaps compared to backpropagation, using fixed-point precision for forward gradients during training or adaptation.
Their findings demonstrate the feasibility of on-device training with fixed-point forward gradients across diverse model architectures (e.g., CNN, RNN, ViT-based) and parameter sizes (100K to 80M), offering practical solutions for model adaptation on edge devices.
The authors also visualize neural loss landscapes and training trajectories, providing insights into the dynamics of training with forward gradients for efficient on-device model adaptation.
strength_weakness: strength_weakness: 1. Sadly no experiments on LLMs on which most fine tuning is done today

2. Marginal novelty: generally they just added quantization to ZO-FF – is that enough?
questions: questions: 1. why you loop over w can’t you just do it vector wise?

2. Can you specify m (the number of pertubations) used for each experiment?

3. Can you calculate the memory consumption (MB/GB)  and computation complexity (in FLops) compared to QLoRA with BP.
limitations: limitations: yes
paper_summary: paper_summary: This paper explores the feasibility of on-device training using fixed-point forward gradients. The authors propose methods including sign-m-SPSA, Momentum Guided Sampling, Sharpness-aware Perturbation, Sparse Update, and Kernel-wise Normalization to reduce memory footprint and accuracy gaps and conduct experiments across various deep learning tasks in vision and audio domains. Key contributions of this paper include formulating forward gradients in the quantized space, demonstrating the feasibility of on-device training, and visualizing the neural loss landscape during training. The study shows that training with fixed-point forward gradients might be a practical approach for model customization on edge devices.
strength_weakness: strength_weakness: 1. Some results are missing in the experiment. For example, (1) the memory cost of (BP, LP, fp16) is not measured. I think the memory cost of LP is important because it seems that the reduction of memory cost mainly comes from LP instead of FF and Quant in Figure 3 and Figure 4, and I think the claim that "this number is further reduced to only 0.28MB" and "the saving increases to 8.1× when sparse update and fixed-point are enabled" in Appendix B is totally misleading and unfair. (2) The accuracy of (BP, LP, quant) is not measured so there is no baseline for (FF, LP, Quant). (3) The accuracy of (FF, FT, Quant) and (BP, FT, Quant) is not measured. (BP, FT, Quant) should be some BP fixed-point training methods like Quantization-Aware Scaling (QAS) mentioned in related work.

2. Lack of ablation studies. The effects of techniques proposed in Section 3.4 are not well-studied. (1) There is no ablation study for Section 4.1. (2) The effect of sharpness-aware and kernel-wise normalization is not measured separately in Section 4.2. (3) I want to know __which__ of these techniques work in __what__ experiment settings. I believe that, as a new algorithm with many enhancement techniques, the authors should inform the readers about which parts of the algorithm are useful under which circumstances.

3. The model size (100K - 80M) is somewhat small compared to the concept of "pretrained models". How does the proposed method perform for larger models and how does the model size affect the effectiveness of the method?
questions: questions: 1. Although one can understand the meaning of these symbols after a careful reading, the notation in equation (6) is somewhat confusing because $1_q$ and $\epsilon_q$ have the same subscript but different scaling factors. I think it would be better to add a notation related to the scaling factor above them.

2. typo in line 271: extenteded
limitations: limitations: The limitations are discussed well by the authors.
paper_summary: paper_summary: **Context**. The focus of the present paper is on-device fine-tuning (gradient computation and weight update **starting from a pre-trained model**) under limited memory budget. One way to cut the memory cost of storing the computational graph for gradient computation by standard backprop is the Memory Efficient Zeroth Order (MeZO) optimizer [Malladi et al, 2023], whereby a directional gradient is computed via weight perturbation (a.k.a SPSA): computing the loss $L$ difference yielded by two forward passes with weights differing by $\epsilon u$ estimates $\nabla L \cdot u$. Since it is a purely forward procedure, it obviates the need to cache activations to execute a backward pass.

**Core contribution**. The present paper proposes a quantized version of MeZO where weight perturbation, gradient computation and weight update are carried out on quantized quantities. The proposed algorithm, coined QZO-FF (Alg. 1), is tested against a variety of fine-tuning tasks (few-shot learning, cross-domain and in-domain adaptation), modalities (image and audio data), and architectures (convolutional, attention-based, recurrent), with several variants being explored (with fp8 / fp32 activations) and benchmarked against standard backprop. The efficiency of QZO-FF, both in terms of resulting performance and memory usage, is demonstrated.

**Paper outline**. More precisely:
- Section 2 provides background knowledge on memory-efficient backprop (2.1), forward-mode differentiation (2.2) and quantized training (2.3).
- Section 3.1 and 3.2 formalizes further "forward gradients" (3.1)  and the SPSA / weight perturbation procedure to estimate them (3.2). A hardware-friendly extension of SPSA coined as "Sign-m-SPSA", which estimates $\text{sign}(\nabla L \cdot u) u$, , is introduced along with the resulting SGD update (3.2).
- Section 3.3 presents the core algorithmic contribution by combining SPSA / weight perturbation and weight quantization (Alg. 1). More precisely, weights and perturbation are statically, symmetrically quantized (e.g. their range are estimated and set once, before fine-tuning), with one scale for each ($\Delta_w$ and $\Delta_q$). Therefore: i) $\Delta_w$ and $\Delta_q$ are fixed, with weights and perturbations quantized with 16 and 8 bits respectively, ii) the integer part of the perturbed weights is accumulated in 32 bit, iii) the dequantized perturbed weights is quantized-dequantized back into 16 bits using the same $\Delta_w$ scale (Eq. 6). The Sign-m-SPSA gradient estimator is applied and quantized-dequantized using the perturbation scale ($\Delta_z$, Eq. 7) . Finally, the weight update itself is quantized, such that it happens in the quantized integer part and is rescaled by $\Delta_w$ (Eq. 8). Alg. 1 summarizes the procedure in the case where the number of perturbed directions at use is 1 ($m=1$).
- Section 3.4 presents several algorithmic "enhancements" of the QZO-FF algorithm to improve the optimization procedure itself or its memory footprint.
- Section 4 presents experimental results. First, few-shot learning is considered (4.1) on visual and audio data. Here, "FF" refers for short to "QZO-FF". A quantized version of FF, where 8 bits activations are used, is also tested. On vision, three architectures are tested (ResNet12, 18 and ViT tiny) on 5 different standard few-short learning datasets. Two scenarii are considered: full fine-tuning and linear probing. It is shown overall that FF always yields better performance than the zero-shot baseline and within 5%, accuracy-wise, to the BP baseline on 26/30 experiments, and that the ViT backbone yields the least degradation. On audio, a similar experiment is done with two architectures (CRNN, AST) on two audio datasets. On 11/16 experiments, FF accuracy is 5% off the BP baseline. Then, a cross-domain adaptation task (4.2) is considered, where the different algorithmic enhancements previously introduced (e.g. quantized FF, gradient averaging, "sharpness aware" scheme...) are tested. Most importantly, it is observed that quantizing weights to 8 bits jeopardize the FF algorithm. Finally, sector, 4.3 presents in-domain OOD adaptation using the same fine-tuning schemes (LP, D-VPT) with three levels of corruptions of the CIFAR-10 dataset as OOD datasets. In this setting, FF achieves comparable performance with BP.
strength_weakness: strength_weakness: - It is unclear what is kept in full precision in the proposed procedure (see my questions below).
- On a related note, it is also unclear that the proposed algorithm enhancements don't offset the advantages of manipulating statically quantized quantities (see my questions below). 
- The experiments aren't all sufficiently well explained, neither in the main nor in the appendix, which is frustrating because there is a lot of work done there and we fail to deeply understand the proposed setups. I would even say that there are almost too many different experimental setups. Under constrained time budget to write the paper, I would have prioritized a lesser number of better detailed experiments rather than a lot of them left unsufficiently explained.
- **There aren't any error bars in any table and figures**, although the authors ticked in their checklist that they reported error bars and provided appropriate information about the statistical significance of their experiments (L. 520). For lack of this, it is very hard to draw any clear conclusion in terms of comparison between the different algorithms at use, e.g. is there a statistically significant gap here, or are these two results within error bar? We don't know.
- I don't understand what the 2D plot of the loss landscape really brings here in terms of insights.
questions: questions: - L. 135: "in order to mitigate the noisy component of forward gradients estimated by SPSA, we propose sign-m-SPSA": do you have evidence that sign-m-SPSA results in less noisy gradients? 
- Section 3.3: could you please clarify what is kept in full precision? I see at least three different quantities not being quantized: i) the scales $\Delta_w$ and $\Delta_z$,  ii) the loss for each perturbed weights and therefore its difference, iii) the averaged gradient (Eq. 7). Most importantly, do you confirm that you need to accumulate gradients across each direction ($i=1 \cdots m$) in higher precision (32 bits I guess?) and then quantize-dequantize it using the scale of the perturbation $\Delta_z$? Your pseudo-algorithm only treats the case $m=1$ so it remains unclear how this all work when $m>1$ and you need to average gradients. **Could you please write a new pseudo-algorithm**, alike Alg.1, **in the case $m>1$**, highlighting with **two different color codes** the quantized (int8 and int16) and full precision (fp32) quantities?
- L.193-198 (momentum-guided sampling): I think that incorporating momentum into your approach is crucial. However, I don't understand neither how it works. What do you mean by "as training progresses, a history of the momentum $z$ is incorporated to guide the new sampling process"?
- L.199 (sharpness-aware perturbation): you mean an "extra step of **directional** gradient ascent"?
- L.204-210 (sparse updates): which sparsity scheme did you employ? top-k magnitude based scheme may be quite costly, if it boils down to ranking all the weights by their magnitude.
- L.211 (kernel-wise normalization): in this case we agree that $\hat{g}$ needs to be stored in full precision? Also, the computation of the norms of $z$ and $w$ is computationally expensive ($O(d)$, where $d$ denotes the dimension of $w$ or $z$), as expensive as it would be to dynamically recompute $\Delta_z$ and $\Delta_w$, which you avoided by statically quantizing them. Don't you lose the advantage here of using static scales if in any case you need to perform this $O(d)$ operations?
- L.216 (few-shot learning): "a few labeled samples are available", but how many? Could you please clarify the experimental setup?
- L.222: "16w8a" means 16 bits for weights and 8 for activations, correct? I may not take this for granted and clearly define this notation.
- Table 2: I would rather compute the **relative** accuracy degradation (acc_BP - acc_qFF / acc_BP) rather than the **absolute** accuracy degradation.
- Table 2: the accuracy degradation when employing FF in the FT setting compared to BP in the same setting is quite severe (11.08% gap), although you are using a relatively small architecture (ResNet12) with relatively small input dimensionality (32x32). Why is this the case?
- Table 2: **there are not any error bars**, which makes it hard to make any sense of a $\sim 0.2/0.5$ difference between two experiments.
- Could you please define precisely what you mean by "zero shot" (I assume no training at all?), "linear probing" (I assume only the last linear layer is learned?) and "full fine-tuning" (all parameters are learned)? 
- L. 238 (audio benchmark): could you please detail the few-shot setup for this task, and the tasks themselves? It is important for people not familiar with this literature.
- L. 252: I really did not understand what "cross-domain adaptation" really is about. Could you please explain better what it is? 
- L.256: what is "visual-prompt tuning with deep prompts"?
- Fig. 2: except for large discrepancies between bars, it is difficult to draw any conclusion from this figure **for lack of error bars**. Could you please add them?
- Which conclusions / insights do you really gain from plotting the 2D contours of the loss landscape in the different settings?
limitations: limitations: See weaknesses and questions above.

If I had detailed description of each of the experimental setups tackled, a precise knowledge of what exactly is kept in full precision, how some of the algorithmic enhancements really work and error bars on all figures, I would be prone to increasing my score. I really want to encourage the authors to do so because I do believe that the core of this work is of interest.

Comments: title: Replying to reviewer's comment
Reviewer-author discussions will end in about 30 hours
Replying to reviewer's comment
Replying to reviewer's comment
Answer to the Authors rebuttal
Replying to reviewer's comment
Answer to rebuttal
Start reviewer-author discussions right now
comment: We appreciate the reviewer’s valuable comments and suggestions, and we are glad that most concerns have been addressed. 

In Section 4.1, we conducted a total number of 115 experiments across two modalities. Due to the number of experiments and the training time required for each, performing ablation studies for all cases is not practical within the given timeline. However, if time allows, we can select one use case and conduct ablation studies similar to those in Section 4.2. 

We welcome any suggestions on which use case would be of particular interest to the research community. 

Thank you!
Dear reviewers,

This is the final reminder for the reviewer-author discussions.
It will end on August 13 11:59 AoE, and then we will start AC-reviewer discussions.

If you have already concluded the discussions with the authors, thank you so much!

If you have not responded to the author rebuttal yet, please do so immediately. We have been waiting for your response.

In case you missed it, the general author rebuttal includes a PDF file.

Best,  
Your AC
Thanks for the response!

Most of my concerns have been addressed except that I'm not convinced by the answer to "why there is no ablation study in Section 4.1". So I keep my score as 5 but vote for acceptance.
We sincerely appreciate the reviewer’s comment on the fair comparison of memory cost of BP and ZO-FF among various training methods. To address this, we would like to make some clarifications.  

__1. Memory cost of BP and ZO-FF__

* For a network with $N$ layers, BP with FT consumes $O(N)$ memory while ZO-FF always uses $O(1)$ memory. With an increased number of $N$, ZO-FF benefits more on memory. 
* In the case of LP, where only the last layers are updated, it is expected that the difference of memory usage between BP and ZO-FF will be very small. For example, in our ViT tiny network for vision tasks, the total memory usage and scratch memory usage of (BP, LP, fp16) is $11.81MB$ and $0.45MB$, respectively. This number is very close to that of (FF, LP).
* We will provide an extra bar in Figure 3 and 4 for the case of (BP, LP, fp16), and make it clear that the comparison of memory is among the same training method. We do not compare memory usage of (BP, FT) vs (FF, LP), since they are updating different number of layers in network. 

__Ablation studies:__

We choose Section 4.2 to study the impact of various factors on ZO-FF. This section has a more controlled experimental setting (ViT tiny backbone, and VWW dataset) between BP and ZO-FF. In our ablation studies, we keep all the settings the same and vary only one factor at a time. In Section 4.1, we would like to focus on the few-shot learning benchmark results, where broader experimental settings are used, including various backbone network architectures, datasets, and vision/audio modalities.

__ZO-FF for larger models:__

We apologize for any confusion regarding "pre-trained models". In our work, the pre-trained model refers to models that are pre-deployed on the device, for instance, an object detection model. These models may need additional adaptation or personalization over time.

The primary focus of our work is to enable such model fine-tuning on existing edge devices with fixed-point engines (e.g., NPUs, DSPs, MCUs). These hardware are primary designed for inference, therefore, typically have very limited memory and lack support for BP. Implementing BP on such hardware requires substantial engineering efforts. Under this context, ZO-FF directly leverages fixed-point forward calls for gradient estimation, with the same memory cost as inference, facilitating model fine-tuning without hardware adaptation. With all the benefits of ZO-FF, we believe that it is an attractive point along the compute-memory trade-off, and especially suitable for on-device fine-tuning use cases when the limited memory is the main stumbling block.

Thank you!
We sincerely appreciate the reviewer’s comment on the comprehensive comparison of QZO-FF and BP regarding memory, computational complexity, and convergence speed. We agree that these factors are essential for evaluating different gradient calculation methods. To address these aspects, we will provide training curves, along with reporting the accuracy of each method in our ablation studies, and include a time-compute-memory trade-off analysis comparing BP, BP with checkpointing and QZO-FF. Specifically,

__1.	Convergence speed and empirical measurements__

Our experiments on the ViT tiny network with $5.2M$ parameters shows that ZO-FF converges approximately $2$x slower than BP, for both LP and VPT-deep training methods. Unfortunately, due to the format limitation, we cannot upload any link or graphs in the current response page. However, we will include the training curves in the Appendix to illustrate the convergence of BP and QZO-FF across various settings (e.g., $m=1$, $m=3$ for gradient averaging) and different precisions (fp16, quant). These curves will show the actual convergence behavior and training time required to achieve the reported accuracy. 

__2.	Compute-Memory Trade-off__

Analyzing the time-memory trade-off for backpropagation (BP) is complex. However, the work of [Griewank et al. 2008] gives a general rule of time-memory trade-off for BP (Rule 21). For a network with $N$ layers, and a time-memory tradeoff hyperparameter $c = O(1)$, there exists a BP algorithm that runs in $O(cN)$ time and consumes memory proportional to $O(N^{1/c})$. 

* In the case of $c=1$ (storing everything during the forward path), BP consumes $O(N)$ compute and $O(N)$ memory. Given the FLOPs of a backward pass are roughly $2$x of that of the forward pass, ZO-FF (with $m=3$) consumes $O(2N)$ compute and $O(1)$ memory, so it uses more compute but substantially less memory.  

* Gradient checkpointing [Chen et al. 2016] reduces memory cost of BP by recomputing some activations. In their experiments, choosing $c=2$ achieves $O(\sqrt{N})$ memory at $O(2N)$ computation. In comparison, ZO-FF is more compute-efficient at the same memory cost. 

We want to emphasize that the memory cost of gradient checkpointing is bounded by ZO-FF and BP, and FF is more compute-memory efficient at the same memory cost as gradient checkpointing along the pareto curve. 

As an example, the memory cost of training a ViT tiny network using different gradient estimation methods is illustrated as following (input size 224x224x3, batch size 1, and FP16 data type. QZO-FF is using16w8a). For BP with gradient checkpointing, we assume half of the activations are stored during forward time.

| **methods** | **weights** (MB)     | **activations (peak + stored)** (MB)          | **weight gradients** (MB)     |
|---------------------------|-----------------------|----------------------------|-----------------------|
| **BP**                    | ██████████ 10.54       | ▓▓▓▓▓▓▓▓▓▓▓▓▓ 19.56        | ▒▒▒▒▒▒▒▒▒▒ 10.54       |
| **BP_grad_checkpointing** | ██████████ 10.54       | ▓▓▓▓▓▓▓ 9.92            | ▒▒▒▒▒▒▒▒▒▒ 10.54       |
| **ZO_FF**                 | ██████████ 10.54       | ▓ 0.14                     | ▒▒▒▒▒▒▒▒▒▒ 10.54       |
| **QZO_FF**                | ██████████ 10.54       |  0.07                     | ▒▒▒▒▒ 5.27           |


__Additional comments:__

Beyond memory savings, QZO-FF provides practical benefits in several areas: 1) it enables training on existing edge devices with fixed-point engines (e.g., NPUs, DSPs, MCUs), which typically have very limited or no support for BP, because these hardware are primarily designed for inference. Implementing BP on such hardware requires substantial engineering efforts. Additionally, the required memory to run training is critical, determining the feasibility of enabling such feature on device. QZO-FF directly leverages fixed-point forward calls for gradient estimation, with the same memory cost as inference, facilitating model fine-tuning without hardware adaptation. This is a primary focus of our work. 2)  QZO-FF is also suitable for training with non-differentiable objectives (e.g., maximizing accuracy, F1-score), where BP cannot be directly applied. 

In summary, we do not expect QZO-FF to outperform BP in convergence speed or as a replacement of BP, and we will include the above analysis to guide readers on the optimal choose of these techniques. With all the benefits of QZO-FF, we believe that it is an attractive point along the compute-memory pareto curve and especially suitable for on-device fine-tuning use cases when the limited memory is the main stumbling block.

Thank you!
Thanks for the authors' response! Some of my concerns haven't been addressed so I keep my score.

**The memory cost of (BP, LP, fp16)**

Will the authors provide the memory cost of (BP, LP, fp16)? How will the author modify their statements?

**Ablation study for Section 4.1**

Why did the author only provide an ablation study in Section 4.2 and not include one in Section 4.1?

**QZO-FF for larger models**

The authors haven't conducted experiments on models larger than 80M so I think the concept of "pre-trained models" in the abstract (line 1) and the introduction (line 20) is overclaimed.
I would like to thank the authors for their detailed response. However, I still believe that a fair comparison of memory and computational complexity should include the number of steps required to achieve a certain level of accuracy. Currently, when a practitioner has a system with limited memory, they often use activation checkpointing, a technique used to manage memory consumption in large language models (LLMs) during training. Instead of storing all intermediate activations for backpropagation, only a subset of activations is saved at "checkpoints." The rest of the activations are recomputed as needed during the backward pass. This approach reduces memory usage significantly at the cost of increased computation time. If QZO-FF requires three perturbations and converges much more slowly, say five times slower to achieve same accuracy, it might not be beneficial to use it. Can you add convergence curves for BP and QZO-FF or numbers of iterations required to reach the reported accuracy? If so I'll consider raising my  score.
We sincerely appreciate the reviewer's recognition of the significance of our work and the increased score.

Thank you!
Dear authors,

Thank you very much for answering my questions and adding the detailed pseudo-algorithm in the global rebuttal.
I am happy to increase my score to accept!
Dear reviewers,

Authors submitted rebuttals, which should be visible to you now.

Please read the rebuttals carefully and start discussions with the authors now. 

The reviewer-author discussion period will end on August 13, 2024. Since authors usually need time to prepare for their responses, your quickest response would be very appreciated.

In case you had requested additional experiments / analysis and the authors provided in the rebuttal, please pay extra attention to the results.

Thank you,  
Your AC