Stepping Forward on the Last Mile
Authors: Chen Feng
Shaojie Zhuo
Xiaopeng Zhang
Ramchalam Kinattinkara Ramakrishnan
Zhaocong Yuan
Andrew Zou Li
Venue: NeurIPS 2024 poster (2024)
Keywords: On-device model adaptation
Fixed-point forward gradient learning
Low memory
Edge devices
Primary area: optimization_for_deep_networks
TL;DR: On-device model adaptation with fixed-point forward gradient learning

Abstract: Continuously adapting pre-trained models to local data on resource constrained edge devices is the \emph{last mile} for model deployment. However, as models increase in size and depth, backpropagation requires a large amount of memory, which becomes prohibitive for edge devices. In addition, most existing low power neural processing engines (e.g., NPUs, DSPs, MCUs, etc.) are designed as fixed-point inference accelerators, without training capabilities. Forward gradients, solely based on directional derivatives computed from two forward calls, have been recently used for model training, with substantial savings in computation and memory. However, the performance of quantized training with fixed-point forward gradients remains unclear. In this paper, we investigate the feasibility of on-device training using fixed-point forward gradients, by conducting comprehensive experiments across a variety of deep learning benchmark tasks in both vision and audio domains. We propose a series of algorithm enhancements that further reduce the memory footprint, and the accuracy gap compared to backpropagation. An empirical study on how training with forward gradients navigates in the loss landscape is further explored. Our results demonstrate that on the last mile of model customization on edge devices, training with fixed-point forward gradients is a feasible and practical approach.

Stepping Forward on the Last Mile
Chen Feng Qualcomm Research Qualcomm Canada ULC chenf @qti qua comm. com
Shaojie Zhuo Qualcomm Al Research Qualcomm Canada ULC shaojiezeqti. qual comm com
Xiaopeng Zhang
Ramehalam Kinattinkara Ramakrishnan
Qualcomm AI Research
Qualcomm Al Research
Qualcomm Canada ULC
Qualcomm Canada ULC
xi aaaenggatiiiccmmm qualiiiiuullcoooooo com
rkinatti@qti .uw.uuuaaaaao............ qualcomm, com
Zhaocong Yuan Qualcomm AI Research Qualcomm Canada ULC
Andrew Zou Li University of Toronto
andrewzou. 1i @mail utoronto.com ca
zhaocong@qti, qualcomm. com
Abstract
Continuously adapting pre-trained models to local data resource constrained edge devices the last mile for model deployment. However, as models increase in size and depth, backpropagation requires large amount of memory, which becomes prohibitive for edge devices. addition, most existing low power neural processing engines (e.g., NPUs, DSPs, MCUs, etc.) designed as fixed-point inference accelerators, without training capabilities. Forward gradients, solely based directional derivatives computed from two forward calls, have been recently used for model training, with substantial savings in computation and memory. However, the performance of quantized training with fixed- fixed-point forward gradients remains unclear. In this paper, we investigate the feasibility of on- device training using exed-point forward gradients, by conducting comprehensive experiments across variety of deep learning benchmark tasks in both vision and audio domains. We propose series of algorithm enhancements that further reduce the memory footprint, and the accuracy gap compared to backpropagation. An empirical study on how training with forward gradients navigates in the loss landscape is further explored. Our results demonstrate that on the last mile of model customization on edge devices, training with fixed-point forward gradients is feasible and practical approach.
Introduction
On-device training allows pre-trained models to continuously adapted to newly collected personal data after deployment. Moving model training from the cloud to local devices is essential for model customization and protecting users' privacy (Moon et al. [2024). However, the constraint on power and memory makes training edge devices extremely challenging (Dhar et al. [2019)) Traditional backpropagation involves forward step, which computes activations given an input, and backward step which computes the gradients. Intermediate activation values must be stored in memory prior to
QQualcomm Al Research is an initiative of Qualcomm Technologies, Inc
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Random Perturbation Vector
Input
Model
Forwart exec.
Loss
Quentized Weights Perfurbation (6)
Quantzee Horward Calculation
Weights
Fowward exec. 2
Loss
leamnngg rate
Quantized Weights Update
An overview of fixed-point forward gradient learning. The pipeline includes quantized weights perturbation, quantized forward gradient calculation through two forward calls with perturbed weights, and quantized weights update. Each process is explained in details in section 3.3
Figure 1:
the gradient of a certain layer is computed (Baldi and Sadowski (2016)) As models increase in size and depth, this process requires prohibitive amount of memory for most existing edge devices.
To avoid large memory consumption, recent studies have re-examined the procedure of computing Forward Gradients an alternative to standard backpropagation (Fournier et al. [2023] D. As
introduced by [Baydin et al.] [2022]. forward gradient, computed through a random, isotropic directional derivative, is an unbiased approximation of weight gradient. Forward gradients can
be further estimated solely with two forward calls of a neural network (Liu al. (0200). ), which
saves computation and memory substantially. The work of Me70 (Malladi et al.
2023 ) applies
forward gradients on fine-tuning Large Lanugage Models (LLMss), and shows a success on diverse downstream tasks, with the same memory footprint inference.
Despite the aforementioned benefits, forward gradients may encounter the curse of dimensionality as the size of trainable parameters increases. Gradient approaximations from two forward calls may be noisy and with large variance (Ren al. (2023] resulting less effective training of large networks. Moreover, most existing low power neural processing engines (e.g.. NPUs. DSPs, MCUs, etc.) are designed as efficient fixed-point inference accelerators. The feasibility of utilizing fixed-point forward gradients for quantized training remains uncertain. Our goal is to gain deeper insights into whether training with fixed-point forward gradients can still result in competitive models while preserving the memory and computation benefits. To answer the question, we conduct comprehensive experiments across variety of deep learning benchmark tasks in both vision and audio domains. series of algorithm enhancements are proposed to further reduce the memory footprint, and accuracy gap compared to backpropagation. We believe our study to be of high interest in making model personalization happen locally on edge devices.
Contributions. (a) We formulate the computation of forward gradients in the quantized space. Weight perturbations and gradient calculations are all in fixed-point precision during model training or adaptation (see Figure [ and Section 3}. (b) We demonstrate the feasibility of on-device training with fixed-point forward gradients, through comprehensive experiments across variety of deep learning benchmark tasks in both vision and audio domains. Although the method model architecture agnostic, the experiments cover most typical model types (e.g., CNN, RNN, ViT-based) and parameter sizes (100K to 80M). (c) We propose series of algorithm enhancements that further reduce the memory footprint and accuracy gap compared to backpropagation, leading to practical solution for model adaptation on edge devices. (d) Finally, we visualize neural loss landscape and trajectories of training with forward gradients, and show its dynamics and characteristies.
2 Related Work
2.1 Memory Efficient Training through Backpropagation
With an increasing number of applications using large neural networks on device, there is demand of moving model training from the cloud to local devices. However, the key bottleneck for efficient on- device training is the limitation of memory resources. For example, training simple Convolutional Recurrent model (CRNN, Keren and Schuller (2017)) with parameter size of 250kB, requires 11.5MB (46x) memory store activations. Training memory is primarily attributed to activations

---

rather than parameters. Studies on algorithms to reduce resource consumption during training have been published, with trade-off between memory usage and model accuracy. ceaumeter-efffiiene
fine-tuning techniques such as LoRA (Hu et al. (02T]) and prefix tuning ([.i and Liang [2021] D)
are proposed to train model with reduced parameters. Dynamic sparse representation (Mostafa and Wang (019)) ) is proposed to reduce memory requirements by making the weight and activation values sparse during training. Low precision training (Micikevicius et al. (2018) reduces model sizes and computation requirements by adopting 16-bit float precision instead of 32-bit. The work of Lin iet al. [2022] pushes conventional convolutional neural network training on devices with only 256kB by pruning the training graph during compilation time. These methods mainly focus reducing the trainable parameters or activation sizes, thus reduce the peak memory required for training a neural network. However, due to the inherent nature of backpropagation, intermediate activations across all layers must be retained until loss is backpropagated and gradients are calculated. Therefore, as models increase in size and depth, parameter-efficient techniques do not fundamentally resolve the training memory problem.
2.2 Forward Gradients through Zeroth-order Optimization
Forward gradient has been recently brought to attention by Baydii et al. 2022 and Silver et al. [2022], which showed that gradients can be computed solely based on the directional derivatives using the forward mode of antoddifferentiation only. The forward gradients can be estimated via two forward calls using zeroth-order optimization (Liu et al. (0200) by incorporating random perturbations on weights, entirely eliminating the need for backpropagation in gradient descent. The work Ren ctt al. [2023] shows that is possible to substantially reduce the variance of the forward gradient estimation by applying perturbations to activations rather than weights. Considering the memory required for storage intermediate activations, only weight-perturber forward gradient estimator can be deployed on low resource constrained devices. While research by [Bllouee [2022] claimed shortcomings of forward gradients in high dimensions, the work of MeZO (Malladi et al. (2023]) proposes contradictory perspective by showing the lower bounds of such zeroth-order optimization is conditioned on loss landscape instead of number of trainable parameters. MeZO further applies forward gradients on fine-tuning LLMs, and shows success on diverse downstream tasks.
2.3 Quantized Training and Quantized Gradients
There is limited literature on gradient computation in the quantized space. Quantization-awan training (QAT Nagel [agel et al. (02T]) has been widely used to simulate the potential quantization loss in the training stage. However, most existing low power neural processors (e.g.. NPUs, DSPs, MCUs, etc.) are designed and optimized for fixed-point inference. Direct training in the quantized space will fundamentally bridge the gap between training and inference, thus being essential for model adaptation on edge devices. However, the work of Lin et al. [2022] observed that the quantization process distorts backward gradients. resulting in significantly lower accuracy in model training through backpropagation. Quantization-aware scaling (QAS) is proposed address this problem. It remains uncertain whether training with quantized forward gradients through zeroth- optimization can still lead competitive models on device, while preserving the memory and computation benefits.
Quantized Forward Gradient Learning
Forward gradients utilize directional derivatives to bypass backpropagation, while retaining unbiased estimations of true gradients. In the following, we first review the technique of forward-mode autodifferrentiation (AD Baydin al. (2022), ) alongside practical implementation known as Simul- taneous Perturbation Stochastic Approximation (SPSA) for zeroth-order gradient estimation (Spall [1992). We then propose sign-m 11---mmSu SPSA. a variant of SPSA alleviate the noisy component of forward gradients estimated by SPSA, which leads to stable performance in many use cases. Once the gradients are estimated, optimizers such as SGD, Adam etc. can be applied to update the weights. Finally, we formulate the Quantized Zeroth-order Forward Gradient (QZO-FF) estimator, mapping the processes of weights perturbation, gradients estimation and weights update in the fixed-point space. An overview of the QZO-FF algorithm is illustrated in Algorithm [
3

---

3.1 Forward Gradients
Definition (Forward Gradients). Consider machine learning function f(w) R RR where w E IR" is the trainable parameters that the gradients are evaluated. Forward gradients R - R" defined as:
g(w) = (V (wo) =)=
(1)
where 2 E R is a perturbation vector taken as multivariate random variable 2 ~ p(2) such that 2's scalar components 2; are independent and have zero-mean and unit variance for all Vf(w) E R. the Jacobian matrix-vector product, defines the directional derivative of f at point in direction
3.2 Zeroth-order Optimization
In order to have runtime advantage backpropagation, a classical zeroth-order estimator, SPSA can be used to estimate the forward gradients by evaluating f in forward path m times, where m < n.
Definition 2 (SPSA). Given a model f with parameters w E R and loss function L(w), SPSA estimates the gradient as:
@(w) L(w + c2) L(w -(w
(2)
2E
where z ~ N(0, In) a weighted vector over all parameter dimensions, randomly sampled from normal distribution with zero-mean and standard deviation. The perturbation scale is small constant value (e.g.. le - 3). For each sampled 2, SPSA only requires two forward calls through the model, with positive and negative perturbed weights respectively, estimate the gradients.
Gradient maganitude defined ( is determined by loss difference of two forward calls based on random perturbation applied weights, which easily becomes noisy. Inspired by many popular optimizers, such as sign-SGD and RMSProp (Bernstein et al. (2018), updating weights through sign-based method achieves good practical performance for many gradient compression use cases. In order to mitigate the noisy component of forward gradients estimated by SPSA, we propose Sgn-m-SPSA by only taking the direction of loss difference under certain perturbation, while disregarding the magnitude component. The estimation can be improved by averaging (ww) over m randomly sampled = (m & n)), with an increased number of training iterations.
Definition (Sign-m-SPSA).
((w) = Σ sign([((w + ez;) - I[w - ez;))=;
The intuition behind sign-m-SPSA that during the training, the estimator samples random perturbation direction 2,, i E {1..., m}. and tests how it aligns with the true gradient by examining the loss change, and then multiplies the alignment direction with the perturbation direction. Weights will be updated along the sampled direction that leads to a decrease in loss. This design is also quantization-frin frienndly constraining the range of gradient values be the same as perturbation for static quantization. Our later experiments show that 8-bit quantization of perturbation and forward gradient is sufficient for preserving the model accuracy across many use cases.
Definition 4 (Sign-m-SPSA-SGGD), With ((w) as the forward gradients estimated through sign-m- SPSA, similar backpropagation, an optimizer such as SGD with learning rate 7) can used to update model parameters:
W++1
W; (w)
3.3 Quantized Weights Perturbation and Forward Gradients
Sign-m-SPSA (3) estimates forward gradients through a minimum of two forward calls, with positive and negative perturbed weights in float precision, respectively. For low power devices with fixed-point computation engines, model weights are quantized in low bit precision. Therefore, the random perturbation needs to be quantized prior to apply on weights.
For given model, consider w the floating point weights of a certain layer. Assume model is per-tensor quantized with symmetric quantization in b-bit, the quantized weights can be represented by:
W
Wy =
(5)
A

---

where Aw, denoted as the quantization scaling factor, is calculated by Ae Wmax/ /maz/(2" (2 1). where waa is the maximum absolute value in w found by quantization method (i.e., TF, MSE, AdaRound, etc.. Nagel et al. 2.22]). [-] represents for the rounding operation.
Quantized Perturbation. With the given quantization method in tion can be defined and calculated as: w + EZ
5
the quantized weights perturba-
w 1.0 E EZ
2 A w Ag 1 + A wey 2g
(6)
=w, 1, = Eq 2g) re re-guant Aw Wyy
Since weights w and perturbation 2 have different quantization scaling factors and possibly different bit-width used, we quantize 1.0 with the scaling factor of and quantize with the scaling factor
of w. prior to direct adding the quantized values in accumulator. 1 = - represents for the
quantized value of floating point 1.0 with A, as its scaling factor. Similarly, <q = [A_]] represents for the quantized value of e with Aw as its scaling factor.
The random perturbation vector is sampled from normal distribution with zero-mean and standard deviation ((0 I.). we can use static quantization with a pre-determined Smaz to pre-calculate Ag. For example, in the case of Smar 3.5, with 8-bit symmetric quantization, A, 0.0276, and can be pre-calculated, a pre-trained model with Wmaz is given. It is noted that is very small value (e.g., Ie 3). Therefore, require 16-bit to be used for weight quantization, such that can be properly represented by the minimum representation power of A. without clipping loss, and small perturbation can be reflected the weights change in the quantized space.
1, = 36. Similarly,
In (6). two values with 16-bit and 8-bit are multiplied, and then fed to quantized add/subtract operation. In hardware, 32-bit accumulator is used to hold the result. The result then re-quantized to 16-bit by multiply and shift operation through post-processee procpprocssing block (Appendix [A). using the original weight scaling factor A The quantized perturbed weights are denoted as (Aw: Wi+ and (Awr Wg-). The above formulation is derived under per-tensor quantization, however, per-channel quantization can be similarly derived with finer granularity.
Quantized Forward Gradients. Based the quantization method in 5 quantized forward gradients, estimated from sign-m-SPSA, can be calculated as:
95 =
si + Fes) - L(w -c2,))z
sign((((w.)) L(w, ))
m71
A29g
where 94 represents for the quantized gradients, and it is using the same quantization scaling factor and bit-width as perturbation vector 2.
Quantized Weights Update. We can further quantize the learning rate to a quantized value of 1, using quantization scaling factor of A, 17. Finally, quantized weights update can be computed by:
Wh+1 Wi n9g
Au Wlg A
A,11A99
A
(8)
W A.
9o
A
(w, w.g))
where W. represents for the change of weights the quantized space, with An as the re-quantized scaling factor (Appendix/A) A). A, can be pre-calculated. In our experiments, we find that it is important keep weights 16-bit, while the perturbation 2 and gradient g can be in 8it representations.
3.4 QZO-FF enhancement
Momentum Guided Sampling. Besides naive SGD, quantized forward gradient learning can also be combined with other optimizers such Adam SGD with momentum, with slight overhead to
5

---

Algorithm QZO-FF: Quantized Zero-order Forward Gradient Learning(quantized, fpl6)
Require:
quantized model parameters Wp E [ loss L: 1 + I,, perturbation scale training steps
T. batch size B, learning rate schedule {
1: Given pre-defined Zmaz of perturbation calculate A, Z/nar/ 1) with b-bit.
Quantize 1.0 to 1 with A..
Get the quantization scaling factor, we, of quantized weights of each layer.
2:
for t= 1, ..... do
3:
for m=l...., do
4:
Sample random seed and batch B
5:
Generate perturbation vector & ~ N(0, In), and quantize the values to (A., Zg), 2g E 1 Perturb in positive direction
6:
Wht PerturbParameters(un 2g: Eq)
7:
4 - L( B)
8:
2Eg)
Perturb in negative direction
9:
L + L(w ; B)
10:
9 sign(l+ -L)
Quantized gradient accumulation Reset weights to original position
11:
Wg
PerturbParamcterst Cq)
12:
end for
9q
99 9
Quantized gradient averaging b. Update weights of each layer
for w
W/g
15:
A.
0g
Re-quantization (see Append.A for fixed-point approximation)
W 17: end for
16:
<- W
18: end for
19:
20: Subroutine: Perturb PerturrParamccerss(u Zyg Cq)
21: for / W.y do
22: w <- ( l + co zg)]. where 6q Lc/Awi)
per-tensor A
23: end for
store the gradient history. Similarly, by allocating additional memory to store the perturbation history, momentum be used guide the sampling process. Instead of sampling solely from zero-centered Gaussian distribution, perturbations are computed from combination of momentum-ccenterd. and a N(0, J, Va), 22 N(2,, I,, /1 a), and Z 3 2 + (1 3) * 22. Here, , is smoothing parameter; a and 3 can be adaptively adjusted during training. For example, during the initial training stage, random perturbations applied with 3 = 1. As training progresses, a history of the momentum is incorporated to guide the new sampling process.
zero-centered Gaussian distribution. Mathematically,
Perturbation. Motivated by the connection between sharpness the loss land- scape and model generalization, This done by performing an additional step of directional gradient ascent through parameter perturbation and loss evaluation, prior QZO-FF, illustrated in Figure 2 This process helps to prevent the model from converging to sharp minimum.
Sharpness-aware
perturb parameter values from its neighborhood location.
Sparse Update. To further reduce memory consumption, the forward gradient learning can be combined with a sparsity algorithm such that only a subset of the weights are selected from the network for updating. Examples of sparsity algorithm may include pruning by top-k magnitude, randomized pruning, pruming values beyond a specified threshold, to determine the importance the weights. Our experiments show that incorporating sparsity with forward gradient learning allows for a 90% reduction in the size of trainable parameters, with only minor decrease in accuracy, as well as slight improvement in convergence speed.
Kernel-wise Normalization. addition, we can also apply a kernel-wise normalization to scale the gradient adaptively. 2 is normalized by the norm w in each layer.
(3). forward gradients are estimated through sign-m-SPS -PSAA. In
g(w') sign(L(w + EZ) - [lw ez)) /1122 [
(9)

---

4 Experiments
4.1 Few-shot learning
We first apply forward gradient learning in the setting of few-shot learning, targeting to adapt a base-learner to a new task for which only a few labeled samples are available. Experiments across variety of challenging few-shot learning benchmarks in both vision and audio domains are explored. Models are trained for each dataset individually and then evaluated with the corresponding test split.
To address whether forward gradient learning (FF) could match the performance of backpropagation (BP), we explore classification tasks on training models with full fine- tne-ttning (FT) and linear probing (LP), utilizzig 1 (fp16) precision. Training accuracy with quantized FF (16-bit weights and 8-bit activations, 16w8a) is also evaluated and compared with that of fp16 precision. Details and analysis on memory usage during training are reported in Appendix
Table I: Vision datasets used for few-shot leamning
Name
Setting
No. Classes (train/valtest)
No. Samples
Resolution
CUB Omniglot
Bird Species
200 (1 (140/30/30)
11,788
84x 84
Handwritten characters
1623 (1000/200423)
32,460
28x 28
CifarO0D_G Is
Color
100 (64/16/20)
60,000
32x 32
minilnageNet
Natural images
100 (64/16/220)
60,000
84x 84
tieredImageNer
Natural images
608 (3///97/160)
779,165
84x
Table 2: Vision tasks: few-shot learning accuracy (%) with Forward (FF) and Backward (BP) gradients. The averaged accuracy 100 testing tasks reported, FT: full fine tuning: LP: linear probing: Quant: 16w8a with symmetric quantization. FF outperforms zero-shot across the board, and achieves comparable performance (accuracy within 5%) to BP 26 out of 30 tasks.
Backbone
Training
CUB
Omniglot
Cifar100 fs
minilmageNet
tieredlmageNeet
Zoro-shot
68.46
92.00
60.44
84.44
80.92
BP. FT
85.32
99.62
82.32
87.34
82.54
Resnet12
BP. LP
84.14
98.64
72:42
87.46
81.96
FF, FT
80.58 c17
97 44 AM
71.24 (100)
87.36 (weats
82. 12: (647)
FE, LP
79.02
96.62
70.30
87.30 (ead)
82. 22
FF, LP, Quant
77.42
96.08
68.54
87.00
81.64
Zero-shot
59.96
86.68
74.60
82.58
80.44
BP.
79.28
$6.34
86.78
Resnet18
BP, LP
78.92
96.48
84.88
87.42
84.68
FF. FT FF. I.P
76.34 case
94.70 :024:
82.20 (.:14)
87.66 ewEre
care
73.64
95.56 (00m)
82.32 (m
87. 14 (n
83.02 (1
FE, Quant
70.54
74.92
85.74
$1.00
Zero shot
90.60
90.96
82.28
98.78
94.30
BP. FT 93.08
99.88
90.88
98.46
96.04
VIT tiny
BP. LP
93.90
95.77 78
84.42
98. 40
95.32
93.58 (estt
-200)
va.22)
FF. LP
92.26 (3.99
95 10. 14mm
84.48
(.200)
99.02
95. 18.18 ce.1
FF, LP, Quant
92.24
95.04
84,40
99.00
95.15
Vision Benchmark. Image classification models are compared across commonly used 5 few-shot learning benchmark datasets (Table D Training methods are evaluated on 3 network backbones (modified R Ye et al. 2220)) Resnet1 @ He et al. (2015) and ViT tiny Dosovitskiy et al. (2020) with ProtoNets Snell et al. 2017] as few-shot classifier.
Table 2 demonstrates the classification accuracy on vision benchmarks. We first show that FF significantly improves over zero-shot performance across model types and tasks. Given that FF solely utilizes directional derivatives for gradient estimation, it is expected that BP generally outperforms FF in most tasks. The accuracy gap between BP and FF can vary based on factors such as backbone architecture, dataset and task difficulty. The largest accuracy degradation is observed when training Resnet12 on Cifar- 1:0 dataset with an input resolution of 32 x 32. However, using a stronger backbone such as ViT, can help bridge this accuracy gap. This indicates that while FF may show some degradation with smaller architectures and low-resolution inputs, performance improvements can be achieved with more advanced models. Overall, FF achieves comparable performance (accuracy within 5%) to BP in 26 out of 30 comparable experiments. A minimal accuracy drop observed in quantized FF training, when strong backbone such as ViT tiny is used. These promising results
7

---

Table 3: Audio datasets used for few-shot leaming. The ESC-50 dataset includes labeled collection 2000 environmental audio recordings, and FSDKaggle2018 is an audio datasel containing 11,073 audio liles annotated with 41 labels of the AudioSet Onlology. Both dalasets used for benchmark- ing methods of environmental sound classification.
Name
Setting
No. Classes (trinnvaattes)
No. Samples
Sample Length
ESC-50 FSDKaggle18
Environmental
50 (35/5/10)
2,000
5s
Mixed
41 (29/577)
11,073
0.3g 30s
indicate that FF can perform comparably to BP with only a slight degradation in accuracy, while significantly reducing the memory cost (see analysis in Appendix B.I). With the same memory footprint inference, model training with FF is feasible on low memory devices where BP cannot be afforded.
Two audio benchmark datasets (ESC-50 and FSDKaggle18) selected (Table 3) for sound classification use cases using few-shot learning. Similar to vision, training methods are evaluated on 2 representative architectures CRNN (Heggan et al. (2022) and Audio Spectrogram Transformer (AST(Gong et al. (2021). D, with SimpleShot (Wang et al. (019)) and ProtoNets (Snell
Audio Benchmark.
al. 2017) as few-shot classifiers.
Table 4: Audio tasks: fow-shot learning accuracy (%) with Forward (FF) and Backward (BP) gradients. FF achieves comparable (accuracy within 5%% better performance to BP II out of tacks.
Backbone
Training
ESC-50
FSDKagele18
SimpleShot
ProtoNet
SimpleShot ProtoNet
BP BP
66.34
73.82
38.89
33.11
72.11
71.30
36.88
32.67
CRNN
FE FT FF, LP
07.20 cuses
64,30 va1.9
36.04 c209)
35,52 (:24))
37.38..
61.62
37,53
34.67
FT, LPP. Quant
67.05
63.43
36.90
35.55
BP, FT BP, LP
68.04
75.85
38.12
46.12
75.98
70.16
42.86
42,64
AST
FF, FT LP
79.70 -110
66.98 c157)
42.92 (45h
40,50 (54b
76.07 CoOns
c430
42.72 (334)
18.18
6440
FT, LPP. Quant
76.13
61.86
42,90
38.10
Table 4 reports classification accuracy on audio benchmarks. Compared to vision tasks, the accuracy gap is larger, ranging from - -1.339% to +11.66%. This may be due to the extremely challenging training setting of 5-way 1-shot, where only 1 example of each class seen in each task. Additionally, we found that the pretrained model from AudioSet (AST) does not produce good zero-shot performance across all tasks. This indicates that a good initial baseline is critical for model adaptation. Overall, FF achieves comparable (accuracy within 5%) or better performance to BP on 11 out of 16 tasks. Training with quantized FF (16w8a) maintains similar accuracy level as fp16. From memory analysis in Appendix [.22] training an AST model with quantized forward gradients combined with sparse update, requires only 0.19MB scratch memory, which fits into most existing edge devices.
4.2 Cross-domain Adaptation
We further conduct experiments on model adaptation to cross-domain datasets, in which a models is fine-tuned on tasks with data distribution significantly different from those of the pre-trained model. For ablation studies on various impacts on the training accuracy, we take ViT tiny (5.5M parameters) as backbone for feature extractor, and apply a randomly initialized linear layer as the decoder for binary classifier. The model is pretrained on ImageNet-1k through DeiT (Touvron et al. (202T), ) and adapted for Visual Wake Word (VWW) task (Chowdhery et al. (2019) through linear probing (LP). where only the decoder layer is fine-tuned, and
tuning with deep prompts (D-VPT, Jia al. (2022) D. where prompts in each Encoder layer are also fine-tuned. Testing accuracy is reported in Figure and detailed training syper-parameters are listed in Appendix C
Effectiveness of Quantized FF. With LP, quantized forward gradient learning is capable of training the model to an accuracy of 87.30% from 48.50% with an accuracy gap of 0.63% compared to BP
in fp16.
Gradient averaging in FF, A larger m, used to average forward gradients, helps to smooth the noisy estimation and increases the model accuracy. With D-VPT training in fp16, simply increasing to 3
8

---

Classfication ssendaadd deviation
(MTT Tiny. VWW)
41.0
LP
S7A33 CU0M
T7 (P82)
60
2010 andt 1p16,
5:: 1o15
1218,
Schematic of sharpness-awas update. weights perturbation at neighborhood posilion.
mn3
#FT, Dinamial
#FF, quant mld Gincmial
#FT: Bulla, mn)
(a). Adaptation eccuracy (quant: 16w8a, unless specified).
Figure 2: Ablation studies cross-domain adaptation. The accuracy numbers (with standard deviation) are averaged over runs.
boosts the accuracy by 1.22%. However, there is trade-off between model accuracy and training efficiency.
Quantization bit-width. Experiments show that 8-bit weights quantization (8w8a) does not lead to model convergence. Therefore, 16-bit weights quantization is necessary to capture the small perturbation, while the perturbation 2 and gradients can use 8-bit.
Perturbation sampling. The random perturbation in Equation 22) is sampled from normal distribution with zero-mean and standard deviation N(0, I,). Other distibutions, such as Binomial distribution, also works well for forward gradient learning.
QZO-FF enhancement. FF can be extended with sharpness-awar aware scheme, where a perturbation performed neighborhood location through extra step of gradient ascent. Together with kernel-wise normalization, this technique results in the closest performance to BP both training methods. Although obtaining the norm of weights involves trade-off between computation and accuracy, efficient implementations using gemm and sqrt operations can minimize the overhead on hardware.
Loss landscape. is believed that the convergence and generalization property of perturbation-base learning, such forward gradient learning, depends on the loss landscape instead of number of parameters. Visualization of loss landscape has the potential to help us answer several important questions about how neural network is trained, and why do the resulting minima generalize under certain training approach. Utilizing the tool provided in Li et al. [2018]. we show the 2D contours of loss landscape of ViT tiny network under the task of cross-domain adaptation, together with the loss trajectory during training, providing an empirical characterization of neural loss functions, and exploring how training with forward gradients navigates in the loss landscape (See Appendix (E).
4.3 In-domain OOD Adaptation
On-device model adaptation often involves fine-tuning on data that is distribbutt (OOD). To evaluate the performance of FF, we pretrain VIT tiny backbone on Cifar10, and fine-tune the decoder on Eifar10-C (Hendrycks and Dietterich (209)). where 15 types corruptions, such as Gaussian noise or pixelation, of varying severity are applied. We take the lowest (easy), middle (medium), and highest (hard) corruption severity from the dataset as separate benchmarks for fine-tuning. Fine- tuning techniques include LP with I linear decoder layer. LP with 3 linear decoder layers, and D-VPT (Jia et al. 20222 ). Additionally, we explore the impact of sparsity by pruning 90% of the trainable parameters using zero-order method (Chen al. (0244). Table 5 shows comparison of accuracy on the test set between BP, FF, quantized FF and Sparsed FF, alongside different fine-tuning methods.
Detailed training hyper-parameters are listed in Appendix D
As the number of trainable parameters increases, forward gradient learning improves the model accuracy on OOD dataset. Even with sparsity level of 90%, FF can still achieve comparable accuracy levels to those of BP. The largest accuracy disparity between the two is 6.98%, observed on the Cifar 10-C (hard) category using the LP method for decoder layers. As corruption intensifies,

---

Table Accuracy (%) of model adaptation in-domain OOD dataset with Forward (FF)
Backward (BP) gradients. LN: linear layer decoder; LN: linear layer of decoder. Quant:
16w8a, Spurse: 9%% weights pruned. The sccuracy numbers (with standard deviation) averaged over runs.
Backbone
Training
Cifar10 C (easy)
CifarlO-C C (median)
CifarlO-C (hard)
Zero-shot
82.48
74.59
62.40
BP
83.75 (1 0.67)
77.88 0.85)
70.03 (1 1.20)
LN
FF
83.37 (#: 0.60)
77.04 (2: 0,66)
68.65 (= 0.70)
FE, Sparse
83,34 0.59)
77.11 0.68)
68.63 (1 0.95)
FE. Quant
83,23 (#: 0.57)
76.73 (2: 0.75)
68.28 (2 0.87)
Zero shot
85.83
77.77
62.25
LP
BP FF
86,99 0.41)
81.57 (# 0.78)
74.76 (= 0.90)
3 LN
86.11 0.59)
79:17 0.70)
67.78 0.72)
FF. Sparse
86.10 0.58)
79.24 (#: 0.63)
68.06 (= 1.11)
FF, Quant
85.77 0.55)
78.67 0.63)
67.25 0.42)
Zero-shot
89,52
82.24
68.95
91.66 0.50)
(+
D.VPT
FF Sparse
90.58 (.1. 0.53)
86.21 (1. 0.49)
78.38 (1L 0.80)
0.48)
(+ 0.81)
FE, Quani
90.41 (1 0.49)
85.77 (1 0.43)
77.45 (1. 0.64)
the loss surface becomes less smooth, potentially causing FF to be impacted more from the noisy gradient estimation.
5 Conclusion
Continuously updating pre-trained models to local data on the edge is the last mile for model adaptation and customization. To overcome the memory limitation of most existing low power devices, forward gradients are used for model adaptation. We have formulated the forward gradient learning the quantized space, where weight perturbations and gradient calculations are all in fixed-point during model training. To investigate the feasibility of on-device training with fixed-poiis forward gradients, we have extensively conducted experiments across a variety of deep learning benchmark tasks in both vision and audio domains. Model adaptation to cross-domain dataset and in-domain OOD datasets are further evaluated and analyzed. We further explore 2D contours of loss landscape, together with loss trajectory during training, providing empirical explanation on how the model is trained. We have shown that quantized forward gradient learning with 16w8a can effectively adapt most typical model architectures (e.g.. Resnet, ViT-tiny, CRNN, AST) and scales. With minimum accuracy reduction, fixed-point forward gradients allows model adaptation using the same memory footprint and operation support as inference, opposed to backpropagation. Therefore, has the potential to enable model fine-tuning on existing edge devices with limited memory and backpropagation support, without requiring additional hardware adaptation.
10

---

References
[1] AST. Pretrained ast checkpoint. URL httpps///iithhn htt.g//www...oouuuuaaaaaaaaaaaaa tree/
master
[2] Baldi and P.J. Sadowski. A theory of local learning, the learning channel, and the optimality of backpropagation. 83:51-74, 2016. doi: 2..101//.eeumee22116...7..
[3] A. G. Baydin, B. A. Pearlmutter, D. Syme, F. Wood, and P Torr. Gradients without backpropa- gation. 2022. URL htttps://arxiv.orggpeeee hwwp://d//200..../22 /2202008877
[4] G. Belouze. Optimization without backpropagation. 2022. /t///ttpps//arxxx..rrr/pdt 063022
[5] Bernstein, Y. Wang, K. Azizzadenesheli. and A. Anandkumar. Signsgd: Compressed optimisation for non-convex problems. Proceedings of the 35th International Conference on Machine Learning, 2018. https:/t::///ar http:::://xxvv.orr///// /ttp:ooror/p/////..032..2... 111/22002004444
[6] L. Bertinetto, J.F. Henriques, P.H.S. Torr. and A, Vedaldi. Meta-learning with differentiable closed-form solvers. ICLR, 2019. URL URL httLhtttp:/aaatt wwtppdf//////oooto http////..000.113111
[7] A. Chen, Y. Zhang, J. Jia, J. Diffenderfer, J. Liu, K. Parasyris, Y. Zhang, 7. Zhang, B. Kailkhura, and S. Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. ICLR, 2024. BtRLstttp://axiii. http:/awwwv...ggaaaamm. org/ 02310000025.
[8] A. Chowdhery, P.
J. J.lens, A. Howard, and R. Rhodes. Visual wake words dataset. wwt.////9////./.....moo 05721
2019. URL ht
[9] S. Dhar, J. Guo, J. Liu, S. Tripathi, U. Kurup, and M. Shah. On-device machine learning: An algorithms and learning theory perspective. 2019. URL httpp:://aaxii..rgg//...o/// /1911 00623v1
[10] A. Dosovitskiy, I.. Beyer, Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. CVPR, 2020. URLhttts:://rrxt http:///xxivv.rr////
abs/2010. 11929
[11] E. Fonseca, M. Plakal, D. P. Ellis, F. Font, X. Favory, and X. Serra. Learning sound event classifiers from web audio with noisy labels. ICASSP, 2019. /tattts::/arriv....
1901.01189
[12] I. Fournier, S. Rivaud, E. Belilovsky, M. Eickenberg, and E. Oyallon. Can forward gradient match backpropagation? Proceedings of the 40 International Conference on Machine Learning, 2023.
/2306.33620 06968
[13] Y. Gong, Y. Chung, and Glass. Ast: Audio spectrogram transformer. Interspeech 2021, 2021. URL
[14] K. He, X. Zhang. S. Ren, and Sun. Deep residual learning for image recognition. In CVPR. 03385
2015. URL htttps://arxivv.or//bbbbb
[15] C. Heggan, S. Budgett, T. Hospedales, and M. Yaghoobi. Metaaudio: A few-shot audio classification benchmark. ICANN, 2022. URL htttps://arxiv.ooo http:/dff/22444 /2204. 02121
[16] D Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representa- tions, 2019.
[17] E.J. Hu, Y. Shen, P. Wallis, Z. Allen-Ziin Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. 2021. URL htttss://arxat org/pdf /2106. 09685
[18] M. Jia, I.. Tang, B. Chen, C. Cardie, S. Belongie, B. Hariharan, and Lim. Visual prompt tuning. In European Conference Computer Vision (ECCV), 2022.

---

[19] G. Keren and B. Schuller. Convolutional mn: an enhanced model for extracting features from sequential data. 2017. JRLpp:tt:///rrci.orrg httpi//.rrgpp/.//////o/e 0wt.:/1122..000 05875
[20] B. M. Lake, R. Salakhutdinov, and B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, pages 1332-13333 2015. 10. 00.222002/D1 (122002/D1.200 20098
org/
[21] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the loss landscape of neural nets. NeurIPS, 2018. https::tt:///aiiiii wtg/ 0191/19912.9993133
[22] X. L. Li and P. Liang, Prefix-tuning: Optimizing continuous prompts for generation. 2021. URI. hul.https:: //trx/aavvvvv.vv////a wtt.aaii....pp///////./ /ttp:/pp/211100000999
[23] Lin, J. Zhu, W. Chen, W. Wang, C. Gan, and S. Han. On-device training under 256kb memory. NeurlPS, 2022. URL /ttp:i..//axxvvvvoo http:/df/222000 /206 15472
[24] S. Liu, P. Chen, B. Kailkhura, G. Zhang, A. Hero, and P.K. Varshney. A primer on zeroth-order optimization in signal processing and machine learning. IEEE Signal Processing Magazine, 2020. htRLshtps::/// https:://arvvvvooo org.//vvvorrroooo org/pdt /pff /2006 06224
[25] S. Malladi, T. Gao, E. Nichani, A. Damian, J.D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. NewrIPS, 2023. JRKLLtttps///rxii.orr/////
2305. 17333
[26] P. Micikevicius, S. Narang. J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and Wu. Mixed precision training. ICIR, 2018. URL https //arxiv.org pdf /1710.100000 03740)
[27] J. Moon, H.S. Lee, J. Chu, D. Park, S. Hong, H. Seo, D. Jeong, S. Kong, and M. Ham. new frontier of ai: On-device ai training and personalization. ICSE, 2024. URL https
7/ /arxiv.org/ /pff /22062262222 04688
[28] H. Mostafa and X. Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. Proceedings of the 36th International Conference on Machine Learning, 2019. uttpttts: htttt://///xxiiit http::::///xvvvv.aaeeeo pdf /1902. 05967
[29] M. Nagel, M. Fournarakis, R.A. Amjad, Y. Bondarenko, M.V. Baalen, and T. Blankevoort. A white paper on neural network quantization. CVPR, 2021. https://///ariti org.gor.gpppf....
[30] K. J. Piezak. Dataset for environmental sound classification. Proceedings of the 23rd ACM international conference on Multimedia, 2015. URL httpp:://t..aataa acm. wwt.aa//mo.///////o..o////////a. 1145/2733373. 2806390
[31] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R.S. Zemel. Meta-learning for semi-supervised few-shot classification. ICLR, 2018. URL http:///xxvvv.oooaabee 00676
[32] M. Ren, S. Kornblith, R. Liao, and G. Hinton. Scaling forward gradient with local losses. ICLR,
2023. URL http://///aavvvvorrrpptt////////
[33] resnet12. Pretrained resnetl2 checkpoint download. URL http::///rrireegoogle.coo
[34] resnet18. Pretrained 1 checkpoint download. URL htttps: pytorch
agg/modess//eeeet1118 f3772ff..
[35] D. Silver, A. Goyal, I. Danihelka, M. Hessel, and H.V. Hasselt. Learning by directional gradient descent. ICLR, 2022. Ut/
[36] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
[37] J.C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, pages 332-341, 1992.
12

---

[38] VIT tiny. Pretrained vit tiny checkpoint download. URL https: U .baiipublicffles http///ttt..iiyyyaaaaiaaieeeee_, a1111bb..1.b.ff..ccm pth
[39] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, volume 139, pages 10347-10357, July 2021.
[40] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching networks for one shot learning. 2016. URL URL WRLLhttps://axxi.orrg httpe/iivvrrga/b///eeeeeee
abs/ 1606. 04080
[41] C. Wah, S. Branson, P. Welinder, Perona, and Belongie. Caltech-uesd birds dataset. 2022. URL htttp:://ooi... org/ 10. 2020/002022/0/1222099
[42] Y. Wang. W. Chao, K. Q. Weinberger, and L. Maaten. Simpleshot: Revisiting nearest-neighhor classification for few-shot learning. CVPR, 2019. URI. httppppss/aarrtii.orr/////////// 04623
[43] H. Ye, H. Hu, D. Zhan, and F. Sha. Few-shot leamiigg via embedding adaptation with set-to-set /aobs///11..00000 1816.03664111
functions. CVPR, 2020. URL
13

---

A Fixed-point re-quantization
The process of quantized perturbation (Equation and gradient calculation (Equation[B) involves a re-quantization process. fixed-point engines, this is approximated by a multiply and a shift operation through post-processing block.
Wl As(u, 1a + Eq. 2q)
(10)
= (wg lg + =q) m > k
where and k are integer numbers, and m
2 A..
22
B Few-shot learning experiments
In experiments, the number of geward-forwarte calls performed (m) for averaging gradients is 3 unless specified. All our experiments are running on single Nvidia Tesla V100 GPU. It is noted that experiments do not aim to beat the benchmark state-of-the-arr (SOTA) performance, but to compare the performance gap between forward and backward gradient learning across datasets and tasks. Due to the limited tuning performed, is possible obtain specific result marginally better than those presented. However, this does not undermine the comparision investigated in this work.
B.1 Vision Tasks
In vision benchmark, five common few-shot learning datasets are explored: CUB (EII), Omniglot (220), Cifar100 fs (l6)), minilmageNet (l40) and magee (3I)). Each dataset is split into three parts based on different non-overlapping sets of classes, for model training, validation, and testing. All recognition tasks across datasets are using 5-way 5-shot setting.
Table 6: The Syyer--paramete used in few-shot learning experiments for vision tasks. For fair comparisons, FF and BP are using the same hyper-parameter Model architectures of Resnet 88, modified Resnet12 and ViT tiny are based on [4] 443
Pre-trained models used for 34 and 38 Different learning rate grids are explored,
and 33
zero-shot evaluation can be found at [33 and the best accuracy is reported.
Experiment
Hyper-parameters
Values
n_way
5
s_ohot
le-3
FF, BP
Epochs
40
Optimizer
SGD
Learning rate
[le-3, Ic-4, lo-5)
Val/test tasks
100/
Total Memory Usage
Scratch Usage
100.00
60.00
80.00
79.75
50.00
40.00
60.00
47
:44.88
30.00
25.34
15.30
30,06
45 a5
(22.58
20.00
1.81 201175
22.58
22.47
25.40
25.13
20.00
10.00
11.43
0.456.40.40
0.55
0.00
0.00
VIT Tiny
Resnet18
FeatResnet12
Tiny
Resnet18
FeatResnet12
BP, FT, fp16
FT.
BP, LP,
BP, FT, 1p18
#FF, FT. fp16
WBP, LP, fp16
FF, LP, fp16
#FF, LP, Quant
FF, LP, fp16
#FF, LP, Quant
(a) Total Memory Usage (MB)
(b) Scratch Memory Usage (MB)
Figure 3: Comparison Memory Usage during Training. BP: backpropagation, FF; forward gradient
learning, fp16: 16-bit float point, Quant: 16w8a, FT; full fine-tuning, LP: linear probing.
Figure 3 shows the memory usage of and FF during the training. The total memory usage during training composed of two parts, scratch buffer used for input and output activation tensors for
14

---

gradient calculation and storage, and allocated memory for weights storage. Without storing the activation tensors, forward gradient learning has a sigmificant reduction on the scratch memory usage. For example, in the case of full fine-tuning on ViT Tiny network, under the same precision of fp16, FF reduces the scratch memory from 31. 14.64MB to 11.43MB (2.8x). When sparse update and fixed-point training are enabled, only 0.40MB of scratch memory is needed for model fine-tuning.
The extent of memory saving with FF depends on the number of layers being fine-tuned, and their positions within the network. When applied to methods such as full fine-tuning, LoRA (177) and other ceraameter-effciien fine-tuning approaches, FF shows significant memory reduction because eliminates the need to store intermediate activations. In the case of LP, where only the last few layers are updated, the difference of memory usage between BP and FF will get smaller. As the number of trainable layers increases, FF benefits more in memory savings. These promising results indicate that FF can perform comparably to BP with only a slight degradation in accuracy, while significantly reducing the memory cost. With the same memory footprint as inference, model training with FF is feasible on low memory devices where BP cannot be afforded.
B.2 Audio Tasks
In audio use cases, two few-shot audio classification benchmark datasets are selected: ESC-50 ((0) and FSDKaggle (Il.. Prior to adaptation, publicly available pretrained models based on AudioSet are adopted (lI). The averaged accuracy after 200 epochs over 10, 000 tasks drawn from the test set is reported.
Table 7: The paper-parrmeters used in few-shot learning experiments for audio tasks. Both datasets are using 5-way 1-shot setting. For fair comparisons, FF and BP are using the same hyper- parameters except that FF uses a smaller learning rate. Model architectures of CRNN and AST based on [15] and [13]. Pre-trained models used for zero-shot evaluation can be found at 15 and Different learning rate grids are explored, and the best accuracy is reported.
Experiment
Hyper parameters
Values
n_way
5
D_o
le-3
BP Epochs
200
Optimizer
SGD
Learning rate
(1e-4, 1e:5)
Valitest tasks
100% 10,000
Figure 4 compares the memory usage of BP and FF during the training. For small model such as CRNN, there at least 4x reduction in total memory when full fine-tuning is used. In the case of AST architecture, model training with quantized forward gradient combined with sparse update only requires 0. 19MB scratch memory, which fits into most existing edge devices.
Total Memory Usage (M)
Scratch Memory Usage (M)
500.00
250.00
215:00
400.00
170.84
300.00
150.00
770.9 18088170.78
200.00
100.00
50.00
12.03 2.062.712711:40
11.79 2.702.264661.28
0.19
0.00
0.00
CRNN
CRNN
BP, FT, Fp16
#FF, FT. 1p16
BP, LP, 1p
BP, FT, fp16
WFF, fp16 #BP, fp16
FF, LP, fp16
FFF, UP, Quant
FF, fp16
FF, LP, Quant
(a) Total Memory Usage (MB)
(b) Scratch Memory Usage (MB)
Figure 4: Comparison of Memory Usage during Training. BP: backpropagation, FF: forward gradient learning, fp16: 16-bit float point, Quant: 16w8a, FT: full fine-tuning, LP: linear probing.
15

---

C Cross-domain adaptation
Cross-domain adaptation is performed on VWW dataset. Table 8 lists all hyper-paramente used in training.
Table 8: The hyper-parameters used in experiments for cross-domain adaptation. All hyper- parameters for FF and BP are the same except that FF uses smaller learning rate. Model architectures of ViT tiny, and the associated pre-trained weights can be found at (3). Different learning rate grids are explored, and the best accuracy is reported.
Experiment
pareaeparameterss
Values
1e.3
Epochs Warmup epochs Optimizer
100
Adamw, betas: (0.9.0.95]
FF.
Learning rate Minimum learning rate Scheduler
[5e-3, le-3)
1e-5
cosine decay
Batch size
256
Weight decay
Irale ftsee, steoy LP
trate/kone jdep, D-UPT
www.
Thaining Stape
Thintig Stpe
(a) Training curves: L.
(b) Training curves: D-VPT
Figure 5: Training convergence curves. BP: backpropagation, FF: forward gradient learning, fp16:
16-bit float point, Quant: 16w8a, LP: linear probing, D-VPT: visual-prompt tuning with deep prompts.
Figure 5 shows the training curves of BP and FF under various settings. In general, FF requires smaller learning rate, resulting more training iterations to converge than BP. However, for a single iteration, BP performs one forward pass and one backward pass, while FF needs two forward passes. The FLOPs of a backward pass are ~ 2x of that of a forward pass (e.g., for both Convolutional and Linear layers). Therefore, FF has 1.5x speedup in one iteration the training. The total training time depends on the number of iterations required for model convergence and the time taken complete each iteration.
D In-domain OOD adaptation
Cifarl0-C provides 5 levels of corruption severity, from which we take the lowest (casy), middle (medium), and highest (hard) corruption severity separate benchmarks for fine-tuning, randomly partitioning each section into 90%-10% train-test split.
E
Empirical Studies, Discussions and Limitations
The convergence and generalization property of perturbation-basel learning, such as forward gradient learning, depends on the loss landscape instead of number of parameters. Visualization of loss landscape has the potential help us answer several important questions about how a neural network trained, and why do the resulting minima generalize under certain training approaches.
16

---

Table The paper-parameters used in our experiments for in in-domain OOD adaptation. All hyper- parameters for FF and BP are the same, except that FF uses smaller learning rate. Model architectures of ViT tiny, and the associated pre-trained weights can be found at Different learning rate grids are explored, and the best accuracy is reported.
Experiment
Hyper parameters
Values
Epochs Warmup epachs Optimizer
100
Adamw, betas: [0.9.0.95] (le-4, Se-5, le-5)
FF, BP
Learning rate
Minimum learning rate
1e.5 cosine decay
Scheduler
Batch size
Weight decay
Figure compares the 2D contour of loss landscape and loss trajectory during training under BP and QZO-FF. Both forward and backward learning shows locally smooth loss contour, and the trajectory follows the gradient descent direction, with forward gradient learning taking a more conservative step after each epoch, resulting in slower convergence. We also observed that good initialization (e.g.. pre-trained model) critical for forward gradient learning. Therefore, the convergence may not be guranteed if model trained from seratch. However, it is still promising that quantized forward gradients to be used for model adaptation on low resource devices, in which a general pre-trained model has been deployed.
In our experiments, is also observed that 8-bit quantization weights does not lead to model convergence. This is because the small perturbation of is quantized using the scaling factor of weights (Aw). It requires higher bits to be properly represented without clipping loss, thus the weights change can be reflected in the quantized space. In future, techniques for ultra low bit (i.e., 8-bit, 4-bit) forward gradient learning can be explored. In addition, experiments beyond classification and across multiple modalities be conducted for further evaluations.
17

---

20d 12:
Pa 68
PC: 62,82
(a) BP, LP, test error 12.13%.
(b) BP, D D-P, test error 7.02%.
200 PC 32.-2
(c) FF, LP, test error 12.49%.
(d) FF, D-VPT test etror 11.06%.
2nd PCP 9.29
(e) FF Quant (16w8a), I.P, 12.74%.
FF Quant (16w8a), D-VPT, 11.38%.
0.00
0.00
15t PC: 100.00
1st PC: 100.00%
(g) Quant (8w8a), LP, test error 51.35%, not con- (h) Quant (8w8a), D-VPT, test error 52.98%, not verged.
converged.
Figure 6: 2D visualization of loss landscape and loss trajectory during training. All hyper-yaramedte parameters used in this experiment is listed in Appendix D LP: linear probing, D-VPT: visual-prompt tuning with deep prompts. Both forward and backward learning shows locally smooth 2D loss contour, and the trajectory follows the gradient descent direction, with FF taking more conservative step after each epoch. It observed that 8-bit quantization weights does not lead to model convergence. Therefore, 16- bit weights quantization is necessary for QZO-FF.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Definitions cross-referenced. Equations and formulas derived in details.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Model architectures and pretrained checkpoints are all from publicly available sites, which are referenced. Experimental hyper-parameters all listed in Appendix.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We use public dataset, known model architectures, and publicly available pretrained model checkpoints for further experiments. Links are all provided, and cross- referenced. All experimental hyper-parameters are given in Appendix. We can also provide instructions to reproduce the results. However, we cannot open source the code.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All results are explained in text about how the accuracy is reported.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: All our experiments are based on single Nvidia Tesla V100 GPU. Since our experiments mainly focus the performance of quantized forward gradient learning, training computation is not the bottleneck. Most experiments the paper only requires few hours of training single GPU.
19

---

9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppweee.epp.....ooooccccaa
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: A foundamental training approach is discussed in the paper for potential model personalization in the future. This aims bring model adaptation local devices, thus protect users' privacy. We do not see any negative societal impacts of this work.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We conduct experiments commonly used public dataset.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
20


Reviews:
[{"review_id": "WwAGaeMsnc", "reviewer": "Reviewer_wcC5", "summary": "The paper proposes a quantization approach for fine-tuning pretrained data to new local data on resource-constrained devices. In particular, the weights perturbation, gradients estimation, and weights updates are quantized to either 8-bit or 16-bit. This quantization approach is combined with Momentum Guided Sampling, Sharpness-aware Perturbation, Sparse Update, and Kernel-wise Normalization to enhance fine-tuning performance. The proposed approaches are evaluated on various AI benchmarks. The results of this study indicate that quantized forward gradients are a good candidate for a fine-tuning approach that can be deployed on edge devices.", "questions": "Why is the random perturbation vector z sampled from a normal distribution with zero mean and standard deviation? Is it possible to sample from a log-normal distribution since activation gradients are shown to be distributed near log-normal [1]?\n\n[1] Chmiel, Brian, et al. \"Neural gradients are near-lognormal: improved quantized and sparse training.\" arXiv preprint arXiv:2006.08173 (2020).", "limitations": "The author addresses the limitations of this study by mentioning the initialization requirements for forward gradients and 16-bit weight quantization.\n\n1- It would be suggested to discuss various initialization approaches that might be used instead of a pretrained network.\n2- It would be suggested to discuss other numerical formats, such as 8-bit floating point or Posit, to solve the 16-bit weight quantization problem.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- The paper is well-written and well-organized.\n\n2- The quantized approach is evaluated on a variety of tasks that show the generalizability of the new approach.\n\n3- The Sign-m-SPSA-SGD approach is interesting and novel.", "weaknesses": "1- The author are recommended to discuss the accuracy degradation of quantized forward gradients compared to the backpropagation algorithm. In some cases, the accuracy degradation is high (more than 5%). A comparison of performance versus hardware complexity (FLOPs or another metric) is recommended, as seen in [1].\n\n2- Evaluating the efficacy of quantized forward gradients on fine-tuning LLM models such as LLaMA-3 is recommended. \n\n[1] Carmichael, Zachariah, et al. \"Performance-efficiency trade-off of low-precision numerical formats in deep neural networks.\" Proceedings of the conference for next generation arithmetic 2019. 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewers for their careful review of our paper, their interest in the core idea of our work, and valuable suggestions. The comments regarding accuracy discussions and comparisons of hardware complexity are well taken, and the manuscript will be revised accordingly. Our responses to the questions and identified weaknesses are given as follows.  \n\n**Question 1 (sampling of $z$):**\n\nThe choice of sampling $z$ from a normal distribution with zero mean and unit variance is supported by the literature [Baydin et al. [2022], Section 3.2], which proved that the forward gradient is an unbiased estimator of the true gradient, when the scalar components of perturbation are independent, and follow a zero mean, unit variance distribution. \n\nIn our case, Sign-m-SPSA is also designed to be compatible with quantization. It constrains the range of gradient values to be the same as perturbation $z$ for static quantization. This maintains consistency in gradient estimation within the quantized space.\n\nIn addition to using a normal distribution, we explored using a binomial distribution from a quantization-friendly perspective. Our experiment, as shown in Figure 2(a), indicates that the binomial distribution is also effective. Sampling from other distributions (e.g., log normal distribution) is another interesting direction. We tested this variation on our cross-domain adaptation setup with fp16 precision, and the results were promising. The quantization impact of such distribution needs to be further investigated.  \n\n**Weakness 1 (accuracy and hardware complexity):**\n\nWe appreciate your recommendations and feedback. We will include a more detailed accuracy analysis, and list the memory cost and FLOPs of all the models used for backpropagation (BP) and forward-forward (FF) in the updated manuscript. Specifically,\n\n__1. Accuracy Discussion:__\n\nSince FF solely utilizes directional derivatives for gradient estimation, it is expected that BP generally outperforms FF in terms of accuracy in most tasks. The accuracy gap between BP and FF can vary based on factors such as the backbone architecture, dataset, and task difficulty. We observed that accuracy gap tends to increase on more challenging tasks. However, using a stronger backbone such as ViT, can help bridge this gap. This indicates that while FF may show more degradation with smaller architectures and low-resolution inputs, performance improvements can be achieved with more advanced models.\n\n__2. Hardware Complexity:__\n\nWe compare BP and FF from two perspectives:\n\n* __Memory efficiency__: BP needs memory to store model parameters, all the intermediate activations and gradients, whereas FF avoids storing intermediate activations, the size of which could be considerably large in many models. \n\n* __Computation complexity__: For a single iteration, BP performs one forward pass and one backward pass, while FF needs two forward passes. The FLOPs of a backward pass are roughly 2x of that of a forward pass (e.g., for both Convolutional and Linear layers). In our experiments, we observed a 1.5x speedup in one iteration of the training. However, the total computation (or training time) depends on the number of iterations required for the training to be converged. \n\n**Weakness 2 (QZO-FF for LLM models):**\n\nWe would like to address this comment in the common rebuttal section. \n\n**Limitation discussion 1 (initialization):**\n\nOur work primarily focuses on model fine-tuning on edge devices with fixed-point engines, where we assume that the pre-trained network provides a good initialization. To facilitate reproducibility, we use widely available open-source pre-trained networks as backbones. In our cross-domain adaptation experiments, we initialize the decoder layers randomly to ensure a consistent and straightforward experimental setup. We have added an accuracy report with mean and standard deviations across 5 runs to our ablation studies in Section 4.2. \n\n**Limitation discussion 2 (numerical formats):**\n\nThank you for the valuable suggestions. Based on our extensive investigations, 16-bit weight quantization is crucial for accurately capturing perturbations and accumulating weight changes in FF. However, we recognize the potential benefits of exploring alternative numerical formats such as 8-bit floating point, or ultra-low bit formats, in forward gradient learning. \n\nIn future work, we plan to explore these numerical formats to address challenges associated with 16-bit weight quantization. Currently, support for such formats is limited on most fixed-point engines.\n\nWe will include additional discussions in the updated manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KfagMIs5Dq", "reviewer": "Reviewer_oWX1", "summary": "The authors investigate fixed-point forward gradients for quantized training. They conduct experiments across various deep learning tasks in vision and audio to assess if this method yields competitive models while conserving memory and computational resources.\nThey introduce algorithm enhancements to reduce memory usage and accuracy gaps compared to backpropagation, using fixed-point precision for forward gradients during training or adaptation.\nTheir findings demonstrate the feasibility of on-device training with fixed-point forward gradients across diverse model architectures (e.g., CNN, RNN, ViT-based) and parameter sizes (100K to 80M), offering practical solutions for model adaptation on edge devices.\nThe authors also visualize neural loss landscapes and training trajectories, providing insights into the dynamics of training with forward gradients for efficient on-device model adaptation.", "questions": "1. why you loop over w can’t you just do it vector wise?\n\n2. Can you specify m (the number of pertubations) used for each experiment?\n\n3. Can you calculate the memory consumption (MB/GB)  and computation complexity (in FLops) compared to QLoRA with BP.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1 .They understand quantization and tried not to leave anything float\n\n2. Experimenting with SAM and ZO is nice \n\n3. The paper is well written", "weaknesses": "1. Sadly no experiments on LLMs on which most fine tuning is done today\n\n2. Marginal novelty: generally they just added quantization to ZO-FF – is that enough?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewers for their careful review of our paper, their interest in the core idea of our work, and valuable suggestions. The comments regarding the motivation, novelty, impact of our work, and detailed comparisons of hardware complexity are well taken, and the manuscript will be revised accordingly. Our responses to the questions are given as follows. \n\n**Question 1 (loop over $w$):**\n\nThe loop of $w$ pertains to performing updates on a per-tensor basis. Each individual tensor $w_i$ is the trainable parameters for each layer of the model. It is processed in a vectorized manner, and is associated with its own quantization scaling factor. We will clarify this notation in the algorithm description to ensure a better understanding of the process.\n\n**Question 2 (specify of $m$):**\n\nIn our experiments, the number of forward-forward calls performed ($m$) for averaging gradients is set to $3$, unless otherwise specified in the ablation studies. Details on this parameter are provided in the appendix.  \n\n**Question 3 (memory consumption and computation complexity):**\n\nWe appreciate your recommendations and feedback. We will list the memory cost and FLOPs for all models used in backpropagation (BP) and forward-forward (FF) process in the updated manuscript. Specifically,\n\nWe compare BP and FF from two perspectives:\n\n__1. Memory efficiency:__ The extent of memory saving with FF depends on the number of layers being fine-tuned, and their positions within the network. When applied to methods such as Full Fine-tuning, LoRA and other Parameter-Efficient Fine-tuning (PEFT) approaches, FF shows significant memory reduction because it eliminates the need to store intermediate activations.\n\n__2. Computation complexity:__ For a single iteration, BP performs one forward pass and one backward pass, while FF needs two forward passes. The FLOPs of a backward pass are roughly 2x of that of the forward pass (e.g., for both Convolutional and Linear layers). In our experiments, we observed a 1.5x speedup in one iteration of the training. However, the total computation (or training time) depends on the number of iterations required for the training to be converged.\n\n**Weakness 1 (QZO-FF for LLM models):**\n\nWe would like to address this comment in the common rebuttal section.\n\n**Weakness 2 (novelty):**\n\nMost existing neural processors on edge devices are optimized as efficient fixed-point inference engines. We believe continuously adapting pre-trained models to local data on the edge is crucial for effective model deployment. \n\nTo enable training on edge devices with constrained memory, we leverage low-bit precision techniques. While previous work by [J. Lin, 2022], introduced Quantization-Aware Scaling for BP, to mitigate accuracy loss due to quantization, BP is memory intensive, and unsupported by many existing inference engines. In contrast, ZO-FF shows a significant advantage in this context. \n\nTo our knowledge, there is no prior research demonstrating the feasibility and impact of quantized ZO-FF, particularly with regard to different bit-widths and their effects on performances. Our work addresses this gap by showing the effectiveness of quantized ZO-FF through extensive experiments across various deep learning benchmarks and modalities. We believe this contribution is of substantial interest to the industry, addressing both practical challenges and opportunities in deploying models on edge devices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BhPlMQicsC", "reviewer": "Reviewer_vPS7", "summary": "This paper explores the feasibility of on-device training using fixed-point forward gradients. The authors propose methods including sign-m-SPSA, Momentum Guided Sampling, Sharpness-aware Perturbation, Sparse Update, and Kernel-wise Normalization to reduce memory footprint and accuracy gaps and conduct experiments across various deep learning tasks in vision and audio domains. Key contributions of this paper include formulating forward gradients in the quantized space, demonstrating the feasibility of on-device training, and visualizing the neural loss landscape during training. The study shows that training with fixed-point forward gradients might be a practical approach for model customization on edge devices.", "questions": "1. Although one can understand the meaning of these symbols after a careful reading, the notation in equation (6) is somewhat confusing because $1_q$ and $\\epsilon_q$ have the same subscript but different scaling factors. I think it would be better to add a notation related to the scaling factor above them.\n\n2. typo in line 271: extenteded", "limitations": "The limitations are discussed well by the authors.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "++ This paper proposes an improved method for forward gradients, called Quantized Zeroth-order Forward Gradient (QZO-FF), which enables forward gradients training using quantization. \n\n++ QZO-FF is quantized and does not require backpropagation, thereby reducing memory overhead and eliminating the need for processors to have training capabilities. However, I doubt this because even though forward gradients do not require backpropagation, they still need to update weights and possibly save momentum and they need to perform additional quantization for $z$ in QZO-FF. Therefore, we may need some hardware adaption to assist feed-forward training.\n\n++ The experiments across various benchmarks show that there is only a slight degradation in accuracy while the memory cost is reduced.", "weaknesses": "1. Some results are missing in the experiment. For example, (1) the memory cost of (BP, LP, fp16) is not measured. I think the memory cost of LP is important because it seems that the reduction of memory cost mainly comes from LP instead of FF and Quant in Figure 3 and Figure 4, and I think the claim that \"this number is further reduced to only 0.28MB\" and \"the saving increases to 8.1× when sparse update and fixed-point are enabled\" in Appendix B is totally misleading and unfair. (2) The accuracy of (BP, LP, quant) is not measured so there is no baseline for (FF, LP, Quant). (3) The accuracy of (FF, FT, Quant) and (BP, FT, Quant) is not measured. (BP, FT, Quant) should be some BP fixed-point training methods like Quantization-Aware Scaling (QAS) mentioned in related work.\n\n2. Lack of ablation studies. The effects of techniques proposed in Section 3.4 are not well-studied. (1) There is no ablation study for Section 4.1. (2) The effect of sharpness-aware and kernel-wise normalization is not measured separately in Section 4.2. (3) I want to know __which__ of these techniques work in __what__ experiment settings. I believe that, as a new algorithm with many enhancement techniques, the authors should inform the readers about which parts of the algorithm are useful under which circumstances.\n\n3. The model size (100K - 80M) is somewhat small compared to the concept of \"pretrained models\". How does the proposed method perform for larger models and how does the model size affect the effectiveness of the method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewers for their careful review of our paper, their interest in the core idea of our work, and valuable suggestions. The comments regarding more detailed explanations of techniques, experimental setups and comparisons are well taken, and the manuscript will be revised accordingly. Our responses to the questions are given as follows. \n\n**Question 1 and 2 (notation in equation 6, typo in line 271):**\n\nThank you for pointing this out.\n\nThe subscript $q$ refers to the quantized value. In this context, $1_q$ and $\\epsilon_q$ each have their own scaling factors ($\\Delta_z$ and $\\Delta_w$) and bit-widths. \n\nWe will correct the typo, and revise the notation to clearly distinguish these elements. \n\n**Weakness 1 (some missing results):**\n\n__1.\tThe memory cost of (BP, LP, fp16):__\n\nThank you for your feedback. We understand the importance of accurately representing the memory costs associated with different training methods. The extent of memory saving with FF depends on the number of layers being fine-tuned, and their positions within the network. When applied to methods such as Full Fine-tuning (FT), LoRA and other Parameter-Efficient Fine-tuning (PEFT) approaches, FF shows significant memory reduction because it eliminates the need to store intermediate activations. This is evident when comparing the memory usage of (FF, FT) vs (BP, FT). In contrast, for Linear Probing (LP), where only the last layers are updated, the memory savings are less. Typically, finetuning with LP results in lower accuracy compared to FT or LoRA.  \n\nWe will revise our statements and ensure that the claims are fair and well-supported. \n\n__2.\tThe accuracy of (BP, LP, Quant) and (BP, FT, Quant):__\n\nWe use (BP, LP, fp16) as a higher accuracy baseline to compare with (FF, LP, Quant). With techniques such as Quantization-Aware Scaling (QAS) reported in the literature, we expect (BP, LP, Quant) to be close of (BP, LP, fp16) in terms of accuracy. Similarly, we expect that (BP, FT, Quant) to perform comparably to (BP, FT, fp16). Currently, support for Quant formats with BP is limited on most fixed-point engines.\n\n**Weakness 2 (Ablation studies):**\n\n__1.\tAblation study for Section 4.1__\n\nWe conducted ablation studies in Section 4.2 to evaluate the effectiveness of quantized FF, the impact of bit-width variations, and different perturbation sampling strategies. This section provides a more controlled experimental setting for these analyses. \n\nIn Section 4.1, we focused on evaluation few-shot learning across a diverse set of benchmark datasets, tasks, backbones, and modalities. This broader approach aims to demonstrate the general applicability of our method across various scenarios, while the more detailed ablation studies in Section 4.2 address specific aspects of our approach.\n\n__2.\tThe measure of sharpness-aware and kernel-wise normalization:__\n\nIn our cross-domain adaptation setup, we observed a consistent, but only a modest increase in accuracy with the combination of sharpness-aware and kernel-wise normalization. \n\n__3.\tVarious techniques and experiment settings:__\n\nThank you for raising this important point. We would like to address this comment in the common rebuttal section, providing a detailed explanation of how each technique performs. We will include a discussion of each technique and its effectiveness across different experimental settings in the updated manuscript, to guide readers on the optimal use of these enhancements.\n\n**Weakness 3 (QZO-FF for larger models):**\n\nThank you for the suggestions. We will address this comment in the common rebuttal section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TbpBmqCCXq", "reviewer": "Reviewer_uc7y", "summary": "**Context**. The focus of the present paper is on-device fine-tuning (gradient computation and weight update **starting from a pre-trained model**) under limited memory budget. One way to cut the memory cost of storing the computational graph for gradient computation by standard backprop is the Memory Efficient Zeroth Order (MeZO) optimizer [Malladi et al, 2023], whereby a directional gradient is computed via weight perturbation (a.k.a SPSA): computing the loss $L$ difference yielded by two forward passes with weights differing by $\\epsilon u$ estimates $\\nabla L \\cdot u$. Since it is a purely forward procedure, it obviates the need to cache activations to execute a backward pass.\n\n**Core contribution**. The present paper proposes a quantized version of MeZO where weight perturbation, gradient computation and weight update are carried out on quantized quantities. The proposed algorithm, coined QZO-FF (Alg. 1), is tested against a variety of fine-tuning tasks (few-shot learning, cross-domain and in-domain adaptation), modalities (image and audio data), and architectures (convolutional, attention-based, recurrent), with several variants being explored (with fp8 / fp32 activations) and benchmarked against standard backprop. The efficiency of QZO-FF, both in terms of resulting performance and memory usage, is demonstrated.\n\n**Paper outline**. More precisely:\n- Section 2 provides background knowledge on memory-efficient backprop (2.1), forward-mode differentiation (2.2) and quantized training (2.3).\n- Section 3.1 and 3.2 formalizes further \"forward gradients\" (3.1)  and the SPSA / weight perturbation procedure to estimate them (3.2). A hardware-friendly extension of SPSA coined as \"Sign-m-SPSA\", which estimates $\\text{sign}(\\nabla L \\cdot u) u$, , is introduced along with the resulting SGD update (3.2).\n- Section 3.3 presents the core algorithmic contribution by combining SPSA / weight perturbation and weight quantization (Alg. 1). More precisely, weights and perturbation are statically, symmetrically quantized (e.g. their range are estimated and set once, before fine-tuning), with one scale for each ($\\Delta_w$ and $\\Delta_q$). Therefore: i) $\\Delta_w$ and $\\Delta_q$ are fixed, with weights and perturbations quantized with 16 and 8 bits respectively, ii) the integer part of the perturbed weights is accumulated in 32 bit, iii) the dequantized perturbed weights is quantized-dequantized back into 16 bits using the same $\\Delta_w$ scale (Eq. 6). The Sign-m-SPSA gradient estimator is applied and quantized-dequantized using the perturbation scale ($\\Delta_z$, Eq. 7) . Finally, the weight update itself is quantized, such that it happens in the quantized integer part and is rescaled by $\\Delta_w$ (Eq. 8). Alg. 1 summarizes the procedure in the case where the number of perturbed directions at use is 1 ($m=1$).\n- Section 3.4 presents several algorithmic \"enhancements\" of the QZO-FF algorithm to improve the optimization procedure itself or its memory footprint.\n- Section 4 presents experimental results. First, few-shot learning is considered (4.1) on visual and audio data. Here, \"FF\" refers for short to \"QZO-FF\". A quantized version of FF, where 8 bits activations are used, is also tested. On vision, three architectures are tested (ResNet12, 18 and ViT tiny) on 5 different standard few-short learning datasets. Two scenarii are considered: full fine-tuning and linear probing. It is shown overall that FF always yields better performance than the zero-shot baseline and within 5%, accuracy-wise, to the BP baseline on 26/30 experiments, and that the ViT backbone yields the least degradation. On audio, a similar experiment is done with two architectures (CRNN, AST) on two audio datasets. On 11/16 experiments, FF accuracy is 5% off the BP baseline. Then, a cross-domain adaptation task (4.2) is considered, where the different algorithmic enhancements previously introduced (e.g. quantized FF, gradient averaging, \"sharpness aware\" scheme...) are tested. Most importantly, it is observed that quantizing weights to 8 bits jeopardize the FF algorithm. Finally, sector, 4.3 presents in-domain OOD adaptation using the same fine-tuning schemes (LP, D-VPT) with three levels of corruptions of the CIFAR-10 dataset as OOD datasets. In this setting, FF achieves comparable performance with BP.", "questions": "- L. 135: \"in order to mitigate the noisy component of forward gradients estimated by SPSA, we propose sign-m-SPSA\": do you have evidence that sign-m-SPSA results in less noisy gradients? \n- Section 3.3: could you please clarify what is kept in full precision? I see at least three different quantities not being quantized: i) the scales $\\Delta_w$ and $\\Delta_z$,  ii) the loss for each perturbed weights and therefore its difference, iii) the averaged gradient (Eq. 7). Most importantly, do you confirm that you need to accumulate gradients across each direction ($i=1 \\cdots m$) in higher precision (32 bits I guess?) and then quantize-dequantize it using the scale of the perturbation $\\Delta_z$? Your pseudo-algorithm only treats the case $m=1$ so it remains unclear how this all work when $m>1$ and you need to average gradients. **Could you please write a new pseudo-algorithm**, alike Alg.1, **in the case $m>1$**, highlighting with **two different color codes** the quantized (int8 and int16) and full precision (fp32) quantities?\n- L.193-198 (momentum-guided sampling): I think that incorporating momentum into your approach is crucial. However, I don't understand neither how it works. What do you mean by \"as training progresses, a history of the momentum $z$ is incorporated to guide the new sampling process\"?\n- L.199 (sharpness-aware perturbation): you mean an \"extra step of **directional** gradient ascent\"?\n- L.204-210 (sparse updates): which sparsity scheme did you employ? top-k magnitude based scheme may be quite costly, if it boils down to ranking all the weights by their magnitude.\n- L.211 (kernel-wise normalization): in this case we agree that $\\hat{g}$ needs to be stored in full precision? Also, the computation of the norms of $z$ and $w$ is computationally expensive ($O(d)$, where $d$ denotes the dimension of $w$ or $z$), as expensive as it would be to dynamically recompute $\\Delta_z$ and $\\Delta_w$, which you avoided by statically quantizing them. Don't you lose the advantage here of using static scales if in any case you need to perform this $O(d)$ operations?\n- L.216 (few-shot learning): \"a few labeled samples are available\", but how many? Could you please clarify the experimental setup?\n- L.222: \"16w8a\" means 16 bits for weights and 8 for activations, correct? I may not take this for granted and clearly define this notation.\n- Table 2: I would rather compute the **relative** accuracy degradation (acc_BP - acc_qFF / acc_BP) rather than the **absolute** accuracy degradation.\n- Table 2: the accuracy degradation when employing FF in the FT setting compared to BP in the same setting is quite severe (11.08% gap), although you are using a relatively small architecture (ResNet12) with relatively small input dimensionality (32x32). Why is this the case?\n- Table 2: **there are not any error bars**, which makes it hard to make any sense of a $\\sim 0.2/0.5$ difference between two experiments.\n- Could you please define precisely what you mean by \"zero shot\" (I assume no training at all?), \"linear probing\" (I assume only the last linear layer is learned?) and \"full fine-tuning\" (all parameters are learned)? \n- L. 238 (audio benchmark): could you please detail the few-shot setup for this task, and the tasks themselves? It is important for people not familiar with this literature.\n- L. 252: I really did not understand what \"cross-domain adaptation\" really is about. Could you please explain better what it is? \n- L.256: what is \"visual-prompt tuning with deep prompts\"?\n- Fig. 2: except for large discrepancies between bars, it is difficult to draw any conclusion from this figure **for lack of error bars**. Could you please add them?\n- Which conclusions / insights do you really gain from plotting the 2D contours of the loss landscape in the different settings?", "limitations": "See weaknesses and questions above.\n\nIf I had detailed description of each of the experimental setups tackled, a precise knowledge of what exactly is kept in full precision, how some of the algorithmic enhancements really work and error bars on all figures, I would be prone to increasing my score. I really want to encourage the authors to do so because I do believe that the core of this work is of interest.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem tackled is highly relevant to on-device training, pragmatic and builds upon recent work [Malladi et al, 2023].\n- The proposed algorithm is sound and well-explained.\n- There are a lot of experimental settings, data modalities and architectures being explored.\n- The proposed technique is effective in providing a learning signal, effectively training models and yielding relatively good performance compared to the BP baseline.", "weaknesses": "- It is unclear what is kept in full precision in the proposed procedure (see my questions below).\n- On a related note, it is also unclear that the proposed algorithm enhancements don't offset the advantages of manipulating statically quantized quantities (see my questions below). \n- The experiments aren't all sufficiently well explained, neither in the main nor in the appendix, which is frustrating because there is a lot of work done there and we fail to deeply understand the proposed setups. I would even say that there are almost too many different experimental setups. Under constrained time budget to write the paper, I would have prioritized a lesser number of better detailed experiments rather than a lot of them left unsufficiently explained.\n- **There aren't any error bars in any table and figures**, although the authors ticked in their checklist that they reported error bars and provided appropriate information about the statistical significance of their experiments (L. 520). For lack of this, it is very hard to draw any clear conclusion in terms of comparison between the different algorithms at use, e.g. is there a statistically significant gap here, or are these two results within error bar? We don't know.\n- I don't understand what the 2D plot of the loss landscape really brings here in terms of insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewers for their careful review of our paper, their interest in the core idea of our work, and valuable suggestions. The comments regarding the detailed experimental setups, precision used in the algorithm, explanations of enhancement mechanisms and notations are well taken, and the manuscript will be revised accordingly. Given the large number of experiments across multiple benchmark datasets, backbones, finetuning methods, and different modalities, we need additional time to report the error bar metrics. However, we will include error bars across as many settings as possible in the final paper. Our responses to the questions are given as follows.\n\n**Question 1 (L. 135):**\n\nThe design of sign-m-SPSA addresses gradient noise in two key ways. \n\n__1.  Sign-based optimizers:__ Sign-based optimizers, such as sign-SGD (Bernstein et al. [2018]), have shown good practical performance for gradient compression. In our case, the magnitude of loss changes due to perturbed weights can be quite noisy. Therefore, incorporating the sign(.) operation in m-SPSA largely reduces the noise and enhances training stability. \n\n__2. Quantization-friendly:__ Sign-m-SPSA is designed to be compatible with quantization. It constrains the range of gradient values to be the same as perturbation z for static quantization. This maintains consistency in gradient estimation within the quantized space.\n\n**Question 2 (precision):**\n\nWe have updated the pseudo-algorithm with gradient averaging, and highlighted different precisions with color codes. Specifically,\n\n__1. Quantization scaling factors:__ The scaling factors $\\Delta_w$ and $\\Delta_z$ are floating-point values, indicating the minimum representation power of the quantization space. In fixed-point engines, these scaling values are approximated using a multiplication ($\\times m$) and a right shift ($\\gg k$) operations in the re-quantization stage through a post-processing hardware block, as detailed in Appendix A. For instance, $\\Delta_z \\approx \\frac{m}{2^k} $. \n\n__2. Loss calculation:__ The loss can be computed either in floating-point precision or in quantized space, depending on the implementation and hardware support. Only the sign of the loss difference is necessary for gradient computation. \n\n__3. Gradient accumulation:__ When m>1, higher precision is used to accumulate gradients, followed by a re-quantization step. For example, if $m = 4$, we require at least 2 additional bits to store the intermediate values. Techniques such as right-shifting the gradients ($g \\gg 2$) can be employed to manage memory usage while still allowing accurate accumulation. There is a trade-off between memory efficiency and accuracy. \n\n**Question 3 - 6 (enhancement techniques):**\n\nWe would like to address various enhancement techniques in the common rebuttal section. \n\n**Question 7 (few-shot learning):**\n\nWe use a 5-way 5-shot setting (5 labeled samples) for vision tasks and a 5-way-1-shot setting (1 labeled sample) for audio tasks. \n\n**Question 8 (16w8a):**\n\nYes, “16w8a” denotes 16-bit weights and 8-bit activations. We will clarify the term.\n\n**Question 9 (relative metric):**\n\nThanks for the valuable suggestion. We agree that relative accuracy degradation is a more informative metric for assessing the impact of the change, while the absolute values intuitively highlight the performance gap. \n\n**Question 10 (accuracy degradation):**\n\nThe accuracy gap between BP and FF can vary based on factors such as backbone architecture, dataset and task difficulty. For instance, with the CIFAR-100 dataset, which only contains low resolution images (32x32), FF faces some challenge. Using a stronger backbone such as ViT, can help bridge this accuracy gap. This indicates that while FF may show more degradation with smaller architectures and low-resolution inputs, performance improvements can be achieved with more advanced models. \n\n**Question 11 and 16 (error bars):**\n\nIt is generally expected that BP outperforms FF in terms of accuracy in most tasks. However, our goal is to narrow this gap while leveraging the memory benefits of FF. We acknowledge the importance of including error bars. We will include error bars across as many settings as possible in the updated manuscript.\n\n**Question 12 (terms of training methods):**\n\nAll terms are correct. We will clearly define these terms to avoid any ambiguity.\n\n**Question 13 (L. 238):**\n\nWe will provide more detailed task information in the audio benchmark section to ensure clarity for readers who may not be familiar with this literature.\n\n**Question 14 (L. 252):**\n\n1.\t\"Cross-domain\" refers to fine-tuning on tasks with data distribution significantly different from those of the pre-trained model. For example, a model pre-trained on the ImageNet might be adapted to perform tasks on the VWW dataset. \n2.\t\"In-domain\", on the other hand, refer to fine-tuning a model on tasks with data distributions more closely related to the pre-trained data, but with some variations. For example, a model pre-trained on CIFAR-100 might be adapted to handle corrupted versions of the same dataset. \n\n**Question 15 (VPT-deep):**\n\nThis refers to VPT-deep, a fine-tuning method for Transformer models, as introduced by (Jia et al. [2022]). \n\n**Question 17 (2D contours):**\n\nPlotting the 2D contours of the loss landscape provides valuable insights into the training dynamics of different methods. These plots visualize how the loss evolves over training epochs and how the optimization paths differ. \n\nFrom the contours, we observe that both FF and BP exhibit locally smooth loss landscapes, with trajectories generally following the gradient decent direction. However, compared to BP, FF tends to take more conservative steps at the beginning of the training, resulting in slower convergence. Despite this, both methods converge to a local minimum after 100 epochs, indicating that FF, while slower, ultimately reaches a comparable solution to BP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude to the reviewers for their careful review of our paper, and their interest in the core idea of our work. We appreciate all the feedback, valuable suggestions and recommendations. The comments regarding notations, technical discussions, experimental clarifications are well taken, and the manuscript will be revised accordingly. Our responses to each reviewer’s questions are submitted separately. Additionally, we would like to address a few comments as follows. 

**Suggestion of evaluating QZO-FF on LLM models:**

Thank you for the valuable suggestion. Our paper currently focusses on the feasibility and the effectiveness of quantized ZO optimization for fine-tuning smaller models on edge devices with fixed-point engines. We have deployed our method on memory-constrained edge devices, and brought the training capability to fixed-point engines.

While our current work centers on these smaller models (ConvNets, and Transformer models), we recognize the potential application of our approach to larger LLM models such as LLaMA-3. Techniques Like LoRA and other Parameter-Efficient Fine-Tuning (PEFT) methods could be combined with our quantized ZO approach for LLMs. Recent literature, such as the MeZO work, indicates that ZO training can be effective across various LLM tasks. However, the impact of quantized ZO with low precision for LLMs remains an open question. We plan to extend our research to evaluate the performances of our quantized ZO approach on LLM models and various benchmark tasks in future work.

**Clarification of various enhancement techniques:**

These enhancement techniques are optional, and often involve trade-offs between memory, computation and accuracy, depending on the hardware memory budget. We will provide a more detailed discussion of each technique and its effectiveness across different experimental settings in the revised manuscript, to guide readers on the optimal use of these enhancements. Specifically,

__1.	Momentum-guided sampling:__ 

This enhancement introduces memory overhead to increase the accuracy. Instead of sampling solely from a zero-centered Gaussian distribution, perturbations are computed from a combination of a momentum-centered and a zero-centered Gaussian distribution. Mathematically, $z_1 \sim \mathbb{N}(0, \mathbb{I}_n* \sqrt{\alpha})$, $z_2 \sim \mathbb{N}(z_t, \mathbb{I}_n* \sqrt{1-\alpha})$, and 
$z _{t+1} = \beta*z_1 + (1-\beta)*z_2$. Here, $\beta$ is a smoothing parameter; $\alpha$ and $\beta$ can be adaptively adjusted during training. While $\beta=1$ corresponds to the baseline version without momentum-guided sampling, this approach enhances performance by improving perturbation quality through sampling history.  

__2.	Sharpness-aware perturbation:__

This technique improves the forward gradient direction by performing an addition step of directional gradient ascent, targeting regions where the loss curve is steeper. We observed that incorporating sharpness-aware optimization enhances the overall performance of QZO-FF across various experiments.

__3.	Sparse updates:__

We base our sparsity update approach on recent work by (Chen et al. [2024]), which utilizes a zero-order sparsity method. We also experimented with a random sparsity scheme, which proved effective as well. Due to the intrinsic properties of ZO method, reducing the number of parameters updated per iteration generally leads to improved accuracy.

__4.	kernel-wise normalization:__

The primary motivation to incorporate static quantization is to ensure compatibility with hardware support. Many existing fixed-point neural processors on edge devices only support static graph quantization, where weights and activations are quantized prior to compilation. Dynamic quantization, which involves recalculating quantization parameters during runtime, is computationally expensive, and often not supported efficiently.

For kernel-wise normalization, obtaining the norm of weights involves a trade-off between computation and accuracy. However, efficient implementations using GEMM and SQRT operations can minimize the overhead on hardware. 

**Figures:**

We have updated our pseudo-algorithm to include gradient averaging when $m>1$, and highlighted different precisions with color codes. Please noted that the scaling factor is approximated through a multiplication and a right shift operation on fixed-point processors (Appendix A). An accuracy report with mean and standard deviations has been added to our ablation studies in Section 4.2. Please refer to attached PDF for the updates.

Meta-review: We received author rebuttals, all reviewers except Reviewer wcC5 acknowledged the rebuttals and left follow-up comments.
They also participated in AC-reviewer discussions and shared their final thoughts. 

In summary, a key concern from the reviewers is limited novelty e.g.("added quantization to ZO-FF" by Reviewer oWX1), which I partially agree to. However, I found that their claimed novelty is of the proposed method. While novelty of the method is often discussed, I believe that novelty of work is more important. This work not only introduces a new proposed algorithm but also performs comprehensive experiments, exploring experimental settings, data modalities and architectures. I appreciate this effort as it should be critical to cover various datasets / modalities (vision and audio) / models in experiments to support the authors' claims.  Reviewer uc7y values the contribution as well.

Reviewers oWX1 and wcC5 suggested lack of (L)LM evaluations as a weakness of this work, but I disagree to the assessment. Since this work is focused on on-device training, training even part of (L)LMs may be technically impossible for relatively weak devices with respect to cloud instances.

Overall, this paper introduces an interesting, practical approach and demonstrates the potential of the approach through various experiments. I recommend accepting this paper for NeurIPS'24.
Last but not least, Reviewer vPS7 pointed out as a weakness that model size in this work is relatively small (100K - 80M) for "pretrained models", which I find fair and not well addressed by the general author rebuttal. The weakness point itself seems not critical enough to reject this paper, but I think it is an important discussio for future work n as the model size has been increasing.
Decision: Accept (poster)