Accelerated On-Device Forward Neural Network Training with Module-Wise Descending Asynchronism
Authors: Xiaohan Zhao
Hualin Zhang
Zhouyuan Huo
Bin Gu
Venue: NeurIPS 2023 poster (2023)
Keywords: asynchronous algorithm
one-device learning
forward gradient descent
directional derivative
forward algorithms

Abstract: On-device learning faces memory constraints when optimizing or fine-tuning on edge devices with limited resources. Current techniques for training deep models on edge devices rely heavily on backpropagation. However, its high memory usage calls for a reassessment of its dominance.
In this paper, we propose forward gradient descent (FGD) as a potential solution to overcome the memory capacity limitation in on-device learning. However, FGD's dependencies across layers hinder parallel computation and can lead to inefficient resource utilization.
To mitigate this limitation, we propose AsyncFGD, an asynchronous framework that decouples dependencies, utilizes module-wise stale parameters, and maximizes parallel computation. We demonstrate its convergence to critical points through rigorous theoretical analysis.
Empirical evaluations conducted on NVIDIA's AGX Orin, a popular embedded device,  show that AsyncFGD reduces memory consumption and enhances hardware efficiency, offering a novel approach to on-device learning.

Accelerated On-Device Forward Neural Network
Training with Module-Wise Descending Asynchronism
Xiaohan Zhao' xiaotuzxh@gmall.. com
Hualin Zhang inanghualli960gmaaae com
Zhouyuan Huo" huozhouyuan@gmail. com
Bin 2u jsbugin@gmail.. com com
1 Nanjing University of Information Science and Technology, China Mohamed bin Zayed University of Artificial Intelligence, UAE School of Artificial Intelligence, Jilin University, China
Abstract
On-device learning faces memory constraints when optimizing or fine-tuning on edge devices with limited resources. Current techniques for training deep models edge devices rely heavily backpropagation. However, its high memory usage calls for a reassessment of its dominance. In this paper, we propose forward gradient descent (FGD) as potential solution to overcome the memory capacity limitation in on-device learning. However, FGD's ' dependencies across layers hinder parallel computation and can lead to inefficient resource utilization. To mitigate this limitation, we propose AsyncFGD, an asynchronous framework that decouples dependencies, utilizes module- wise stale parameters, and maximizes parallel computation. We demonstrate its convergence to critical points through rigorous theoretical analysis. Empirical evaluations conducted on NVIDIA's AGX Orin, a popular embedded device, show that AsyneFGD reduces memory con- sumption and enhances hardware efficiency, offering novel approach on-device learning.
Introduction
Deep learning models have increasingly gained attraction in multitude of applications, showcasing exceptional predictive capabilities. Nevertheless, their rapidly expanding size [99 poses a formidable challenge for resource-limited edge devices, such as mobile phones and embedded systems. These devices are pervasive our society and continuously generate new data. To attain model customiza- tion, user privacy, and low latency, these devices necessitate on-device learning, involving training and fine-tuning models on freshly gathered data [46). However, the restricted memory capacity of these devices emerges significant hindrance. For example, the Raspberry Pi Model A, introduced in 2013, only featured 256 MB of memory [17], while the more recent Raspberry Pi 400, released in 2020, modestly increased this capacity to mere GB.
Various techniques have been proposed address this issue, encompassing quantized training, effi- cient transfer learning, and rematerialization. Quantized training curtails memory consumption by Efficient transfer learning dimin- 44 Lastly, rematerialization
utilizing low-precision network representation [22 15
ishes training costs by updating merely subset of the model 20
5
Author Zhouyuan Huo is currently at Google. No work performed at Google.
I Corresponniig author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

w
1
A
2
B
3
C
4
w
R o'
Mp..
R ok
A
Broadcast of
Host
wt
B
2
we+
Find local tangent u² Update with of x
Ag
Client:
Client:
M
JVP
14 Activation,
Time Stamp
Message Passing
Worker
14.09
Forward
0
JVP value
Figure 1: Comparison of Vanilla FGD and AsyncFGD, where and signify workers in the system. Through the allocation of tasks from varying iterations, AsyncFGD breaks forward locking in FGD, thereby maximizing worker utilization.
conserves memory the expense of time by discarding and recomputing intermediate variables
3
16. 35
Although most of these techniques can be applied irrespective of the optimization strategy, they are frequently employed in tandem with backpropagation deep learning. Due to the reciprocal nature of backpropagation (i.e.. activations from the forward pass preserved for the subsequent backward pass), the training of deep models utilizing backpropagation typically commands memory footprint that is 3-4x larger than the number of parameters involved. Consequently, reexamining backpropagation within the context of surpassing the memory capacity barrier is crucial due to its elevated memory consumption.
The recent revival of interest in the forward-forward algorithm proposed by [TQ] along with other algorithms predicated on forward computation 39 28 backpropagation, The question persists: to propagate gradients? An alternative, forward gradient descent (FGD), leverages the Jacobian- vector product (JVP) under automatic differentiation [37|]]]] to assess the effect of a stochastic perturbation in conjunction with the forward pass, yielding an unbiased approximation of the true gradient. FGD offers benefits in terms memory consumption and biological plausibility(3I, as solely employs forward passes and substantially reduces the storage of intermediate variables [1]] Moreover, FGD can be combined with other existing techniques, such as quantized training and efficient transfer learning, potentially further diminishing memory costs.
33 has prompted reevaluation of essential to store intermediate variables and pause
1
Nonetheless, the intrinsic locking mechanism within FGD, more specifically, the layer-by-layer sequential computation during the forward pass, poses an impediment to parallel computation across disparate layers as shown in Fig. [ In response these challenges, this paper aims training deep models on edge devices with memory constraints from optimization perspective, while improving resource utilization by breaking the lock inside the forward pass with provable convergence. Thus, propose AsyncFGD, an asynchronous version of Forward Gradient Descent with module-wise asynchronism in the forward pass to disentangle its dependencies, allowing simultaneous computation on different workers. The induced module wistee decreasing and bounded staleness in the parameters not only accelerates prediction and training but also provides theoretical guarantees in convergence. We empirically validate our method across multiple architectures and devices, including CPUs, GPUs, and embedded devices. Our results indicate that AsyneFGI) reduces memory consumption and improves hardware efficiency, achieving efficient training on edge devices.
2 Related Works
2.1 Forward Gradient with Reinforcement Learning
This research builds upon the concept of forward-mode automatic differentiation (AD) initially introduced by [37]. It has since been applied to learning recurrent neural networks (39 and calculating

---

Hessian vector products (28]. However, exact gradient computation using forward-mode AD requires the complete and computationally expensive Jacobian matrix. Recently, Baydin et al. [ and Silver et al. [1 proposed an innovative technique for weight updating based on directional gradients along randomly perturbed directions. These algorithms have connections to both reinforcement learning (RI.) and evolution strategies (ES), since the network receives global rewards in each instance. RI. and ES have been successfully utilized in specific continuous control and decision-male making tasks in neural networks [38 34 322 4) Clark al. 4 observed that global credit assignment performs well in vector neural networks with weights between vectorized neuron groups. However, in forward-model AD methods, much like in evolution strategies, the random sampling approach can lead to high variance in the estimated gradient, particularly when optimizing over a Harge-dimensions parameter space 331
2.2 Parallel Strategies and Asynchronous Approaches
Various parallel frameworks have been developed accelerate computations by utilizing multiple
workers. These frameworks include data parallelism [21], pipeline parallelism [12]30] 30 [2] [1, and tensor parallelism [29 24 Each approach leverages different dimensions of data batch dimension
for data parallelism, layer dimension for pipeline parallelism, and feature dimension for tensor parallelism to improve computational efficiency.
However, in edge computing environments, data is often collected and processed frame-by-frame rather than in batches. This is particularly true for scenarios requiring real-time adaptation and streaming data learning. In such contexts, edge models must quickly adapt to newly acquired data in real-time, leaving limited scope for collecting, fetching, and batching historical data. Therefore, algorithms that do not depend on batch size are preferable. The ability to align with this pipeline and train the model on the fly with incoming data is crucial. Therefore, our research specifically focuses on pipeline parallelism.
Pipeline parallelism can be categorized into synchronous and asynchronous pipelines, which divide computations into sequential stages and align well with the data stream on edge devices. However, in edge devices with limited computational power, potential idle workers due synchronization in synchronous pipelines like [2] are not optimal.
Asynchronous parallelism, which allows non-simultaneous task execution to enhance resource utilization, is particularly relevant to our method. 23 14 parallelize tasks by executing them from different iterations. However, the additional memory overhead of storing replicate copies to handle staleness in (23) and the use of multiple activations for backpropagation make the algorithm memory inefficient. This memory overhead is mitigated in strategies such "rematerialization [40] 13 and weight estimation [1) However, these strategies were originally designed for backpropagation. while our focus is on FGD, which has distinct computational characteristics, rendering the existing "I "IBIF" strategy for work scheduling in [40 13 41 23 unsuitable.
Preliminaries
Gradient-based Method. We commence with a succinct introduction to gradient-based methods deployed in neural networks. Envision the training of a L-layer feedforward neural network, where each layer / E 1, 2.... L accepts h1-1 as an input, generating an activation by = Fi wy) with weight wy E Rdd We represent all parameters by w = [wn T, w wz T] E Rd and the output of the network by h2 F F(ho, w). where ho symbolizes the input data Given a loss function f and targets y, the objective problem is as follows:
min (FFx(((;;
(I)
Here, we use f(z; w) in subsequent sections for the sake of brevity.
Gradient-basse based methods widely used optimize deep learning problems. At iteration we feed data sample Ze(1) into the network, where i(l) signifies the sample's index. As per the principles of stochastic gradient descent (SGD), we define the network parameters follows:
wf+1 = wi V V fi (w"), E (1,
(2)
Here, n E R. is the stepsize and V fi (w*) E Rd is the gradient of the loss function with respect to the weights layer I and data sample Z.()).
3

---

The crux of the matter is obtaining V fi.zgg (w'). Both scientific and industrial sectors lean towards backpropagation underpinned by automatic differentiation (AD). However, FGD could be another alternative, because it can approximate the true gradient without bias by using forward-mode AD, and importantly has low memory consumption as only preserve the intermediate variable passed from previous layer, while backpropagation need to store intermediate variable in each layer.
Forward Gradient Descent. Let Jr, represent the Jacobian matrix of layer while Uwy represent the random perturbation on wi (more precisely, tangent vector around wi), we can calculate the JVP value of, in each layer sequentially by
h4 = Fi (7 wt) Rd
= J/z (1-1,wi)uu E Rd,, where in = [ot-1 the T]T I
(4)
Mathematically. or is the directional derivative of Fi along u;; intuitively, of can be interpreted as the influence of the perturbation Uw; function value h; (we set og to be 0 since don't 't need to calculate the JVP value with respect to the input data). This process can be arranged within the forward pass such that (3) (4) are computed in the same pass. Also, since we don't need to propagate backward, h and 0t-1 are thrown away right after the computation of h4 and or. Then
we can approximate the true gradient V (a; w) unbiasedly (Lemma 3.1 by
01 "r.) u, where "hh
(Uw, ..... Uwe T
Lemma 3.1. Lef Man E ER E (1, L} be normally distributed random Gaussian vectors, then the forward gradient computed through Eq. B). (4) is an unbiased estimate of the true gradient
(x; = E. (
as
1z
1
ahg
More specifically, each layer receives or from the last layer and updates its own parameters locally by using Euny (Mw, o1). We can then rewrite ? as
((Of.
(+1 1i
wi 7
Oh of uin
(5)
Forward Locking. is evident from Eq. and JVP value 14-1, o from layer
4 that the computation in layer I depends on the activation This dependency creates a "lock" that prevents all layers from updating before receiving the output from dependent layers, thus serializing the computation in the forward pass (refer to Figure for illustration).
4 AsyneFGD
In this section, we propose an innovative approach that utilizes module-wise staleness to untether the dependencies inherent in FGD. This method, which we've named AsyneFGD, facilitates the simultaneous execution of tasks originating from disparate iterations. Suppose a L-layer feedforward neural network is divided into K modules, with each module comprising a set of consecutive layers and their respective parameters. Consequently, we have a configuration such that w (urg(o)* T, (g(1)* .... ... "o((-1) TTT with ((k) denoting the layer indices in group
Now, let's delve into the details of how AsyncFGD untethers iteration dependencies and expedites the training process.
4.1 Detaching Iteration Dependency
Forward. At the timestamp t, the data sample zt) is pumped the network. In contrast to sequential FGD [1) AsyncFGD permits modules to concurrently compute tasks, each originally belonging distinct iteration. All modules, with the exception of the last, operate using delayed parameters. We designate the activation and its JVP value originally ascribed to iteration t in [3) (4) as of. hi respectively. Though the superscript is no longer time-dependent, it maintains its role in indicating sequential order. Consider L E (kk) as the final layer in module & and fx as the activation of this last layer. The computation in module & E 0, 0. K - 1 at timestamp t is defined recursively as follows:
h =ʃ4 (Ni=-,; w 1-2K 2
+k+22
(6)
D(k)

---

o% 6t= =J
(Rree ; woik)
2k+
Ug(k)) where dir
ot- I
(7)
Concurrently, each module receives and stores output from its dependent module for future computa- tions.
Update. The update phase in AsyneFGD parallels that in FGD [. The final module broadcasts its JVP value, triggering each module to perform local updates to their parameters. To summarize, at timestamp we execute the following update:
K K+2 w ---- M- (1-K+1 u (
(8)
Staleness. We measure the time delay in Eq. (6. with g(k) = K - k. and g(K -) = indicates that last module employs up-to-date parameters.
This approach effectively disrupts the "lock-in" characteristic of FGD, facilitating parallel forward pass. A comparative illustration of the execution details in sequential FGD and AsyncFGD is provided Figure
4.2
Stochastic AsyncFGD Alogrithm
To better illustrate the working state of AsyncFGD, we make the following definitions:
I -++114
0,0,
L-k
otherwise
and ii-k Lx o% =
hi-k st-k otherwise
(9)
Unlike FGD, AsyncFGD forwards the JVP value and activations by parameters in different time
delays, which can be concluded as f (2=( 9(0)
2K-
g()
9(1)
4-2K33 w t- K+1
9(K-1))
A detailed
illustration of the AsyncFGD with = 3 is shown in Appendix G.2 We summarize the proposed
algorithm in Algorithm by example of sampling one tangent vector per iteration.
Algorithm AsyncFGD-SGGi
Initialize: Stepsize sequence (x T--1
(wi---1 weight w w0 [0(10r"""
Rd
1: for = 0, ... T - do
2: for k = 0,
K 1 in parallel do
Read hi-k 0 k from storage if
L2-1
4: Compute h o
c 5: Send 11 T
to next worker's storage if k: + K
6: end for
7: Broadcast 0L
K+1
8: for k 0, 1, 1,...
K 1 in parallel do
9: Compute Aw 1-K+1 K+1 0 K+ K+1
A-K+2 1-K+1 Th- T K.KIAA 1-K+1 10: Update wo)(k)
gg(k)
11: end for
12:
end for
Additionally, the procedure in line 5 could overlap with the operations in line 7,9 and 10. We can also apply this approximation algorithm to gradient- based methods like Adam [18 with little modification to the original. Details can be found in Appendix G.I
Tangent checkpoint. However, some workers, especially those which closer to the input, store duplicated tangent vectors. To tackle this issue, we use tangent checkpoint,.ee. storing the seed of tangent vectors and reproducing them in the update stage.
Integration with Efficient Transfer Learning Although AsyncFGD offers several advantages over backpropagation, shares a common challenge with random Zeroth-Order Optimization and Evolution Strategy methods: the variance of the approximation increases with the dimension of random perturbations(25) Reducing the learning rate can help but may result in slower convergence. However, observe that deploying models on edge devices typically involves fine-tuning rather

---

than training from scratch. Our method can flexibly incorporate the principles of efficient transfer learning by introducing a scaling factor a E [0. 1] the randomly sampled tangent:
The modified vig still supports approximation of the true gradient, with the expectation of the modified estimator being a V [æ; wi). When Owr is set to 0, the corresponding parameter is "frozen," resulting in no perturbation or updates and a transparent transmission of JVP values. By adjusting to various values, aligning our approach with the objectives of efficient transfer learning.
either fix reduce the influence and learning of specifie layers,
4.3 Acceleration of AsyncEGD
When K = 1 the AsyncFGD is reduced to vanilla FGD without any time delay in param- eters. When K 2 2, we can distribute the net- work across multiple workers. Figure 2 shows the computational time of different algorithms in ideal conditions (i.e. the network evenly dis- tributed and the communication is overlapped by computation and update). TF denotes the time for forward pass and Tú denotes the time for updates. It easy to see that AsyneFGD can fully utilize the computation resources, thus achieving considerable speedup.
Computation Time
Method
Training Progress (K=3)
T₃ To
Sync FGD
Tp + Ty
TF + Ty
Async FGD
Updatte
Higure Comparison computation and training process when the network is deployed across K workers, Since Tp is much larger Tv. AsyneGGD can achieve considerable speedup.
Convergence Analysis
In this section, we provide the convergence guarantee of Algorithm to critical points in non-convex setup. We first make the following basic assumptions for nonconvex optimization: Assumption 5.1. The gradient of f (w) is Lipschitz continuous with Lipschitz constant > 0, i.c.,
Vx, E Rd [V V f()ill - L|[x -yl-
Assumption 5.2. The second moment of stochastic gradient is bounded, i.e., there exist a constant M 2 0, for any sample and Vu E Rd:
II (w,)(1 (w)ll3 S
Lemma 5.3 (Mean and Variance). Let t' = t- and diagonal matrices
Io.... Ik,... ... I E Raddd such that all the principle diagonal elements of I in G(k) are
1, and all the principle diagonal elements of T in other than G(k) are O. Then obtain the mean value and the variance of the forward gradient as follows,
Eyy
(oLR.
- Weae)) = Sgccsizue
(w='K+t+)) -4++1).
(wo
(10)
K-1
K-1
& Sg(k), (w (w
E
Σ Ts IE.of k=0
S (d 4
Σ
(11)
k=0
Remark 5.4. Note that, from modules to K
1, the corresponding time delays are from
K to 0. Specifically, in timestamp f', when K
Egy Be Oo ulngx-))
wt indicates that the forward gradient in module K 1 is unbiased esti- mation the da-to-date gradient.
fe (fe(K-1).sue)
Under Assumption 5.1 and
5.2
we obtain the following descent lemma about the objective function
value:
Lemma 5.5. Assume that Assumption 5.1 and 5.2 hold. In addition, let t- K + 1,0
max+ [max(0,''-K1) --11, Mx KM + K M and choose "(r S +. The iterations in Algorithm I
"t satisfy the following descent property in expectation, Vt! E N:
f(w('t1) f(w"') < 7t'
[VV ) 4(d 4)L
(12)

---

Theorem 5.6. Assume Assumption 5.1 and 5.2 hold and the fixed stepsize sequence {re) satisfies N/' =7 - I v'' {0, ......I [j. In addition, we assume w" to be the optimal solution to f(w)
and les = - K + 1, a such that MR- KM + +KM. M. Then, the output of Algorithm I
satisfies that:
T Σ
V N|? 2(f(w9) f(w))) + + 44)yMKK
(13)
T
YT
Theorem 5.7. Assume Assumption 5.1 and 5.2 hold and the diminishing stepsize sequence tre 3 _
satisfies "V = F41 S +. In addition, we assume wt to be the optimal solution to f(w) and let
= K 1, o= = K such that Mx- = KM+ K. Let LetTr = =t We, the outpu of Algorithm
1 satisfies that:
2 (f((w)- - (((w)))((w))) )1 (1)))
Dn + 2N'0 4(d 4)
Th EV
(14)
Fr.
TT
Remark 5.8. Since the stepsize sequence % =
7o ++1
satisfies that limy Σi=o 7 = cc.
1om E1 7i < oc, when T + oo, the RHS of Eq. will converges to O. Let w TT be randomly chosen from Ir= with probabilities proportional to (we)e- then we have lim E [[
Experiments
This section embarks on meticulous examination of our proposed method, AsyncFGD. We as- sess its performance through three distinct facets: memory consumption, acceleration rate, and accuracy. We initiate our analysis by outlining our experimental setup. To validate the efficacy of applying directional derivatives and utilizing module-wiss stale parameters, we contrast Asy with an array of alternative methods encompassing backpropagation, conventional FGD, and other backpropagation -free algorithms. Subsequently, our focus shifts towards scrutinizing the potential of AsD within the sphere of efficient transfer learning, conducting experiments on prevalent efficient networks. Memory consumption represents another cardinal aspect that we explore, bench- marking AsyncFGD against popular architectures and unit layers like fully-connected layers, RNN cells, and convolutional layers. Concluding empirical investigation, assess the speedup of our method relative to other parallel strategies under a diverse set of conditions across multiple platforms.
6.1 Experimental Setup
Methods. We contrast our proposal's memory footprint with Backpropagation, Sublinear
[3]
Backpropagation through time, and Memory Efficient BPTT (7]. Accuracy-wise, we compare with
hackpropagation-ffee methods: Feedback Alignment (FA) (27]. Direct Feedback Alignment (DFA)
26). Direct Random Tangent Propagation, and perro-sign-based Direct Feedback Alignment (sDFA)
6 We also apply parallelization to FGD through FGD-DP (data parallelism) and FGD-MP (model parallelism).
Platform. Experiments utilize Python 3.8 and Pytorch, primarily on Vidia's AGX Orin. Additional results on alternate platforms are the appendix.
Training. Batch size is 64 unless noted. The optimal learning rate (chosen from {le 5, le 4, le 3, 1e - 2} with Adam optimizer (8)) is based on validation performance. The parameter a is initially set to for the classifier, with others at for the first 10 epochs. Subsequently, is gradually increased to 0.15 for specific layers. More details are in the appendix.
6.2 Effectiveness of Directional Derivative and Asynchronism
This section documents our experimentation on the effectiveness of using random directional deriva- tive to approximate the true gradient by contrasting it with other BP-free algorithms. Furthermore, we demonstrate that the consistency remains intact when using module-wise stale parameters to uncouple the dependencies, comparing AsyneFGD with vanilla FGD. Results in Table Taaee[innde indicate that AsyncFGD produce results closely aligned with vanilla FGD. Notably, FGD and AsyncFGD yield
7

---

Table 1: Comparison for AsyncFGD with other BP-free methods. ConvS and FCS refers to small convlutional network and full-connected network while ConvL and FCL refers to their slightly bigger couterpart. Different activation functions are marked as subscript. Details of network architecutre
can be found in Appendix H.2
BP-free
Dataset
Model
BP
FA
DFA
sDFA
DRTP
FGD
Async
ConvSroohh
98.7
88.1
95.9
96,8
95.4
94.6
94.4
Lo
99.2
12.0
11.5
13.8
13.6
95.5
95.5
ConvLTanh
99.3
8.7
92.2
93.4
92.6
94.4
94.2
Lo
99.3
89.7
93.0
93.1
93.2
93.0
93.2
MNIST
FCS
98.9
83,2
95.6
94.2
94.5
94.4
94,3
FCSI
98.5
8.8
10.0
10.8
9.9
93.6
93.7
FCL
98.8
89.8
93.0
92.0
92.4
95.2
95.4
FCLRELU
99.3
86.3
93.8
94.3
94.1
94.8
ConySTanh
69.1
33.4
56.5
57.4
57.6
46.5
46.0
ConvS
69.3
12.0
11.5
12.0
40.0
39.7
Convl.Tenh
71.0
40.4
42.0
43.6
44.1
47.3
47.3
ConvLRatu
71.2
40.4
42.0
43.6
44.1
44.2
44.1
CIFAR-10
FCS
47.8
46.2
46.4
46.0
46.2
42.0
42.3
FCS
52.7
10.2
12.0
10.3
43.7
43.6
FCL
54.4
17.4
44.0
44.3
45.5
47.2
47.2
FCLRELU
55.3
40.4
42.0
43.6
44.1
46.0
46.7
the most optimal outcomes when the network is more complex, or when we employ ReLU as the activation function devoid of batchnorm layers (results from ConyS and FCSReLU). situations where FA, DFA and sDFA often fail to propagate effective error message. Backpropagation results are also furnished solely as reference to true gradient results. However, when the network grows larger, all BP-free algorithms grapple with variance. The use of larger networks results in only minor improvements compared to BP. We try to address this challenge through efficient transfer learning in the subsequent section.
Memory Consumption (40)
Asymi
1.0>
10a
90
Sublinan
Efficien
10
Mobllcket
1.0
Restict
Async
4 xx 1.00..
80
Mhasset
Axyre!
Shufflckis
Async*
Sublinwar
50 160
150
200
250
560 1000
1500 Number of Time Steps
2000
2500
1000
2:00
3000
4030
2000
Number Layers
Mamory Consumption
(a) FC Layers
(b) RNN Cell
(c) Popular Efficient Architectures
Figure 3: Memory footprint comparison across methods. Asynet is AsyneFGD without tangent checkpoint while Async* refers to AsyncFGD using efficient training strategy. (a) FC layer memory vs. layer count; (b) RNN memory sequential length; (c) Accuracy vs. memory on efficient architectures.
6.3 Efficacy of Efficient Transfer Learning
In this segment, we delve into the efficacy of amalgamating AsyneFGD with efficient transfer learning, focusing on popular architectures like ResNet- 18fteet-88Res-1885 18) ();; MoblieNet (Mobile) M 36) and ShuffleNNe((Shffie)155555 with their lightweight counterpart. The models are fine-tuned with weights pre-trained on Imagenet. AsyncFGD denotes AsyncFGD utilizing the efficient training strategy. can observed from Table 2 the application of an efficient transfer learning strategy brings about substantial performance enhancement, yiclding superior results

---

compared to training with perturbation on the full model. Ablation studies on a provided in Appendix also shows that, compared optimizing subset of the model, FGD suffers more from variance.
6.4 Memory Footprint
As illustrated in Fig [(c) when we plot accuracy against memory consumption, it is evident that AsyncFGD employs approximately one-third of the memory while maintaining close accuracy. Further experimentation on memory consumption with respect to computing unit reveals, as shown in Fig 3(a) and Fig 3(b) that the additional memory consumption in AsyneFGD mainly serves as placeholders for random tangents, while the JVP computation consumes a negligible amount of additional memory. Memory consumption of other basic units like CNN and batch norm layers, provided in Appendix.
6.5 Acceleration on Input Stream
In this final section, we assess the acceleration of ResNet-18 with varying K. In this setting, the batch size is set to 4 to better reflect the mechanism of streamlined input edge device. As demonstrated in AsyncFGD, by detaching dependencies in the forward pass, can outperform other parallel strategies terms of acceleration rate. While pipeline parallelism is fast, the locking within the forward pass still induces overhead for synchronization, ultimately leading to lower hardware utilization and speed. Results pertaining different network architectures and other platforms like CPU and GPU well as more generalized case for larger batch size can be found in the Appendix I.2
4
3.0
Async
FGD MP
Table 2: Results for different algorithms in transfer learning. Async* refers to using efficient transfer learning strategy.
2.5
FGD-OP
2.0
2.5
1.0-
Dataset
Model
BP
FGD
Asyne
Asyne
0.5
0.0
Res-18
98.5
90.6
90.4
Mobile
99.2
89.3
88.4
97.1
(a) Cluster with 4 GPUs
MNIST
Efficient
99.2
90.4
90.1
95.9
Mnas
98.9
86.6
86.3
96.0
Shuffic
99.0
85.8
85.8
96.4
3.0
Async
Res-18
93.0
71.2
71.2
87.8
Accellealla on
2.5
FGD DP
Mobile
94.0
72.3
72.2
91.1
2.0
Sync
CIFAR
Efficient
94.9
70.2
70.1
90.2
1.5
L.1
1.5.
Mnas
84.2
68.8
68.5
78.9
Shuffle
89.9
72.5
72.5
82.0
1,0
Res-18 94.2
80.2
80.2
88.0
Mobile
93.2
82.3
83.1
90.6
0.0
FMNIST
Efficient
92.8
79.8
79.8
90.4
Mnas
92.1
77.1
77.0
(b) Single Embedded Device
Shuffic
92.8
78.4
78.4
87.3
Figure 4: Comparison for acceleration of dif- ferent parallel methods.
7 Limitations and Discussion
While AsyncFGD offers computational benefits, is without limitations. A key drawback is its inferior performance compared to traditional Backpropagation (BP). This performance disparity largely attributed to the use randomly sampled directional derivatives the forward gradient computation, aligning AsyncFGD with Zeroth-Order (ZO) optimization methods and evolutionary strategies. This sampling introduces a significant source of gradient variance, challenge that part of larger problem in the field of stochastic optimization. However, we take encouragement from

---

recent advancements aimed at reducing this variance, some of which have even facilitated the training of large-scale models [T.
Another constraint pertains to the availability of idle workers on edge devices-a a condition that is universally applicable given the wide variety of edge computing environments. These span from IoT chips with limited computational resources, where even deploying a standard deep learning model be problematic, high-capacity micro-computers used autonomous vehicles.
Nevertheless, our experimental findings suggest that AsyncIGD is particularly beneficial for specific edge computing scenarios. In such settings, it may serve viable alternative for reducing memory usage while fully leveraging available computational resources.
Conclusion
In the present paper, introduce AsyncFGD, innovative approach designed shatter the shackles of locking within the forward pass in FGD. This is achieved by incorporating module-wise stale parameters, simultaneously retaining the advantage of minimized memory consumption. In the theoretical segment, we offer lucid analysis of this partially ordered staleness, demonstrating that our proposed method is capable of converging to critical points even in the face of non-convex problems. We further extend our algorithm to efficient transfer learning by introducing a scale parameter. Our experimental reveals that a sublinear acceleration can be accomplished, without compromising accuracy well as huge performance gain when utilizzig efficient transfer learning strategy. While the exploration of large models employing extensive datasets will undoubtedly continue to rely backpropagation [10], we assert that the potential of asynchronous algorithms predicated on forward computation should not overlooked. It offers promising avenue for fully harnessing limited resources in on-device learning scenarios.
10

---

References
[1] Atilum Güney Baydin, Barak A Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backpropagation. arXiv preprint arXiv:2202.08.. 08528787..88 2022.
[2] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinyl:: Reduce activations, not trainable parameters for efficient on-device learning. arXiv preprint arXiv:2007.11622, 2020.
[3] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: wwXivv/604.00 /604.66144. 2016.
[4] David Clark, LF Abbott, and Yuo Chung. Credit assignment through broadcasting a global error vector. Advances in Neural Information Processing Systems, 34:10053-10066 2021.
[5] Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specifie transfer learning. Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4109-4118, 2018.
[6] Charlotte Frenkel, Martin Lefebvre, and David Bol. Learning without feedback: Fixed ran- dom learning signals allow for feedforward training of deep neural networks. Frontiers in neuroscience, 15:629892, 2021.
[7] Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory- efficient backpropagation through time. Advances Neural Information Processing Systems, 29, 2016.
[8] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pages 1737-1746. PMIR, 2015.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[10] Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212. 13.44555...3...... 2022.
[1]] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for mobilenetv3, 2019.
[12] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[13] Zhouyuan Huo, Bin Gu, and Heng Huang. Training neural networks using features replay. Advances Neural Information Processing Systems, 31, 2018.
[14] Zhouyuan Huo, Bin Gu, Heng Huang, et al. Decoupled parallel backpropagation with conver- gence guarantee. In International Conference on Machine Learning, pages 2098-2106. PMLR, 2018.
[15] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeticonn inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018
[16] Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E, Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerialization, 2020.
[17] Steven Johnston and Simon Cox. The raspberry pi: A technology disrupter, and the enabler of dreams, 2017.
11

---

[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[19] Skanda Koppula, Lois Orosa, A Giray Yağlikęı, Roknoddin Azizi, Taha Shahroodi, Konstantinos Kanellopoulos, and Onur Mutlu. Eden: Enabling -ffrgyeffficiett,, high-performance deep neural network inference using approximate dram. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchirecture, pages 166-183 2019.
[20] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2019.
[21] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[22] Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. Advances in Neural Information Processing Systems, 30, 2017.
[23] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium Operating Systems Principles, pages 1-15, 2019.
[24] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-Im. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15, 2021.
[25] Yurii Nesterov and Vladimir Spokoiny, Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-55 2017.
[26] Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
[27] Paul Orsmond and Stephen Merry. Feedback alignment: effective and ineffective links between tutors' and students' understanding of coursework feedback. Assessment & Evaluation in Higher Education, 3652):1 2011.
[28] Barak A Pearlmutter. Fast exact multiplication by the hessian, Neural computation, 6(1):147-
160, 1994.
[29] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEF, 2020.
[30] Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances neural information processing systems, 24, 2011.
[31] Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. arXiv preprint arXiv:2210.033100 2022.
[32] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv: /eXXvv:/03... 1703.038644 2017.
[33] David Silver, Anirudh Goyal, Ivo Danihelka, Matteo Hessel, and Hado van Hasselt. Learning by directional gradient descent. In International Conference Learning Representattonss 2022.
[34] Kenneth O Stanley and Risto Müikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-127, 2002.
[35] Benoit Steiner, Mostafa Elhoushi, Jacob Kahn, and James Hegarty. Olla: Optimizing the lifetime and location of arrays to reduce the memory usage of neural networks, 2022.
12

---

[36] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-awand aware neural architecture search for mobile, 2019.
[37] Robert Edwin Wengert, simple automatic derivative evaluation program. Communications of the ACM, 7(8):463-464, 1964.
[38] Darrell Whitley, Stephen Dominic, Rajarshi Das, and Charles W Anderson. Genetic reinforce- ment learning for neurocontrol problems. Machine Learning, 13:259-284 2244, 1993.
[39] Ronald Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
[40] An Xu, Zhouyuan Huo, and Heng Huang. On the acceleration of deep learning model parallelism with staleness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2088-2097, 2020.
[41] Bowen Yang, Jian Zhang, Jonathan Li, Christopher Ré, Christopher Aberger, and Christopher De Sa. Pipemare: Asynchronous pipeline parallel dnn training. Proceedings of Machine Learning and Systems, 3:269-296. 2969 2021.
[42] Guandao Yang. Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Chris De Sa. Swalp: Stochastic weight averaging in low precision training. In International Conference on Machine Learning, pages 7015-7024. PMLR, 2019.
[43] Yukuan Yang. Lei Deng, Shuang Wu, Tianyi Yan, Yuan Xie, and Guoqi Li. Training high- performance and large-scale deep neural networks with full 8-bit integers. Neural Networks, 125:70-82. 2020.
[44] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint (xXi::2106.10199 2021.
[45] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices, 2017.
[46] Qihua Zhou, Zhihao Qu, Song Guo, Boyuan Luo, Jingeai Guo, Zhenda Xu, and Rajendra Akerkar. On-device learning systems for edge intelligence: A software and hardware synergy perspective. IEEE Internet of Things Journal, 8(15)11116--11944 2021.
13

---

A Appendix
The organization of the Appendix is follows: Appendix sections provide detailed proofs of the lemmas and theorems presented in the main text. This is followed by additional insights, including operational specifies of working with Adam and details on how AsyncFGD can be extended for parallel processing on recursive networks, covered in Appendix G Lastly, Appendix and [ provide a comprehensive view of the training process and additional experimental results, respectively.
B Proof of Lemma 3.1
According to the update rule of Eq. 3)
we have
=Fi(ho w1) (I, w1)
Jr, (ho, wi wi)lo uT,, 1T Jn (ho)on Jm. (wi ) Jm (wi )
hg2 F2(11, W2)
Jr. (h1, wo)[oT TTg TT = Jr, (h1)01 + JF,( (w2) = -ra((h))d [ba))Fi (w1) Mun + JFL (102) Ues
J (wi )wws + JF, (w2)uas
JFL (wy Mu
Then for any (1,2,.... LJ. take expectation with respect to Uayz have
Buny (or Umn)) = Bu
(Jr, (wi) (wr)) they +
Σ kFf
(Jrr (wx) )um)) Mur
Note that
0J
8J
8Jp 8 Banid 0JF,
"Dur.1
00112
OJrm.
0JFr.2
Uurz
Donia
Uary
Ems, [[Jr, (wi) ) Uewl = BBu.,
2 ... Merr
HJ. v.
Uun
Owi.1
Owi,a
Dwi.d
di
0J
Owi.
Uuri
0J
1=
Swn,i
-Eaay
Uwi
aJr, dnp Want
Σd. -==1
= EEuny (D),
where
F.
aJ Jr 2
JJ Fl.m thri Warinn
Dm.
Uwi.
Uwin
Owisi
Iwi.n
Owi
k
with m E (1,2.... dr J.n n E f1,2..... .., Since each Mhr ~ N(0, I). we have
di}.
F.
Eu,, (Dm,n) = Iwrn
Eus, (D) JFz (wz).
Similarly, we can prove that Ev.,
EEAes (JFr, (wk) Uwu
= E Rdo, xd. So we have,
0f
0f
E
or Uwn
JFz (wi) = (.. E RR
OF,
DF,
14

---

C Proof of Lemma 5.3
Proof of Mean.
h1- ht-K+1 Fi 1.t-Ko K+1 wi 2 =
of K+l Jr, (1) (11-- k+1 wi ( |0:: uwn
k Jr, (110 Jn (w) -2--2k+2jul u 22
= Jrs (w) [w]-200 2k+ uu t-K+1
h2 -- k+1
F/ (hi k+
02 k+1 JrH (h) 1 K+1
wiz x - wwz
- Jr, 1o---+1ppp K+1
- + Jr, (
Jrz (1i 1 K ) (wi
1-2 -2 +2 yuu -+] + JrL (w2) (
--K+1
1022 6-2K+2
Lo
K++++1
Σ
J fo (wi
12--2k+2 -22 --
Σ
(s (f(((-2k+2) 22 juu
1-x+1
1=1
lED(0)
%.
6t-K+1_ K+1
Σ
(1t 21 + Σ
(uc
J/,(w, 13 ---KI
Σ
-2 -2k+++2 )u ju
(EG(O)
(EG(1)
LEG())
Σ - -+1
k0EE((())
K-1
K-1
Σ
Σ
Jjk [ (w]
Σ E
- Juhn K
LK-1
k=0 lED(K)
k=0 LEG(K)
K-1
Σ
V F1 (k T
IEG(K)
I-K+1 where 1 E g(k), we have Take expectation with respect to Wwe
E x-x (t-K+1
w [-KK1]-- = V f1.z, k11) (w 2k+
May
LK--
So we have,
E K mugg))
(ot- K+1
-+t]+ V((k) (w (2k
Lemma C.1 ((5),, Theorem 3). Let gu(x) = fx)) u) u, where E IRd is normally distributed
Gaussian vector, then we have
Eg (x)la(()) < (d f())[
Lemma C.2. Let 9us (z) (V)()), u1) u2 where i1 E Rd u2 Rd are two i.i.d. normally distributed Gaussian vectors, then we have
Euy 1 (x) < del (
Proof.
Eug Air.uullga,,.o
(x)112 Buy, me (VV((x)) uz) 12 Eug,us ((V((), u1)² (1)21121133
Ew, (1)())) Eugg (11w211))
di
SdzEn, (1)())
d2Eng Σ
Vel(r)u1s
15

---

d1
=d2Bu1
Σ
Vis())n
Σ
i71
(z))
where the first inequality is due to Lemma in 25
Proof of Variance.
K
1
E
Σ
I
0%
E Uincees
k=U
2
K
K-1
off Oix- M6(R))
Vocu)aa (w - -+J+1), w whnaws umugen
E 41p
k=0
1-1
V Soo) (w" -K++t)),, (lcces))
E
k=U
K-1
( (Vtt))aaa (w-- -K---)t winecce)
y++
2
)² 2 Minoess
=E
V ffG0)
km
V fstm) (w--K-min -+mtt), uh ulipcms) )
12 V (8(n)) (w -Ktntt ah abreeco)) |
m
K-1
V feoisae -K+st+ wlrgees )²) ||m66co)) 12"
+
h=0
J
(ds 44) V Vogurraa (w0 -K-'-K-ktt -K-bt))
+
Σ
dR
Vigc) (w -KKstttt
Jzk
V Your (wf /|²
V Ta(M) (w0" (wo ( ||²
dx
+
K
V Veen (w (w0 -Ktst))
V Vogus (w >4
dy
+
1=0
R=U
V fa(N) (wo -Ktt1))
(w" (wit Σ I V (a(K)
=dd+ 4)
(d +
where the inequality due to Lemma and C.2
D proof of Lemma 5.6
Proof. For the convenience of analysis, we denote t' =t=t- K + 1, then the update rule of algorithm
1 can be rewritten as
w 4'+1 witn .v (0% whrross) ((k)
Take expectation with respect to " we have
''+1 whyn) vo TerBnion (0% weewn) wb V Viokk)aau
(e-K+h+1
w g(k)
(wg(k)
16

---

_II1 E R such that all the principle diagonal elements of I in (k) are 1, and all the principle diagonal elements of Is in other than (kk are O. Then we have
Define diagonal matrices Io....
I.....
K-1 Σ In
oz
k=
K-1 (oWN))a
V Vizer (wtt-
Σ
(w)())
k=0
Since f(-) is L-smooth, the following inequality holds that:
L 2
f( S f(w")) ) + (V((""), w w'
|w0
d'
w² 112
From the update rule of Algorithm 1, we take expectation with respect all random variables both sides and obtain:
K-1
}
Ine
K-
[[ +1)]
<s(w")
IB
V((wn)) Σ K=
I of
minscos
e'
E
Σ k=0
Is o% Ox
u Wuoces
K-1
=f(w") V
Vf(w") (ViOCK) (wi (utt-
FoCI) (w") + Vfock) (w)"))
L
K-1
+
Σ
Is
1x woow Vf(w")) v
2
=s(w") -7 (((((()) 1|² 1|²-~
K-1
Vs(w" Ih (VSCCH) (w0' -[ack) (w²"))
k=0
117
VS(wo=) l|2
L E2
K
Σ
I oi mireee V((w(')
+
+
2
2
4=0
K-1
+ L
Σ
Vf(w") I (VSSC
(w" -ktttt))
Fo(C)) (w")) (w²))
k=
=f(ww')
(T0)
Li/
(Vllw') s|?
L 2
E
K-1
Σ k=
L
of
-u
vf(w"))
2
Q1
-(re L7?
V Vf(w" I (Voo (w²-ktbttt)
V f((k))
(1")) (u²)).
Qa
Using the fact that
|| + y11² < 2||0||2 + 2||y/12 and xy < ||/11| + E1l/l2 we have
Q
L
2
E
K
Σ I ot
wwean vf(w") Σ
K-1
Is ( (w²- -xtttt)t
2
k=0
k=
wt- -
+
Is V To(k)
k=0
K-1
K
2 E
Σ km
I 0L
wh Wagge
I ((gik) (w"'KK+tt))
k=
17

---

In
Is V.S (wtt-kthtz) V V((w")
K
f( (Viaw)) (w- (woe-ttttt) VSE(N) (w²))
Q2
(3 Li:e)
k=
W L77
V[(w"() j|²
+ 7'' L
I [V(()) ( - V Vl(w)) Σ
2
Using Ellç - BiEl112 < E we have
K-1
K-1
Qs =E
L Lx-1
w Vagao
I V (w" --kbtt)
k=
k=
K-1 1
SE
Σ k=0
I 0 1x
d'
K
I V fG(M) (wtttkthtt))
S(d+4) 4)
Σ (eck) (w"k (KtKkttt// 1/2
=(d+ 4)
<(d 4)MM
where the second inequality is due to Lemma
5.3
Then we bound Q4.
K
K
I V S0(K) (w'-Kt Vf(w")
Vouay -Ktkt))- -K4tot)) - V Focn)(
Q1
k=O
0=
V fl -K++1)
-k+t))) - vs(w
K-1
10 wn w'' -
212
2
K-1
K-1
=1?
Σ
Σ
(wi+1 w²)
L2
Σ
/=max (0,1' K+h+1}
k=0
f=max -K
K-1
f-1
<I mmax -K+I
Σ
K
Σ
(d+4)
VoCk) (w (wo -KtRt1)
0=
K-1
<(d+ 4
(0m K+1]
foun) (w -Ktbt))
K=0 m=max K K+++1)
<(d+ 4)L M,
where the second inequality is from Assumption 5.] the third inequality is due to Lemma [.3 the fourth inequality follows from Lyy < 1, the last inequality follows from the inequality of anthmetic and geometric means, Assumption 5.2 and max "maa (oo.'=+1)) Integrating the upper bound together, we have
f(wltt)) -f(w') f(w0))=- s- T' Vf(w") | (d (d+ W'' +
(d+ + 4)L M
2
18

---

S Vf(w²)
-[(d 4)47 (KM +oK
"
Vf(w²))
2 - 2(d 4)
where we let Mr = KM + toK
E Proof of Theorem 5.6
Proof. Let 7/t 7 be constant, taking total expectation in Lemma 5.5 have
E [(w - [f(w") - - - ||2 + 2(d + 2)
where = 1 and M2- = KM + K. summing the above inequality from t' = 0 to T we have
T-1
B[f(w")] f(ww)=(w--)))))))){()))))) S
Σ
E V I1 27 2 LMK
P=
Then we have
Vf 112 2(5(w²) - -f(w))
+4(d+ 4)
T
7T
0
F Proof of Theorem 5.7
Proof. Let } be diminishing sequence and T >. such that o K and Mr KM K M. Taking expectation Lemma 5.5 and summing it from t' = to T 1, we have
T-1
Els(w])) - f(wo)) S "BIIV/("
111
2 + LMK Σ P==
n==0
Letting IT 2e=0 Yos then we have
2 (f((w)) - f(ww))
Exz 4(d + 4) LMR
Yv B/ )12
)12 -
Ff P=0
Tr
G Details of AsyneFGD
G.1 Working with Adam
provide example for AsyncFGD working with Adam Algorithm 2 Minimal changes are made on Adam by substituting the gradient the estimator using Forward Gradient.
G.2 Execution Details
Details are presented in Figure 5
By pipelining over time dimension, we can preserve buffer for input in only one timestamp and still achieve parallel computation.
19

---

Algorithm 2 AsyncFGGD-Aam
Initialize: Stepsize sequence - K-1 weight w
w "00o)'" "M w 9(K-1)
Rd, mg(x))
0, (g(k)) = 0, 31 = 0.999, le for = 0.1. T do
2: for 0.1 ... K 1 in parallel do
3:
Read hi L2-1
from storage if k: + 0
d 4: Compute h/ Ln
5: Send h ,0 to next worker's storage if l + K
6: end for
7: Broadcast 6t-K+1 LK-1
8: for & = 0, 1, ... K in parallel do
9:
Compute 1- K+1 = LK-1 (ig(())
g(k)
xt-K+1 K++1
10:
Update mg(k) = B/Au
((k)
+ +11 -1-81)maus
11:
Update (g(k)) 332 --
g(k)
+ (1 (2)
12:
Compute mg(()) = mg(k) /39
13:
Compute ()(()) = tig(k) 13
Update w K+2 u w K+1 7t- K K
end for 9(k)
g(k)
15:
16:
end for
0
Initialize
2
Module Mr
wi,
6 of.
w
f. O7.
wF
S... 63,
st, 07
w
42, 68.
87,- 07.
Module
wx
0,0
wig
wig
w
8i..
67
w
41 61
w2
60g BC
Module
0,0
w
0.0
$8 oE
w2,
$2 oz
w
92, of
$2 oz.
$2,. 22,
92, OF
warm-up
steady
Skip
Update
Figure 5: Details in executon of AsyncFGD-RNN with 3 modules, In the skip stage, only the host accumulate loss and its jvp value and other workers will jump right into the next state.
G.3 Extension:
In this section, we extend the potential AsyncFGD by exploring the parallelization of sequential inputs in RNNs with reduced memory footprint, necessitating the preservation of input for only single timestamp.
We adopt ane-to-many RNN network for ease of illustration and denote the equal length of each sequence n. We begin by refactoring the original loss for RNNs in terms cumulative loss and new activation. Here, of signifies the hidden state at timestamp on layer 1. At timestamp f, each layer ingests (sf_1: s - input, generating si = Fi(s 2-1 w.). We represent the stacked latent states passed from 1 as 1 |s 8/ st. 1 and the output as yt F(s st w.) where si symbolizes the input data x,. The cumulative loss from timestamp 1 ~ T' is given by:
E s
(15)
We next refactor equation for the ie sequential input in iteration i, 2 0 as:
20

---

ac: 76 Owt
wi wi
E ,,,,...,
(16)
where Li :- Σt= in + 11 11+1)) s w)) 3/)) represents the loss for the in sequence. We break the dependency between timestamps and iterations by employing dynamic staleness in AsynFGD. Specifically, the computation in module & 1,2, K at timestamp is defined as follows:
31 1-k k = fo (SL 1-k (0(4) (((k))*k-111 k-1 (Q(()) ( -kthh))
(17)
δL* 1-k Jhk (&L - 1- 1 (k())) 1- -- ; (k(( 1-2K+ (
"((k))
1-b
(18)
where
u t-k 9(k)
[ōT. tk (k) --111 uwgg()) "-T
Given that tasks belonging to the same iteration use identical parameters, we use 8 L,
(-ni ni - - (i++ (im]) to quantify this difference for the ie sequential. If L(k,,,) S O, then module & uses stale parameters from iteration updates the parameter upon the completion of the last computation in the sequence. Specifically, use:
at timestamp 4. AsyncFGD-RNN only
k-K+
if
t-
K +
$ N
L-K-2
[BLL r- K Lx
k= n
w x+1
Euw
t-k
u
(L-
otherwise
OsLk
Refer to figure 5 for detailed execution instructions. By combining training and prediction, we process data from different timestamps of sequential input, maintain buffer for just single timestamp, and still achieve parallelization among various workers.
H Training Details
In this section, we explain some details in the trainning process.
H.1 Random Seed
The seeds for all experiments are fixed to 0.
H.2
Description of Network Architecture
H.2.1 Models Deployed in Section 6.2
The network structures of ConvS, ConeL, FCS and FCL are enumerated in Tables g and 10 correspondingly. In these designations, the subscripts ReLU and Tanh signify the particular acti- vation function used in the model. Moreover, the larger models, denoted as counterparts, incorporate batch normalization layers for enhanced performance.
H.2.2 Models used in Section 6.3
delve into the specific models employed Section 6.3 For MobileNet, we utilize the structure of MobileNet_ V3_ Small. The ResNet- 18 structure is used when implementing ResNet. The model EfficientNet. BO signifies the EfficientNet architecture. The MNASNet0.5 used for MNasNet. Lastly, adopt ShuffleNet V2 X0_ 5 for ShuffleNet.
21

---

H.3 Model Splitting
In this section, we provide details for how to split the model into consective moduels and distribute them in different worker, we will first provide how to split models in Section 6.2 then we provide how to split model in 6.3
H.3.1 Model Splitting in Section 6.2
In Section 6.2 all models are split with K 3. Details for how to split the ConvS, ConvL, FCS, FCL are repented in Table 5 Table Tablo and Table 4 respectively.
Table 3: Details for model splitting for ConvS, definition for layers can be found in Table 8
Table 4: Details for model splitiing for ConvL, definition for layers can be found in Table
K
Layers
K
Layers
conv1, bnl, act pooll, conv2, bn2, act2, pool2
conv1, act
2 conv3, bn3, act3, pool3,conv4, bn4, act4, pool4 conv5, bn5, act5, pool5, fcl
2
pooll, fel
3
act2, fc2
Table 5: Details for model splitting for FCS, definition for layers can be found in T
Table Details for model splitting for FSL., definition for layers can be found in Table
K
Layers
K Layers
fel, bn1, acl, fc2, bn2, ac2
1
fel,acl
fc3, bn3, ac3 ,fc4, bn4, ac4
2
fc2,ac2
3
fc5,bn5,ac5,fc5
3
fc3,ac3
H.3.2 Model Splitting Section 6.3
In section 6.3 all models divided into four parts (K 4). Detailed descriptions of how each model is split are provided below. Note that 'head' and "tail" refer to the layers before and after the main blocks of each architecture, respectively, which are assigned the first and the last worker:
ResNet-18: The core of ResNet-18 consists of Residual Blocks, each distributed one of the four workers.
The of EfficientNet consists of 7 Mobile Inverted Bottlenecks (MBConv). The first worker handles MBCony 1, the second handles MBConv to 3, the third manages MBConv and the last one manages MBCony
EfficientNet::
MoblieNet:
includes 13 layers of bottlenecks. The first worker handles layers 1 to 3, the second manages layers 4 to 7, the third manages layers to 11, and the last worker handles layers 12 to 13.
The core of MoblieNer
MnasNet: The of MnasNet consists of 6 blocks of inverted residuals. Blocks to 2, 3 to 5, and 6 are assigned to workers 2, 3, and respectively, while the first worker only handles the head.
ShuffleNet: The core of ShuffleNet consists of stages, each assigned workers 3, and 4, respectively.
22

---

I Additional Experimental Results
L.1 Ablation study a
We have incremented the value of Obbias gradually, with step size of 0.0075, over 20 epochs. This process can be generalized using the following equations:
Diaa
t x
<<= 20
Obiaa
20
Obias::
otherwise
Here, haas control the rate of increase and the maximum attainable value of dbiass respectively, The ablation study with respect to abins is presented in Table 7
We observe that reducing oos to O, which corresponds to only updating the classifier, still results performance gains compared to updating the full model. This improvement can be attributed to reduced variance. As assus increases, we generally see better results, since the norm of the gradient approximation increases. However, when Obias exceeds 0.25, we sometimes observe performance drop due to the corresponding increase in variance.
Table 7: Ablation study on different value of Oiins
Obias
Dataset
Model
0.00
0.03
0.06
0.09
0.12
0.15
0.20
0.25
Res-18
0.838
0.838
0.845
0.855
0.865
0.878
0.872
0.885
Mobile
0.898
0.912
0.911
0.910
0.913
0.911
0.914
0.909
CIFARIO
Efficient
0.892
0.900
0.902
0.903
0.902
0.902
0.887
0.895
Shuffle
0.788
0.805
0.808
0.812
0.820
0.820
0.822
0.825
Mnas
0.782
0.790
0.788
0.789
0.788
0.789
0.777
0.782
Res-18 0.866
0.869
0.871
0.873
0.875
0.880
0.882
0.884
Mobile
0.890
0.908
0.906
0.906
0.906
0.906
0.899
0.901
FMNIST Efficient
0.889
0.904
0.906
0.902
0.905
0.904
0.908
0.897
Shuffle 0.849
0.854
0.857
0.860
0.864
0.870
0.870
0.877
Mnas
0.854
0.868
0.870
0.870
0.870
0.870
0.864
0.864
I.2 Acceleration across Various Platforms and Architectures
examined the acceleration of AsyncFGD in comparison vanilla FGD ResNet- 18, using two hardware platforms: 1) NVIDIA AGX Orin, an embedded device, and 2) a cluster of four NVIDIA 1080 Ti GPUs. These platforms were chosen to reflect real- world edge device scenarios and to simulate situations of ample computational power, such as in the case of stacked chips.
In Section 6.5
In this section, we expand our scope of investigation by incorporating two additional devices: 1) NVIDIA A100, and 2) Intel(R) Xcon(R) CPU E5-2678 v3 @2.50GHZ. These additions allow us to further examine acceleration under various conditions. We also provide supplementary results on acceleration with respect to different batch sizes to reflect variable input streams. Moreover, to emulate streamlined input, the mini-batch size of the synchronized pipeline is set to 1.
The performance ResNet- 18 with different batch sizes on the four NVIDIA 1080Ti GPUs, A100, and AGX Orin platforms is illustrated in Figures 6 7 and respectively. Results for MobileNetV3. small on AGX Orin are presented Figure [0] notable observation is that AsyneFGD performance appears largely insensitive to batch size. In contrast, other algorithms typically exhibit poorer performance with smaller batch sizes. Particularly, when the batch size reduced to 1, these algorithms offer negligible performance improvements over vanilla FGD. Furthermore, the overall
23

---

acceleration on a single device is constrained by computational power. For instance, while AsyncFGD achieves speedup of x on a four GPU cluster, only delivers a 2.11 speedup on a single AGX Orin. Communication also imposes a limit on the overall acceleration, as demonstrated by the superior performance on the A100 comparison to the four-GPU cluster. This is attributable to the elimination of communication overhead on a single device, except for the sending and receiving operations of CUDA kernels.
Results for MobileNerV3-snnne with different batch sizes on CPU are depicted in Figure 9 Due to the inherently sequential execution pattern CPUs, the acceleration constrained, resulting in only modest speedup and advantage over other algorithms.
4.0
4.0
3.5
Async
3.5
Asyni
FGD-DP
FGD-DP
2.5
2.5
2112
6
1.5
2.11
1.5
1.22
AMM22A
25a
221
0.0
3
0.0
Batch size
Batch size
4.0
4.0
Async
FGD-DP
FGD-DP
3.0
FGD-MP
2.83
3.0
FGD-MP
2.84
2.5
Sync
2.5
1.5-
2.38
1.5-
1.54
3.65
1.2%
(c) Batch size
(d) Batch size
Figure 6: Acceleration with differernt batch size ResNer-18, cluster with 4 Nvidia 1080 Ti
I.3 Memory Profiling on Other Basic Units of Convolutional Neural Networks
This section outlines memory profiling for basic units within a Convolutional Neural Network (CNN). Commonly, CNN layer is coupled with batch normalization layer and an activation layer using ReLU, so we' ve combined these clements for our memory testing. We examine the memory consumption against the number of layers and present the results in Figure [I For further examination, we also assess the memory consumption against the number of output channels and batch size, with results shown in Figures [1<b] and [[(c) respectively.
Our findings reveal that implementing forward gradients can significantly reduce memory consump- tion. Generally, the majority of memory usage in CNNs stems from intermediate results, since CNNs often operate in a "broadcast then product' pattern (to be specific, they are referred as 'img2col'), Consequently, the additional memory required by the random tangent in AsyncFGD is minimal. As such, the memory consumption appears be invariant to the number of layers, mainly because the forward pass discard almost all the intermediate variables.

---

4.0
3.5
Async
3.5
Async
FGD-DP
FGD-DP
211
2.5
Sync
2.5
Sync
1.43
1.5-
1.5-
1.0
M0.50
56004
1.0
0.0
0.0
Batch size
(b) Batch size 2
4.0
4.0
3.5
Async
3.5
Async
FGD-DP
2.2
FGD-DP
3.25
FGD-MP
2.5
2.4
2.5
1.5
1.43
1.0
6 1.5
2.55
0.0
0.0
Batch size
(d) Batch size 4
Figure 7: Acceleration with differernt batch size ResNet-11, A100
3.5
Async
3.5
Async
FGD-DP
FGD-DP
2.5
Sync
2.5
Sync
2.2
1.5
1.38
1.5
1.0
Labo scont
1.0
0.0
0.0
Batch size
Batch size 2
4.0
4,0
3.5
Async
Async
FGD-DP
FGD-DP
2.5
2.5
1.5
1.5
1.5
2.43
0.0
0.0
Batch size
Batch size 4
Figure 8: Acceleration with differemnt batch size on ResNet- 18, AGX Orin
25

---

4.0
4.0
3.5
Async
Asynic
FGD-DP
FGD-DP
FGD-MP
FGD-MP
2.5
2.5
Sync
1.5
1.5
3.23
A.41 64
2360
0.0
0.0
(a) Batch size
(b) Batch size
4.0
4.0
Async
Async
FGD-DP
FGD-DP
3.0
FGD-MP
3.0
FGD-MP
2.5
Sync
2.5
Sync
1.00
1.5
1.22
1.25
141
1.5
3.24
0.262 14
0.5
0.5
(c) Batch size 3
(d) Batch size 4
Figure 9: Acceleration with differernt batch size on MobileNenV3-ssmll. Intel(R) Xeon(R) CPU
ES-2678 v3 @ 2.50GHz
3.5
Async
3.5-
Async
FGD-DP
FGD-DP
FGD-MP
FGD-MP
2.5-
Sync
2.5-
Sync
2.0
2.0
1.43
15
1.0-
1.0-
0.0
0.0
(a) Batch size
(b) Batch size = 2
3.5
Async
3.5
Async
FGD
FGD.D9
FGD-MP
2.5
Sync
2.5
Sync
2.24
1.5-
2.52
1.5-
L36
1.0
240024
0557.9
2400244
0.0
0.0
Batch size 3
Batch size 4
Figure 10: Acceleration with differernt batch size on MobileNerV3-small AGX Orin
26

---

Table 8: Network architecture for CO and Con CoLUSReLU denotes using ReLU for the activation functions and ConuSTank denotes using Tanh as activation functions
Layer
Type
Params
convl
Conv2d
out channelnnis kerne_ Sizuel__sze=S,,,ee stride=1, padding-2
act1
ReLU//anh
N/A
pooll
Maxpool12d
kernel_size=2) size=2, stride=2, padding=O, dilation=1
fel
Linear
out_feature==1000
act2
ReLU/Tanh
N/A
fc2
Linear
out_ Lut_feeture==10
Table 10: Network architecture for LE and LC FCLR.LU denotes using ReLU for the activation functions and FTTaa denotes using Tanh activation functions
Layer
Type
Params
flatten
Flatten
N/A
Table 9: Network architecture for FC and
fel
Linear
out 10l_fuuures=1024
FCStanh. FCS denotes using ReLU for the activation functions and FCSTank denotes us- ing Tanh as activation functions
bnl
BatchNormid
N/A
ac
ReLU/Tanh
N/A
Layer
Type
Params
fc2 Linear
out features=1024
bn2
BatchNormid
N/A
flatten
Flatten
N/A
ac2
ReLU/Tanh
N/A
fcl
Linear
out_ 10a
fc3 Linear
out features= 1024
ac1
ReLU/Tanh
N/A
bn3
BatchNormid
N/A
fc2
Linear
out features=512
ac3
ReLU/Tanh
N/A
ac2
ReLU/Tanh
N/A
fc4 Linear
out 10atures=1024
fc3
Linear
out features=256
bn4
BatchNormid
N/A
ac4
ReLU/Tanh
N/A
fc5
Linear
out features=512
bn5
BatchNormid
N/A
ac5
ReLU/Tanh
N/A
fc6
Linear
out features=10
27

---

Memery Consumptoo (MB)
1500
Async
Memory Consumption (MB)
12500
Async
Async
Asynet
10000
BP
Memory Consumption
BP
1000
7500
300
5000
500
2500
100
10"
10
250
500
750
1000
200
400
600
800
1600
Number Layers
Channels
Batch Size
(a) Different number of layers
(b) Different channels
(c) Different batch size
Figure 11: Memory consumpition of basic units in convoluntional networks, batch size=64, chan- nels=3 and number of layers= 18 unless appears in the axis
Table 1I: Network architecture for COU and Lon Cone denotes using
ReLU for the activation functions and Lonn denotes using Tanh as activation functions
Layer
Type
Params
conv1
Conv2d
out_channels=32, simel_size=3, stride=1, padding=2
bnl
BatchNorm2d
N/A
actl
ReLUUTanh
N/A
pooll
Maxpool12d
sermel_siz==2,, stride=2, padding=0, dilation=1
conv2
Conv2d
out_channels=64, kernel_size=13 stride=1, padding=2
bn2
BatchNorm2d
N/A
act2
ReLU/Tanh
N/A
pool2
Maxpool12d
stride=2, padding=0, dilation=1
conv3
Conv2d
out_channels=128, kernel_size=3, stride=1, padding=2
bn3
BatchNorm2d
N/A
act3
ReLU/Tanh
N/A
poo13
Maxpool12d
kernel_size=2, stride=2, padding=0, dilation=1
conv4
Conv2d
out mw__channels=256,, kernel size=3, stride=1, padding-2 <emmel_sie=33,,
bn4
BatchNorm24
N/A
act4
ReLU/Tanh
N/A
pool4
Maxpooll2d
kernel_size=2, stride=2, padding=0, dilation=1
conv5
Conv2d
out channels=512, kernel_size=3, stride=1, padding-2
bn5
BatchNorm2d
N/A
act5
ReLU/Tanh
N/A
pool5
Maxpool12d
kerne__size=2, stride=2, padding=0, dilation=1
flatten
Flatten
N/A
fel
Linear
out features= 10
28


Reviews:
[{"review_id": "QeaqsPXbzH", "reviewer": "Reviewer_empf", "summary": "This paper proposes a novel forward gradient descent method named AsyncFGD to decouple dependencies between layers and thus maximize parallel computation. The authors demonstrate that their method can reduce memory consumption and enhance hardware efficiency through empirical evaluations on AGX Orin.", "questions": "Could this method also improve the energy efficiency? Could there be some comparison, even if basic analysis?", "limitations": "This paper did not talk about limitations. One potential issue might be related to the environment issue, like carbon emission of the proposed method compared to conventional ones.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method of using asynchronized forward gradient descent is interesting and effective, and the authors provide both theoretical analysis and empirical verification, where the latter includes both accuracy and efficiency. The authors also provided extensive analysis and detailed setup.", "weaknesses": "One potential issue is that it is not very clear if this work is very suitable for a machine learning venue, as a large portion of the paper are talking about architecture- or system-level issues. Also, from Table 1, it seems the proposed method is not able to achieve consistently better results,like the sDFA or DFA can be better for MNIST, among others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer empf,\n\nWe sincerely thank you for your comprehensive review and the constructive feedback you provided. Your insights are invaluable, and we have made efforts to address each of the concerns and questions you raised.\n\n**Weaknesses:**\n\n**Q1:** \"Unclear Relevance to Machine Learning Venue.\"\n\n**A1:** You highlighted that a significant portion of our paper focuses on architecture- or system-level issues, which might seem tangential to a machine learning venue. We understand this perspective. However, the system we are exploring possesses unique properties intrinsically tied to Machine Learning. For instance, the layer-wise dependency in each iteration and the iteratively based workload are not typical of arbitrary system processes. Moreover, the strategies we employed to loosen the dependencies among workers are fundamentally rooted in ensuring the convergence of a Machine Learning Algorithm, as evidenced by our comprehensive proof, thus also indicating that we are trying to acclerate FGD from a more algorithmetic perspective. Nevertheless, we acknowledge your feedback and will strive to align our discussions more closely with the machine learning context in the revised manuscript.\n\n**Q2:** \"Inconsistency of Advantageous Results of AsyncFGD in Table 1.\"\n\n**A2:** We value your observation of the advantage of AsyncFGD over other algorithms. To clarity, the target of training on edge device typically not in a train-from-strach, but a online learning fashion, with previous experince and incoming samples from conditions where the device operates. Therefore, our initial Table 1 results may not fully capture AsyncFGD's strengths. Additional experiments simulating online learning offer a more fitting comparison, demonstrating that AsyncFGD's unbiased gradient estimation surpasses other BP-free algorithms relying on random feedback weights.\n\n**Questions:**\n\n1. **Could this method also improve the energy efficiency? Could there be some comparison, even if basic analysis?**\n\n   You raised an insightful query regarding our method's potential to enhance energy efficiency. We believe that online learning on edge devices, where a combination of previous experience (weights) and incoming samples are processed, is an important implementation scenario for both on-device learning and our AsyncFGD. In such a context, our method offers efficiency by requiring fewer computations per input sample. Specifically, AsyncFGD needs only 2 GEMM (General Matrix Multiply) operations per layer—one for the forward pass and the other for the Jacobian-vector product (JVP)—compared to 3 for BP. This reduction in total GEMMs per input sample could translate into potential energy savings. We recognize the importance of this aspect and will include a more detailed analysis, along with benchmarks on different platforms, in the revised version of our paper.\n2. **One potential issue might be related to the environment issue, like carbon emission of the proposed method compared to conventional ones.**\n\n   You raise an excellent point about potential environmental impacts like carbon emissions. Analyzing this aspect is indeed complex, as many factors contribute to the overall environmental footprint. On one hand, AsyncFGD might produce more emissions due to its emphasis on maximizing worker utilization, extra context switching, and random tangent regeneration. On the other hand, leaving workers idle and prolonging training can also waste energy, contributing to extra emissions. Thus, there exists a nuanced tradeoff between emissions from longer training time and emissions from additional computations. To provide a comprehensive understanding, we will conduct benchmarks to compare energy consumption and emissions for AsyncFGD against conventional methods across various scenarios. This analysis will enable us to assess the full environmental impact of our method. We appreciate your valuable suggestion to examine this critical aspect, and we are committed to addressing it in our revised manuscript.\n\n   We trust that our responses address the concerns and questions you raised. We are committed to refining our work based on your feedback, confident that the revisions will bolster the quality of our paper. Once again, we extend our gratitude for your constructive comments.\n\nWarm regards"}, {"review_id": "7peIFrC4AE", "reviewer": "Reviewer_Kesv", "summary": "This paper proposes an asynchronous version of the forward gradient descent (FGD) method, in order to alleviate the forward locking exhibited with FGD and enable more efficient implementation. Specifically, AsyncFGD decomposes a network into K decoupled subnetworks that work as a delayed pipeline and enable asynchronous forward propagation. \nThe authors benchmark their method on an NVIDIA AGX Orin, using a variety of network architectures and datasets. Their method seems to perform better in the fine-tuning setup, where learning rates are small, due to the high variance of forward gradient methods in general.", "questions": "- typo in line 104: define \n- What is the dataset being evaluated in Fig 3c? \n- Why isn't the efficient learning strategy applied to FGD in Table 2? ", "limitations": "No discussion on limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well motivated, and tackles an important subject: efficient on-device training\n- The authors properly explain their proposed method, the presentation is clear\n- The theoretical analysis on the convergence of the proposed method is appreciated\n- The authors benchmark their method on actual device and report measured performance", "weaknesses": "\n- The original FGD paper [1] does not report improvements in memory consumption (see Fig 6). I find it interesting that the proposed method does not suffer from the same. I also found that the overall memory discussion a bit lack-luster. It would greatly improve the quality of this paper, and emphasizes the importance of the proposed method, if some details were provided on how memory consumption is computed (or measured?) for all methods (BP, FGD, and AsyncFGD). How bad is the additional memory overhead due to the proposed asynchronous implementation?\n\n- The method has not been benchmarked on big datasets, such as ImageNet. Furthermore, it appears that, much like other forward methods, AsyncFGD cannot scale well to larger problems. This explains why the authors focused on transfer learning applications. This, in my opinion, limits the usefulness of such methods.\n\n\n\n[1] https://arxiv.org/pdf/2202.08587.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Kesv,\n\nFirstly, we would like to express our gratitude for the meticulous review and valuable feedback you provided on our paper. Your insights are instrumental in refining our work, and we have made concerted efforts to address each of the concerns you highlighted.\n\n**Weaknesses:**\n\n**Q1:**: \"Why AsyncFGD's memory consumptino is better than that of the origianl paper, how is the memory mesured or computed?\"\n\n**A1**: We appreciate you noting the difference in memory consumption between our paper and the original FGD paper [1]. A potential explanation is that the original paper relied on an early beta version of functorch, which may have led to inaccurate measurements. This could explain the perfect alignment of their BP and FGD memory lines. In our study, we used PyTorch's built-in profiler to precisely measure peak memory allocation during optimization. This allowed us to obtain more reliable memory consumption results. Theoretical computation for memory consumption will also be provided in the revised manuscript.\n\n**Q2:** \"Worry about scalability to larger datasets.\"\n\n**A2:** We acknowledge your valid concerns regarding the applicability of AsyncFGD to larger datasets, such as ImageNet. While our initial focus was on specific use cases, we have broadened our experiments to encompass ImageNet in response to your feedback. These findings are detailed in Table 2 of the rebuttal PDF. We recognize a performance gap between AsyncFGD and BP, particularly due to the variance introduced by random perturbation. However, this challenge can be mitigated in online training and adaptive learning scenarios. In such contexts, edge devices initialized with pre-trained weights need to further adapt to different conditions, like auto-driving models, where devices must adapt to specific road conditions using existing knowledge from previously learned experiences, making our method more applicable. Moreover, our explorations into variance reduction with efficient transfer learning are positive. We are also buoyed by parallel efforts of variance reduction for FGD for large scale model and datasets[2].\n\n**Questions:**\n\n1. **Typo in Line 104**: We apologize for the oversight and have corrected the typo in line 104. We appreciate your attention to details.\n2. **Unclear Dataset in Fig 3c**: The dataset evaluated in Fig 3c is CIFAR-10. We recognize that this have not been explicitly mentioned in the original manuscript, and we assure that the necessary clarifications will be incorporated in our revised version.\n3. **Efficient Learning Strategy in Table 2**:\n\n   We apologize for any ambiguity caused by the presentation in Table 2. Our primary objective was to illustrate that AsyncFGD can achieve performance metrics comparable to FGD in transfer learning tasks, even when working with stale parameters. The introduction of the AsyncFGD* column aimed to juxtapose it with AsyncFGD, emphasizing that when paired with efficient transfer learning, our method's accuracy can be further bolstered due to diminished variance. Hence, we did not apply the efficient transfer learning strategy to FGD.\n\nIn conclusion, we genuinely hope that our clarifications address the concerns you raised. We remain committed to refining our work based on your invaluable feedback and are confident that our revisions will elevate the quality of our paper. Once again, we extend our heartfelt thanks for your constructive insights.\n\nWarm regards\n\n[1] [https://arxiv.org/pdf/2202.08587.pdf](https://arxiv.org/pdf/2202.08587.pdf)\n\n[2] Ren M, Kornblith S, Liao R, et al. Scaling forward gradient with local losses[J]. arXiv preprint arXiv:2210.03310, 2022."}, {"review_id": "7LhabaonE4", "reviewer": "Reviewer_yH2P", "summary": "ThIS paper introduces AsyncFGD, a forward-only based training method. The key idea behidn this paper is integrating asynchronous updates with Forward Gradient Descent (FGD). The authors test their method on several small-scale datasets.", "questions": "- Can the author explain the fundamental difference between the original forward-only training methods by introducing a pipeline mechanism among different layers and this methods?\n- Can the author show the large-scale dataset experiments?", "limitations": "Please refer to the weakness sections. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed proof of the convergence guarantee of AsyncFGD-SGD is new.\n- According to the experiment results, this framework achieves a good acceleration rate on small-scale datasets.", "weaknesses": "- The novelty of this approach is not introduced clearly. There is no fundamental difference between the original forward-only training methods by introducing a pipeline mechanism among different layers and this methods.\n- The experimental results are only shown on small-scale datasets.\n- The comparison with other works is not sufficient [1].\n- The proposed method has on-par memory consumption with BP based training yet with lower accuracy. The memory cost is crucial for some edge devices.\n\n[1] Krithivasan S, Sen S, Venkataramani S, et al. Accelerating DNN Training Through Selective Localized Learning[J]. Frontiers in Neuroscience, 2022, 15: 759807.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer yH2P,\n\nThank you for your comprehensive review and the valuable insights you provided. We genuinely appreciate your feedback, as it offers us a clear direction for refining our work. We have taken the time to address each of your concerns in detail.\n\n**Weaknesses:**\n\n**Q1:** \"Unclear novelty and distinction from original forward-only training Methods with pjipeline parallelism.\"\n\n**A1:**  Thank you for your insightful question. Our method's novelty lies in its design to detach dependencies in each module during the forward pass, most importantly, with theoretical grantee and performance comparable to the original version. Specifically:\n\n- Dependencies among layers are preserved within each iteration.\n- The staleness of each module is bounded on a module-wise basis.\n\nOur approach's primary goal is to ease the dependency in the forward pass from an iterative optimization standpoint. Supported by our mathematical proof, it converges to critical points under non-convex conditions. Though the asynchronous pipeline parallelism that results from this relaxed constraint may seem similar to merely adding pipeline parallelism to forward gradient, it stems from a fundamentally different motivation and design philosophy. Moreover, it is worth noting that our method is the first to provide asynchronous acceleration on FGD.\n\n**Q2:** \"Experimental results focus on small-scale datasets.\"\n\n**A2:** We recognize your point regarding the scale of our datasets. To address this, we have conducted additional experiments on ImageNet, ILSVRC2012 datasets. While training the model from scratch led to a gap in accuracy compared to BP, we found that in online training scenarios, efficient transfer learning strategy can be employed to reduced this difference. As detailed in Table 2 of the rebuttal PDF, our AsyncFGD method, when combined with this strategy (Async$^*$), outperformed other BP-free algorithms on these larger datasets, approaching closer to BP's accuracy.\n\n**Q3:** \"Insufficient comparison with other works like [1].\"\n\n**A3:** We appreciate your emphasis on a more comprehensive comparison with other works, especially the one mentioned [1]. In response, we conducted further experiments comparing the two methods on ResNet-18 with CIFAR-10 and MNIST. Our results, which are detailed below, reveal that the maximum memory consumption of [1] aligns with BP. However, a direct comparison of accuracy is not entirely equitable, as [1] employs a hybrid approach that combines Forward-only and backpropagation methods. To provide a more nuanced comparison, we also adjusted the ratio of BP-pretrained weights (counted from the front) and presented the findings in the ratio column. This approach ensures a more balanced evaluation, reflecting that our method can achieve comparable accuracy with less memory consumption. \n\n   *Accuracy and Memory Consumption on CIFAR-10*\n\n|  ALGORITHM  | MAX MOMORY CONSUMPTION | RATIO | ACC |\n| :----------: | :--------------------: | :---: | :--: |\n|      BP      |          3128          |   -   |  93  |\n|  LoCal+SGD  |          3128          |   -   | 92.8 |\n|     FGD     |          1125          |  0.0  | 45.7 |\n| Async (k=3) |          1187          |  0.0  | 44.8 |\n| Async (k=3) |          1387          |  0.1  | 48.3 |\n| Async (k=3) |          1754          |  0.3  | 64.1 |\n| Async (k=3) |          2247          |  0.5  | 88.3 |\n| Async (k=3) |          2849          |  0.8  | 92.4 |\n\n   *Accuracy and Memory Consumption on MNIST*\n\n|  ALGORITHM  | MAX MOMORY CONSUMPTION | RATIO | ACC |\n| :---------: | :--------------------: | :---: | :--: |\n|     BP     |          3120          |   -   | 98.5 |\n|  LoCal+SGD  |          3120          |   -   | 98.3 |\n|     FGD     |          1120          |  0.1  | 63.2 |\n| Async (k=3) |          1179          |  0.0  | 65.4 |\n| Async (k=3) |          1347          |  0.1  | 69.3 |\n| Async (k=3) |          1788          |  0.3  | 84.9 |\n| Async (k=3) |          2043          |  0.5  | 97.3 |\n| Async (k=3) |          2831          |  0.8  | 98.4 |\n\n**Q4:** \"The proposed method has on-par memory consumption with BP based training yet with lower accuracy. The memory cost is crucial for some edge devices.\"\n\n**A4**: We appreciate your observation regarding the memory consumption of our method compared to BP-based training. Indeed, the gap between BP and AsyncFGD narrows when BP-based methods utilize re-materialization techniques [2, 3], in MLP or RNN-based networks. However, as detailed in Figure 1 of the rebuttal PDF and the Appendix in the original paper, it's essential to highlight that AsyncFGD presents a distinct advantage in convolutional layers, which often constitute the majority of computation in networks designed for edge devices. This advantage arises from the intrinsic duplication of the kernel matrix during the forward pass and the fact that the primary memory consumption in AsyncFGD is for placeholders of random tangents of parameters. These factors contribute to the efficiency of our method in convolutional layers. This makes AsyncFGD particularly suitable for edge devices, where memory cost is a crucial consideration.\n\n**Questions:** Please refer to weaknesses part. \n\nWe trust that our detailed responses offer clarity on the concerns you raised. Once again, we extend our gratitude for your constructive feedback.\n\nWarm regards\n\n[1] Krithivasan S, Sen S, Venkataramani S, et al. Accelerating DNN Training Through Selective Localized Learning[J]. Frontiers in Neuroscience, 2022, 15: 759807.\n\n[2] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016.\n\n[3] Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. Advances in Neural Information Processing Systems, 29, 2016."}, {"review_id": "HIm0J6Pdpd", "reviewer": "Reviewer_bhPv", "summary": "The paper focuses on enabling training on memory-constrained platforms. Instead of using conventional backpropagation-based methods, the proposed approach is based on forward gradient descent (FGD), which approximates the gradients through only forward passes. The paper claims that a brute-force utilisation of FGD leads to resource underutilisation in memory-constrained devices. To counteract this, the work proposes AsyncFGD, a method that enables multiple parallel workers to process different parts of the network using different samples, while allowing varying levels of staleness in the parameters of these parts. ", "questions": "1) Is something accidentally missing from Section 6.1?  \n2) Please also see points 5) and 6) above for things that were not clear.  \n3) Please proof-read the paper as there are some typos.\n", "limitations": "The paper does not explicitly discuss its limitations.\n\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1) Reducing the memory requirements of the training stage is a key and still open challenge towards enabling on-device training. In this context, FGD-based algorithms constitute an interesting approach that is still largely unexplored.  \n2) The paper provides both theoretical analysis and empirical evaluation of the proposed approach. ", "weaknesses": "1) In the motivation of work, the paper makes a number of arguments that are hard to justify. In Section 2.2, the paper claims that \"in edge devices with limited computational power, potential idle workers due to synchronization in synchronous pipelines like [12] are not optimal.\" However, this is rarely, or not at all, the case. Specifically, edge devices typically utilise their processing resources fully by exploiting the intra-layer parallelism, *e.g.* by performing convolutions or matrix multiplications. For the processor of such platforms to remain underutilised, either the device lies more towards server-grade or the target model is very lightweight, with small layers that contain minimal parallelism. This could make more sense in server settings, where layer pipelining has already been proposed (GPipe, PipeDream) as also mentioned in the paper. Therefore, the argument about Forward Locking and the serialisation of computations in the forward pass of FGD-based methods does not seem to be relevant for memory-constrained edge devices, where layer pipelining is hardly used during training.\nAs such, there are important questions with respect to the suitability of AsyncFGD. The presented implementation uses multiple workers to process different parts of the network, *i.e.* it employs module pipeling. For this to make sense as a strategy, the computational resources of the target device must be underutilised in the first place, *i.e.* when module pipelining is not used. In this context, what about batching? Batch processing is not touched upon in the paper. Why not have different workers processing different samples from the same batch, thus not requiring any stale parameters?  \n2) In Section 4.3, the paper makes a claim with respect to the attainable speedup. Specifically, it states that \"It is easy to see that AsyncFGD can fully utilize the computation resources, thus achieving considerable speedup\". This statement is very strong and cannot be supported solely by Fig. 2. In Fig. 2, we see a simple performance model for AsyncFGD, which indicates the potential acceleration *under certain conditions*. These conditions include (among others) *i)* whether the processor is really underutilised, *i.e.* whether there are really free cycles to have more than one module running, and *ii)* whether the multiple workers become external memory bandwidth-bound, in which case AsyncFGD can lead to slowdown - this is very common in edge devices where multiple components of the target chip share the same bandwidth to the external memory. In the scope of this work that aims at memory-constrained devices, the assumption that the memory bandwidth is abundant contradicts the whole setup. Overall, the logic of the paper with respect to acceleration leaves out many important considerations.  \n3) In Section 6.6, one would expect the speedup to plateau or even lead to slowdown, when *K* increases above a certain point, where the workers would start to context-switch excessively. How is *K* selected? What exactly affects it? Both the model architecture, the target dataset and the target platform would play important roles (see also point 2 above on this). In its current version, the paper is missing a thorough investigation of this fact.  \n4) In Section 6.2, while the objective of the experiments and the selection of baselines are sound, the network architectures are arbitrary and simple. It is hard to draw conclusions on the efficacy of AsyncFGD.  \n5) In Section 6.4, it is not clear what the setup is. The models are pre-trained on ImageNet and then fine-tuned on a given dataset using one of the algorithms in Table 2. Why does AsyncFGD* give higher accuracy than FGD and plain AsyncFGD? Furthermore, even Async* falls short compared to the BP-based baseline. Especially, given the issues of point 1) above, it is not evident that AsyncFGD has useful merits or under which conditions it would be a better choice than competing methods.\n6) In Section 6.5, the memory footprint comparison between FGD and AsyncFGD is missing in Fig. 3a and b - how do they compare? Importantly, how does the selection of *K* affect the memory footprint? Having more parallel workers can have a significant impact on memory consumption, through thread bookkeeping and cache misses.  \n7) As specified in the Appendix, only a single random seed is used in the experiments.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bhPv,\n\nThank you for your comprehensive review and insightful comments. We have carefully considered your feedback and offer the following responses:\n\n**Weaknesses:**\n\n**Q1:** \"Does idle worker really exists when training on Edge device? If so, why not batching inputs to parallel computation among workers ?\"\n\n**A1:** Your insightful question touches on two critical dimensions of efficient training on edge devices: the existence of idle workers and the potential for parallel computation through batching. Let's delve into these aspects:\n\n1. **Existence of Idle Workers:** Our method, although not a one-size-fits-all solution, can specifically addresses scenarios where edge devices, despite unused computational power, face memory constraints with BP-based algorithms, i.e. mismatch for parallisim and memory consumption. Specifically, temporal models like RNNs optimized by BPTT or SNNs optimized by Spatio-temporal backpropagation (STBP), can be highly memory-intensive. On edge devices, this often leads to reaching memory limits before fully utilizing all computational resources, resulting in idle workers, for example, on AGX Orin, which is employed in our experiment,  its typical when RNN or SNN dealing inputs with more than 300 timestamps, the overall utilization of computing power only takes 40% without any spare memory to further utilize the idle workers. As emphasized in [1], the substantial memory demands pose a challenge in implementing RNNs on edge devices. AsyncFGD, our solution, employs Forward Gradient to relax memory constraints, thus fully utilizing otherwise idle workers.\n2. **Batch Processing Considerations**: We concur with your insights on the merits of batch processing. While it is effective for historical data, there exist real-time scenarios, especially when edge devices are interfaced with specific sensors, that necessitate sequential data processing:\n\n   - **Real-time Adaptation and Streaming Data Learning**: Edge devices might need to swiftly adapt to and learn from newly acquired data in real-time fashion [2]. Moreover, when processing sequential information with temporal connections, as collected directly by sensors [3, 4], batching could interfere with these inherent relationships, disrupting the learning process [5].\n   - **Data Privacy and Security**: For applications dealing with sensitive data, storing it for batching could introduce privacy and security vulnerabilities. Immediate processing, followed by data disposal, can mitigate these risks.\n   - **Energy Efficiency**: Given the energy constraints typical of edge devices, batching can sometimes be more energy-intensive due to the demands of memory storage and data retrieval. In contrast, processing data as it arrives can be more energy-efficient in specific scenarios.\n\nIn summary, we view our method as a viable solution for specific situations, particularly when edge devices face memory constraints with BP-based algorithms. However, we acknowledge that it may not be suitable for all scenarios. Your valuable feedback is instrumental in refining our approach for the revised manuscript.\n\n**Q2:** \"Claims about Attainable Speedup ignore certain considerations.\"\n\n**A2:** We recognize your feedback on our speedup claims, especially in the context of bandwidth and computing power bottlenecks. In our experiments, we demonstrated positive results on AGX Orin, an edge device used in applications such as autonomous driving, robotics, and drones [7]. AGX Orin's off-chip bandwidth supports enough throughput, and its on-chip bandwidth facilitates communication between different workers on the chip. These features align with our AsyncFGD method, allowing for maximum utilization of computing resources and achieving acceleration. Given the broad definition and varied capabilities of edge devices, we acknowledge that our method might not achieve speedup on IoT-targeted chips, where bandwidth and computing power are restricted to even deploy a deep learning model. However, we believe our method has its merits in certain contexts, especially considering the rapid advancements in edge device's bandwidth, as evidenced by High Bandwidth Memory FPGAs. In light of your practical considerations, we will refine our claims and narrow the scope of our research in the revised manuscript. Your insights have been instrumental, and we appreciate your constructive feedback.\n\n**Q3:** \"How is $K$ selected, considering model architecture, dataset, and platform?\"\n\n**A3:** We appreciate your inquiry about the selection of $K$. Our extra experiments, detailed in Table 3 of the rebuttal PDF, confirm that as $K$ increases, the overheads can outweigh the benefits, leading to a plateau or even slowdown. This often happens when $K > 4$. Thus, the choice of $K$ requires careful consideration of the model's complexity, data characteristics, and platform constraints and we will add more experiments in the revised manuscript. Thank for your observation that has guided us to refine our work.\n\n**Q4:** Section 6.2's simple, arbitrary network architectures hinder AsyncFGD conclusions.\n\n**A4:** We apologize for any confusion. For comparisons with other BP-free algorithms, we intentionally used simpler network architectures in Section 6.3, as architectures used in the original papers are rather simple (for example, 2$\\times$ 1000 FC layers in DFA [8] and two convolutional layers with 64 and 256 channels with $3 \\times 3$ kerne in DRTP [9]). For efficient transfer learning in Table 2, and Fig 3(c), we selected widely-accepted lightweight models to represent a broader range of efficient models suitable for edge devices.\n\nWe have reached the page limit. Please refer to the global author rebuttal for the remainng."}]

Author rebuttal:
### Sincere Gratitude
Dear All Reviewers and Program Chairs of NeurIPS 2023,

We would like to extend our heartfelt gratitude to each of you for the time, effort, and expertise you have invested in reviewing our manuscript. Your constructive feedback, insightful comments, and valuable suggestions have been instrumental in identifying areas for improvement and refinement.

We have taken all your comments into careful consideration and have made corresponding revisions to address the concerns and enhance the quality of our work. We believe that these changes not only clarify our contributions but also strengthen the overall impact and relevance of our research.

We sincerely appreciate the opportunity to engage in this collaborative process and look forward to any further feedback you may have. Your dedication to maintaining the rigor and integrity of the scientific process is truly commendable, and we are honored to be part of this scholarly community.

Thank you once again for your thoughtful review.

Warm regards

### Remaining Rebuttal
#### Remaining Rebuttal for Reviewer bhPv:

**Q5:** "Section 6.4 unclear for AsyncFGD* accuracy, Async* falls short of BP-baseline."

**A5**: The enhanced accuracy of AsyncFGD* is a result of our efficient transfer learning strategy, which effectively reduces variance. Historically, we've observed that FGD suffer more from variance than optimizing on the subset of parameters. While our method's accuracy might trail BP, we have acknowledged this limitation in our conclusion. We are also optimistic about ongoing efforts to further reduce the variance of forward gradient [6].

**Q6:**: "Section 6.5 lacks memory footprint comparison; how does K affect it?"

**A6:** In response to your query, we conducted further experiments. As detailed in Table 4 of the rebuttal PDF, our findings reveal a 30% increase in memory usage when $K=4$. These results emphasize the importance of carefully selecting the value of $K$, considering both memory constraints and performance goals. Please refer to the rebuttal PDF for a comprehensive overview.

**Questions:**

1. **Missing Content in Section 6.1:** Our apologies for the oversight. We inadvertently added a redundant subsection 6.1.
2. **Clarifications on Sections 6.5 and 6.6**: We appreciate your questions regarding Sections 6.5 and 6.6. We have answered them above.
3. **Proofreading and Typos**: We apologize for any oversight in proofreading. We have taken your feedback into account and have thoroughly reviewed the paper to correct any typos.

We hope our responses address your concerns adequately. We're committed to refining our work based on your feedback and believe that the revisions will enhance the paper's quality. Once again, thank you for your constructive comments.

Warm regards

[1] Rezk N M, Nordström T, Ul-Abdin Z. Shrink and eliminate: A study of post-training quantization and repeated operations elimination in RNN models[J]. Information, 2022, 13(4): 176.

[2] Pellegrini L, Lomonaco V, Graffieti G, et al. Continual learning at the edge: Real-time training on smartphone devices[J]. arXiv preprint arXiv:2105.13127, 2021.

[3] Hagenaars J, Paredes-Vallés F, De Croon G. Self-supervised learning of event-based optical flow with spiking neural networks[J]. Advances in Neural Information Processing Systems, 2021, 34: 7167-7179.

[4] Schaefer S, Gehrig D, Scaramuzza D. Aegnn: Asynchronous event-based graph neural networks[C]. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 12371-12381.

[5] Pan Y, Cheng C A, Saigol K, et al. Agile autonomous driving using end-to-end deep imitation learning[J]. arXiv preprint arXiv:1709.07174, 2017.

[6] Ren M, Kornblith S, Liao R, et al. Scaling forward gradient with local losses[J]. arXiv preprint arXiv:2210.03310, 2022.

[7] [DRIVE Hyperion Autonomous Vehicle Development Platform | NVIDIA Developer](https://developer.nvidia.com/drive/hyperion)

[8] Nøkland A. Direct feedback alignment provides learning in deep neural networks[J]. Advances in neural information processing systems, 2016, 29.

[9] Frenkel C, Lefebvre M, Bol D. Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks[J]. Frontiers in neuroscience, 2021, 15: 629892.

### Additional PDF
We have included a PDF file containing the majority of the results from our additional experiments. This supplementary material offers an extended view that contributes to a more comprehensive understanding of our research.

Meta-review: This paper introduces AsyncFGD, an asynchronous adaptation of forward gradient descent aimed at mitigating memory constraints in on-device learning. Supported by robust theoretical analysis and empirical tests on cutting-edge hardware, AsyncFGD succeeds in both minimizing memory usage and enhancing hardware efficiency. After a thorough review, there's a consensus among reviewers to accept the paper. Although I noted a significant accuracy gap in the ImageNet results, the innovative development of an asynchronous algorithm for scalability is an exciting direction. I suspect that some limitations might stem from the nature of forward gradient descent itself, and it would enrich the paper to discuss the differences between backpropagation and forward gradient in the related work section. Given the paper's novel approach and strong justification, I recommend its acceptance.
Decision: Accept (poster)