SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 5.6. Assume Assumption 5.1 and 5.2 hold and the fixed stepsize sequence  $\{\gamma_{t'}\}$  satisfies  $\gamma_{t'} = \gamma \leq \frac{1}{L}, \forall t' \in \{0,1,\dots,T-1\}$ . In addition, we assume  $w^*$  to be the optimal solution to  $f(w)$  and let  $t' = t - K + 1$ ,  $\sigma = 1$  such that  $M_K = KM + K^4 M$ . Then, the output of Algorithm 1 satisfies that:
$$
\frac {1}{T} \sum_ {t ^ {\prime} = 0} ^ {T - 1} \mathbb {E} \| \nabla f \left(w ^ {t ^ {\prime}}\right) \| ^ {2} \leq \frac {2 \left(f \left(w ^ {0}\right) - f \left(w ^ {*}\right)\right)}{\gamma T} + 4 (d + 4) L \gamma M _ {K}. \tag {13}
$$
Theorem 5.7. Assume Assumption 5.1 and 5.2 hold and the diminishing stepsize sequence  $\{\gamma_{t'}\}$  satisfies  $\gamma_{t'} = \frac{\gamma_0}{t' + 1} \leq \frac{1}{L}$ . In addition, we assume  $w^*$  to be the optimal solution to  $f(w)$  and let  $t' = t - K + 1$ ,  $\sigma = K$  such that  $M_K = KM + K^5 M$ . Let  $\Gamma_T = \sum_{t'=0}^{T-1} \gamma_{t'}$ , the output of Algorithm 1 satisfies that:
$$
\frac {1}{\Gamma_ {T}} \sum_ {t ^ {\prime} = 0} ^ {T - 1} \gamma_ {t ^ {\prime}} \mathbb {E} \| \nabla f \left(w ^ {t ^ {\prime}}\right) \| ^ {2} \leq \frac {2 \left(f \left(w ^ {0}\right) - f \left(w ^ {*}\right)\right)}{\Gamma_ {T}} + \frac {\sum_ {t ^ {\prime} = 0} ^ {T - 1} 4 (d + 4) \gamma_ {t ^ {\prime}} ^ {2} L M _ {K}}{\Gamma_ {T}} \tag {14}
$$
Remark 5.8. Since the stepsize sequence  $\gamma_{t} = \frac{\gamma_{0}}{t + 1}$  satisfies that  $\lim_{T\to \infty}\sum_{t = 0}^{T - 1}\gamma_t = \infty$ $\lim_{T\to \infty}\sum_{t = 0}^{T - 1}\gamma_t^2 < \infty$ , when  $T\to \infty$ , the RHS of Eq.(14) will converge to 0. Let  $w^{s}$  be randomly chosen from  $\{w^{t'}\}_{t' = 0}^{T - 1}$  with probabilities proportional to  $\{\gamma_{t'}\}_{t' = 0}^{T - 1}$ , then we have  $\lim_{s\to \infty}\mathbb{E}\| \nabla f(w^s)\|^2 = 0$ .
This section embarks on a meticulous examination of our proposed method, AsyncFGD. We assess its performance through three distinct facets: memory consumption, acceleration rate, and accuracy. We initiate our analysis by outlining our experimental setup. To validate the efficacy of applying directional derivatives and utilizing module-wise stale parameters, we contrast AsyncFGD with an array of alternative methods encompassing backpropagation, conventional FGD, and other backpropagation-free algorithms. Subsequently, our focus shifts towards scrutinizing the potential of AsyncFGD within the sphere of efficient transfer learning, conducting experiments on prevalent efficient networks. Memory consumption represents another cardinal aspect that we explore, benchmarking AsyncFGD against popular architectures and unit layers like fully-connected layers, RNN cells, and convolutional layers. Concluding our empirical investigation, we assess the speedup of our method relative to other parallel strategies under a diverse set of conditions across multiple platforms.
6.1 Experimental Setup
Methods. We contrast our proposal's memory footprint with Backpropagation, Sublinear [3], Backpropagation through time, and Memory Efficient BPTT [7]. Accuracy-wise, we compare with backpropagation-free methods: Feedback Alignment (FA) [27], Direct Feedback Alignment (DFA) [26], Direct Random Tangent Propagation, and Error-sign-based Direct Feedback Alignment (sDFA) [6]. We also apply parallelization to FGD through FGD-DP (data parallelism) and FGD-MP (model parallelism).
Platform. Experiments utilize Python 3.8 and Pytorch, primarily on nVidia's AGX Orin. Additional results on alternate platforms are in the appendix.
Training. Batch size is 64 unless noted. The optimal learning rate (chosen from  $\{1e - 5,1e - 4,1e - 3,1e - 2\}$  with Adam optimizer [18]) is based on validation performance. The parameter  $\alpha$  is initially set to 1 for the classifier, with others at 0 for the first 10 epochs. Subsequently,  $\alpha$  is gradually increased to 0.15 for specific layers. More details are in the appendix.
6.2 Effectiveness of Directional Derivative and Asynchronism
This section documents our experimentation on the effectiveness of using random directional derivative to approximate the true gradient by contrasting it with other BP-free algorithms. Furthermore, we demonstrate that the consistency remains intact when using module-wise stale parameters to uncouple the dependencies, comparing AsyncFGD with vanilla FGD. Results in Table 1 indicate that AsyncFGD can produce results closely aligned with vanilla FGD. Notably, FGD and AsyncFGD yield
Table 1: Comparison for AsyncFGD with other BP-free methods. ConvS and FCS refers to small convolutional network and full-connected network while ConvL and FCL refers to their slightly bigger counterpart. Different activation functions are marked as subscript. Details of network architectures can be found in Appendix H.2
Dataset	Model	BP	BP-free
FA	DFA	sDFA	DRTP	FGD	Async
MNIST	ConvS\( _{Tanh} \)	98.7	88.1	95.9	96.8	95.4	94.6	94.4
ConvS\( _{ReLU} \)	99.2	12.0	11.5	13.8	13.6	95.5	95.5
ConvL\( _{Tanh} \)	99.3	8.7	92.2	93.4	92.6	94.4	94.2
ConvL\( _{ReLU} \)	99.3	89.7	93.0	93.1	93.2	93.0	93.2
FCS\( _{Tanh} \)	98.9	83.2	95.6	94.2	94.5	94.4	94.3
FCS\( _{ReLU} \)	98.5	8.8	10.0	10.8	9.9	93.6	93.7
FCL\( _{Tanh} \)	98.8	89.8	93.0	92.0	92.4	95.2	95.4
FCL\( _{ReLU} \)	99.3	86.3	93.8	94.3	94.1	94.8	95.3
CIFAR-10	ConvS\( _{Tanh} \)	69.1	33.4	56.5	57.4	57.6	46.5	46.0
ConvS\( _{ReLU} \)	69.3	12.0	11.5	10.8	12.0	40.0	39.7
ConvL\( _{Tanh} \)	71.0	40.4	42.0	43.6	44.1	47.3	47.3
ConvL\( _{ReLU} \)	71.2	40.4	42.0	43.6	44.1	44.2	44.1
FCS\( _{Tanh} \)	47.8	46.2	46.4	46.0	46.2	42.0	42.3
FCS\( _{ReLU} \)	52.7	10.2	12.0	10.0	10.3	43.7	43.6
FCL\( _{Tanh} \)	54.4	17.4	44.0	44.3	45.5	47.2	47.2
FCL\( _{ReLU} \)	55.3	40.4	42.0	43.6	44.1	46.0	46.7
the most optimal outcomes when the network is more complex, or when we employ ReLU as the activation function devoid of batchnorm layers (results from  $\mathrm{ConvS}_{ReLU}$  and  $\mathrm{FCS}_{ReLU}$ ), situations where FA, DFA and sDFA often fail to propagate an effective error message. Backpropagation results are also furnished solely as a reference to true gradient results. However, when the network grows larger, all BP-free algorithms grapple with variance. The use of larger networks results in only minor improvements compared to BP. We try to address this challenge through efficient transfer learning in the subsequent section.
(a) FC Layers
(b) RNN Cell
Figure 3: Memory footprint comparison across methods.  $\mathrm{Async}^{\dagger}$  is AsyncFGD without tangent checkpoint while  $\mathrm{Async}^{*}$  refers to AsyncFGD using efficient training strategy. (a) FC layer memory vs. layer count; (b) RNN memory vs. sequential length; (c) Accuracy vs. memory on efficient architectures.
(c) Popular Efficient Architectures
6.3 Efficacy of Efficient Transfer Learning
In this segment, we delve into the efficacy of amalgamating AsyncFGD with efficient transfer learning, focusing on popular architectures like ResNet-18(Res-18) [9]; MoblieNet (Mobile)[11]; MnasNet(Mnas)[36] and ShuffleNet(Shuffle)[45] with their lightweight counterpart. The models are fine-tuned with weights pre-trained onImagenet. AsyncFGD† denotes AsyncFGD utilizing the efficient training strategy. As can be observed from Table 2, the application of an efficient transfer learning strategy brings about substantial performance enhancement, yielding superior results
compared to training with perturbation on the full model. Ablation studies on  $\alpha$  provided in Appendix I.1 also shows that, compared to optimizing subset of the model, FGD suffers more from variance.
6.4 Memory Footprint
As illustrated in Fig 3(c), when we plot accuracy against memory consumption, it is evident that AsyncFGD employs approximately one-third of the memory while maintaining close accuracy. Further experimentation on memory consumption with respect to the computing unit reveals, as shown in Fig 3(a) and Fig 3(b), that the additional memory consumption in AsyncFGD mainly serves as placeholders for random tangents, while the JVP computation consumes a negligible amount of additional memory. Memory consumption of other basic units like CNN and batch norm layers, are provided in Appendix.
6.5 Acceleration on Input Stream
In this final section, we assess the acceleration of ResNet-18 with varying  $K$ . In this setting, the batch size is set to 4 to better reflect the mechanism of streamlined input on edge device. As demonstrated in 4, AsyncFGD, by detaching dependencies in the forward pass, can outperform other parallel strategies in terms of acceleration rate. While pipeline parallelism is fast, the locking within the forward pass still induces overhead for synchronization, ultimately leading to lower hardware utilization and speed. Results pertaining to different network architectures and other platforms like CPU and GPU as well as more generalized case for larger batch size can be found in the Appendix I.2.
Table 2: Results for different algorithms in transfer learning.  $\text{Async}^*$  refers to using efficient transfer learning strategy.
Dataset	Model	BP	FGD	Async	\( \text{Async}^* \)
MNIST	Res-18	98.5	90.6	90.4	96.4
Mobile	99.2	89.3	88.4	97.1
Efficient	99.2	90.4	90.1	95.9
Mnas	98.9	86.6	86.3	96.0
Shuffle	99.0	85.8	85.8	96.4
CIFAR	Res-18	93.0	71.2	71.2	87.8
Mobile	94.0	72.3	72.2	91.1
Efficient	94.9	70.2	70.1	90.2
Mnas	84.2	68.8	68.5	78.9
Shuffle	89.9	72.5	72.5	82.0
FMNIST	Res-18	94.2	80.2	80.2	88.0
Mobile	93.2	82.3	83.1	90.6
Efficient	92.8	79.8	79.8	90.4
Mnas	92.1	77.1	77.0	87.0
Shuffle	92.8	78.4	78.4	87.3
(a) Cluster with 4 GPUs
(b) Single Embedded Device
Figure 4: Comparison for acceleration of different parallel methods.
7 Limitations and Discussion
While AsyncFGD offers computational benefits, it is not without limitations. A key drawback is its inferior performance compared to traditional Backpropagation (BP). This performance disparity is largely attributed to the use of randomly sampled directional derivatives in the forward gradient computation, aligning AsyncFGD with Zeroth-Order (ZO) optimization methods and evolutionary strategies. This sampling introduces a significant source of gradient variance, a challenge that is part of a larger problem in the field of stochastic optimization. However, we take encouragement from
recent advancements aimed at reducing this variance, some of which have even facilitated the training of large-scale models [31].
Another constraint pertains to the availability of idle workers on edge devices—a condition that is not universally applicable given the wide variety of edge computing environments. These can span from IoT chips with limited computational resources, where even deploying a standard deep learning model can be problematic, to high-capacity micro-computers used in autonomous vehicles.
Nevertheless, our experimental findings suggest that AsyncFGD is particularly beneficial for specific edge computing scenarios. In such settings, it may serve as a viable alternative for reducing memory usage while fully leveraging available computational resources.
8 Conclusion
In the present paper, we introduce AsyncFGD, an innovative approach designed to shatter the shackles of locking within the forward pass in FGD. This is achieved by incorporating module-wise stale parameters, simultaneously retaining the advantage of minimized memory consumption. In the theoretical segment, we offer a lucid analysis of this partially ordered staleness, demonstrating that our proposed method is capable of converging to critical points even in the face of non-convex problems. We further extend our algorithm to efficient transfer learning by introducing a scale parameter. Our experimental reveals that a sublinear acceleration can be accomplished, without compromising accuracy as well as huge performance gain when utilizing efficient transfer learning strategy. While the exploration of large models employing extensive datasets will undoubtedly continue to rely on backpropagation [10], we assert that the potential of asynchronous algorithms predicated on forward computation should not be overlooked. It offers a promising avenue for fully harnessing limited resources in on-device learning scenarios.
[36] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile, 2019.
[37] Robert Edwin Wengert. A simple automatic derivative evaluation program. Communications of the ACM, 7(8):463-464, 1964.
[38] Darrell Whitley, Stephen Dominic, Rajarshi Das, and Charles W Anderson. Genetic reinforcement learning for neurocontrol problems. Machine Learning, 13:259-284, 1993.
[39] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
[40] An Xu, Zhouyuan Huo, and Heng Huang. On the acceleration of deep learning model parallelism with staleness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2088-2097, 2020.
[41] Bowen Yang, Jian Zhang, Jonathan Li, Christopher Ré, Christopher Aberger, and Christopher De Sa. Pipemare: Asynchronous pipeline parallel dnn training. Proceedings of Machine Learning and Systems, 3:269-296, 2021.
[42] Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Chris De Sa. Swalp: Stochastic weight averaging in low precision training. In International Conference on Machine Learning, pages 7015-7024. PMLR, 2019.
[43] Yukuan Yang, Lei Deng, Shuang Wu, Tianyi Yan, Yuan Xie, and Guoqi Li. Training high-performance and large-scale deep neural networks with full 8-bit integers. *Neural Networks*, 125:70–82, 2020.
[44] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021.
[45] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices, 2017.
[46] Qihua Zhou, Zhihao Qu, Song Guo, Boyuan Luo, Jingcai Guo, Zhenda Xu, and Rajendra Akerkar. On-device learning systems for edge intelligence: A software and hardware synergy perspective. IEEE Internet of Things Journal, 8(15):11916-11934, 2021.
A Appendix
The organization of the Appendix is as follows: Appendix sections B to F provide detailed proofs of the lemmas and theorems presented in the main text. This is followed by additional insights, including operational specifics of working with Adam and details on how AsyncFGD can be extended for parallel processing on recursive networks, covered in Appendix G. Lastly, Appendix H and I provide a comprehensive view of the training process and additional experimental results, respectively.
B Proof of Lemma 3.1
According to the update rule of Eq.(3), (4), we have
$$
h _ {1} = F _ {1} \left(h _ {0}, w _ {1}\right) = F _ {1} \left(x, w _ {1}\right)
$$
$$
o _ {1} = J _ {F _ {1}} \left(h _ {0}, w _ {1}\right) \left[ o _ {0} ^ {\top}, u _ {w _ {1}} ^ {\top} \right] ^ {\top} = J _ {F _ {1}} \left(h _ {0}\right) o _ {0} + J _ {F _ {1}} \left(w _ {1}\right) u _ {w _ {1}} = J _ {F _ {1}} \left(w _ {1}\right) u _ {w _ {1}}
$$
$$
h _ {2} = F _ {2} \left(h _ {1}, w _ {2}\right)
$$
$$
\begin{array}{l} o _ {2} = J _ {F _ {2}} \left(h _ {1}, w _ {2}\right) \left[ o _ {1} ^ {\intercal}, u _ {w _ {2}} ^ {\intercal} \right] ^ {\intercal} = J _ {F _ {2}} \left(h _ {1}\right) o _ {1} + J _ {F _ {2}} \left(w _ {2}\right) u _ {w _ {2}} = J _ {F _ {2}} \left(h _ {1}\right) J _ {F _ {1}} \left(w _ {1}\right) u _ {w _ {1}} + J _ {F _ {2}} \left(w _ {2}\right) u _ {w _ {2}} \\ = J _ {F _ {2}} \left(w _ {1}\right) u _ {w _ {1}} + J _ {F _ {2}} \left(w _ {2}\right) u _ {w _ {2}} \\ \end{array}
$$
$$
o _ {L} = \sum_ {l = 1} ^ {L} J _ {F _ {L}} (w _ {l}) u _ {w _ {l}}
$$
Then for any  $l \in \{1, 2, \dots, L\}$ , take expectation with respect to  $u_{w_l}$ , we have
$$
\mathbb {E} _ {u _ {w _ {l}}} \left(o _ {L} \cdot u _ {w _ {l}}\right) = \mathbb {E} _ {u _ {w _ {l}}} \left[ \left(J _ {F _ {L}} (w _ {l}) u _ {w _ {l}}\right) \cdot u _ {w _ {l}} + \sum_ {k \neq l} \left(J _ {F _ {L}} (w _ {k}) u _ {w _ {k}}\right) \cdot u _ {w _ {l}} \right]
$$
$$
\begin{array}{l} \mathbb {E} _ {u _ {w _ {l}}} \left[ \left(J _ {F _ {L}} (w _ {l}) u _ {w _ {l}}\right) \cdot u _ {w _ {l}} \right] = \mathbb {E} _ {u _ {w _ {l}}} \left(\left[ \begin{array}{c c c c} \frac {\partial J _ {F _ {L} , 1}}{\partial w _ {l , 1}} & \frac {\partial J _ {F _ {L} , 1}}{\partial w _ {l , 2}} & \dots & \frac {\partial J _ {F _ {L} , 1}}{\partial w _ {l , d _ {l}}} \\ \frac {\partial J _ {F _ {L} , 2}}{\partial w _ {l , 1}} & \frac {\partial J _ {F _ {L} , 2}}{\partial w _ {l , 2}} & \dots & \frac {\partial J _ {F _ {L} , 2}}{\partial w _ {l , d _ {l}}} \\ \vdots & \vdots & \ddots & \vdots \\ \frac {\partial J _ {F _ {L} , d _ {h _ {L}}}}{\partial w _ {l , 1}} & \frac {\partial J _ {F _ {L} , d _ {h _ {L}}}}{\partial w _ {l , 2}} & \dots & \frac {\partial J _ {F _ {L} , d _ {h _ {L}}}}{\partial w _ {l , d _ {l}}} \end{array} \right] \left[ \begin{array}{c} u _ {w _ {l}, 1} \\ u _ {w _ {l}, 2} \\ \vdots \\ u _ {w _ {l}, d _ {l}} \end{array} \right] \cdot \left[ \begin{array}{c c c} u _ {w _ {l}, 1} & u _ {w _ {l}, 2} & \dots u _ {w _ {l}, d _ {l}} \end{array} \right]\right) \\ = \mathbb {E} _ {u _ {w _ {l}}} \left(\left[ \begin{array}{c c c} \sum_ {i = 1} ^ {d _ {l}} \frac {\partial J _ {F _ {L} , 1}}{\partial w _ {l , i}} \cdot u _ {w _ {l, i}} \\ \sum_ {i = 1} ^ {d _ {l}} \frac {\partial J _ {F _ {L} , 2}}{\partial w _ {l , i}} \cdot u _ {w _ {l, i}} \\ \vdots \\ \sum_ {i = 1} ^ {d _ {l}} \frac {\partial J _ {F _ {L} , d _ {h _ {L}}}}{\partial w _ {l , i}} \cdot u _ {w _ {l, i}} \end{array} \right] \cdot \left[ \begin{array}{c c c} u _ {w _ {l}, 1} & u _ {w _ {l}, 2} & \dots u _ {w _ {l}, d _ {l}} \end{array} \right]\right) \\ = \mathbb {E} _ {u _ {w _ {l}}} (D), \\ \end{array}
$$
$$
D _ {m, n} = \left(\sum_ {i = 1} ^ {d _ {l}} \frac {\partial J _ {F _ {L} , m}}{\partial w _ {l , i}} \cdot u _ {w _ {l, i}}\right) u _ {w _ {l, n}} = \frac {\partial J _ {F _ {L} , m}}{\partial w _ {l , n}} u _ {w _ {l, n}} ^ {2} + \sum_ {k \neq n} \frac {\partial J _ {F _ {L} , m}}{\partial w _ {l , k}} u _ {w _ {l, k}} u _ {w _ {l, n}}
$$
with  $m\in \{1,2,\ldots ,d_{h_L}\} ,n\in \{1,2,\ldots ,d_l\}$  . Since each  $u_{w_l}\sim \mathbb{N}(0,I)$  , we have
$$
\mathbb {E} _ {u _ {w _ {l}}} (D _ {m, n}) = \frac {\partial J _ {F _ {L , m}}}{\partial w _ {l , n}}, \quad \mathbb {E} _ {u _ {w _ {l}}} (D) = J _ {F _ {L}} (w _ {l}).
$$
Similarly, we can prove that  $\mathbb{E}_{u_{w_l}}\left[\sum_{k\neq l}\left(J_{F_L}(w_k)u_{w_k}\right)\cdot u_{w_l}\right] = 0\in \mathbb{R}^{d_{h_L}\times d_l}$ . So we have,
$$
\mathbb {E} \left(\frac {\partial f}{\partial F _ {L}} \cdot o _ {L} \cdot u _ {w _ {l}}\right) = \frac {\partial f}{\partial F _ {L}} J _ {F _ {L}} (w _ {l}) = \nabla_ {l, x} f (w) \in \mathbb {R} ^ {d _ {l}}
$$
C Proof of Lemma 5.3
Proof of Mean.
$$
\begin{array}{l} \hat {h} _ {1} ^ {t - K + 1} = F _ {1} (h _ {0} ^ {t - K + 1}, w _ {1} ^ {t - 2 K + 2}) = F _ {1} (x _ {i (t - K + 1)}, w _ {1} ^ {t - 2 K + 2}) \\ \hat {o} _ {1} ^ {t - K + 1} = J _ {F _ {1}} (h _ {0} ^ {t - K + 1}, w _ {1} ^ {t - 2 K + 2}) [ o _ {0} ^ {\intercal}, u _ {w _ {1}} ^ {t - K + 1 \intercal} ] ^ {\intercal} = J _ {F _ {1}} (h _ {0} ^ {t - K + 1}) o _ {0} + J _ {F _ {1}} (w _ {1} ^ {t - 2 K + 2}) u _ {w _ {1}} ^ {t - K + 1} \\ = J _ {F _ {1}} \left(w _ {1} ^ {t - 2 K + 2}\right) u _ {w _ {1}} ^ {t - K + 1} \\ \end{array}
$$
$$
\hat {h} _ {2} ^ {t - K + 1} = F _ {2} (\hat {h} _ {1} ^ {t - K + 1}, w _ {2} ^ {t - 2 K + 2})
$$
$$
\begin{array}{l} \hat {o} _ {2} ^ {t - K + 1} = J _ {F _ {2}} (h _ {1} ^ {t - K + 1}, w _ {2} ^ {t - K + 1}) [ o _ {1} ^ {t - K + 1 \top}, u _ {w _ {2}} ^ {t - K + 1 \top} ] ^ {\intercal} = J _ {F _ {2}} (h _ {1} ^ {t - K + 1}) o _ {1} ^ {t - K + 1} + J _ {F _ {2}} (w _ {2} ^ {t - 2 K + 2}) u _ {w _ {2}} ^ {t - K + 1} \\ = J _ {F _ {2}} (h _ {1} ^ {t - K + 1}) J _ {F _ {1}} (w _ {1} ^ {t - 2 K + 2}) u _ {w _ {1}} ^ {t - K + 1} + J _ {F _ {2}} (w _ {2}) u _ {w _ {2} ^ {t - 2 K + 2}} ^ {t - K + 1} \\ \end{array}
$$
$$
\hat {o} _ {L _ {0}} ^ {t - K + 1} = \sum_ {l = 1} ^ {L _ {0}} J _ {f _ {0}} \left(w _ {l} ^ {t - 2 K + 2}\right) u _ {w _ {l}} ^ {t - K + 1} = \sum_ {l \in \mathcal {G} (0)} J _ {f _ {0}} \left(w _ {l} ^ {t - 2 K + 2}\right) u _ {w _ {l}} ^ {t - K + 1}
$$
$$
\begin{array}{l} \hat {o} _ {L _ {t}} ^ {t - K + 1} = \sum_ {l \in \mathcal {G} (0)} J _ {f _ {t}} (w _ {l} ^ {t - 2 K + 2}) u _ {w _ {l}} ^ {t - K + 1} + \sum_ {l \in \mathcal {G} (1)} J _ {f _ {t}} (w _ {l} ^ {t - 2 K + 3}) u _ {w _ {l}} ^ {t - K + 1} + \dots + \sum_ {l \in \mathcal {G} (t)} J _ {f _ {t}} (w _ {l} ^ {t - 2 K + t + 2}) u _ {w _ {l}} ^ {t - K + 1} \\ = \sum_ {k = 0} ^ {t} \sum_ {l \in \mathcal {G} (k)} J _ {f _ {t}} \left(w _ {l} ^ {t - 2 K + k + 2}\right) u _ {w _ {l}} ^ {t - K + 1} \\ \end{array}
$$
$$
\begin{array}{l} \hat {o} _ {L _ {K - 1}} ^ {t - K + 1} = \sum_ {k = 0} ^ {K - 1} \sum_ {l \in \mathcal {G} (k)} J _ {f _ {K - 1}} \left(w _ {l} ^ {t - 2 K + k + 2}\right) = \sum_ {k = 0} ^ {K - 1} \sum_ {l \in \mathcal {G} (k)} J _ {f} \left(w _ {l} ^ {t - 2 K + k + 2}\right) u _ {w _ {l}} ^ {t - K + 1} \\ = \sum_ {k = 0} ^ {K - 1} \sum_ {l \in \mathcal {G} (k)} \nabla f _ {l, x _ {i (t - K + 1)}} \left(w ^ {t - 2 K + k + 2}\right) ^ {\intercal} u _ {w _ {l}} ^ {t - K + 1} \\ \end{array}
$$
Take expectation with respect to  $u_{w_l}^{t - K + 1}$ , where  $l\in \mathcal{G}(k)$ , we have
$$
\mathbb {E} _ {u _ {w _ {l}} ^ {t - K + 1}} \left(\hat {o} _ {L _ {K - 1}} ^ {t - K + 1} \cdot u _ {w _ {l}} ^ {t - K + 1}\right) = \nabla f _ {l, x _ {i (t - K + 1)}} \left(w ^ {t - 2 K + k + 2}\right)
$$
So we have,
$$
\mathbb {E} _ {u _ {w \mathcal {G} (k)} ^ {t - K + 1}} \big (\hat {o} _ {L _ {K - 1}} ^ {t - K + 1} \cdot u _ {w _ {l}} ^ {t - K + 1} \big) = \nabla f _ {\mathcal {G} (k), x _ {i (t - K + 1)}} \big (w ^ {t - 2 K + k + 2} \big)
$$
Lemma C.1 ([25], Theorem 3). Let  $g_{u}(x) = \langle \nabla f(x), u \rangle u$ , where  $u \in \mathbb{R}^d$  is a normally distributed Gaussian vector, then we have
$$
\mathbb {E} _ {u} \| g _ {u} (x) \| ^ {2} \leq (d + 4) \| \nabla f (x) \| ^ {2}
$$
Lemma C.2. Let  $g_{u_1, u_2}(x) = \langle \nabla f(x), u_1 \rangle u_2$ , where  $u_1 \in \mathbb{R}^{d_1}$ ,  $u_2 \in \mathbb{R}^{d_2}$  are two i.i.d. normally distributed Gaussian vectors, then we have
$$
\mathbb {E} _ {u _ {1}, u _ {2}} \left\| g _ {u _ {1}, u _ {2}} (x) \right\| ^ {2} \leq d _ {2} \left\| \nabla f (x) \right\| ^ {2}
$$
$$
\begin{array}{l} \mathbb {E} _ {u _ {1}, u _ {2}} \| g _ {u _ {1}, u _ {2}} (x) \| ^ {2} = \mathbb {E} _ {u _ {1}, u _ {2}} \| \left\langle \nabla f (x), u _ {1} \right\rangle u _ {2} \| ^ {2} = \mathbb {E} _ {u _ {1}, u _ {2}} \left(\left\langle \nabla f (x), u _ {1} \right\rangle^ {2} \| u _ {2} \| ^ {2}\right) \\ = \mathbb {E} _ {u _ {1}} \left(\left\langle \nabla f (x), u _ {1} \right\rangle^ {2}\right) \cdot \mathbb {E} _ {u _ {2}} \left(\left\| u _ {2} \right\| ^ {2}\right) \\ \leq d _ {2} \mathbb {E} _ {u _ {1}} \left(\langle \nabla f (x), u _ {1} \rangle^ {2}\right) = d _ {2} \mathbb {E} _ {u _ {1}} \left(\sum_ {i = 1} ^ {d _ {1}} \nabla_ {i} f (x) u _ {1, i}\right) ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} = d _ {2} \mathbb {E} _ {u _ {1}} \left(\sum_ {i = 1} ^ {d _ {1}} \nabla_ {i} ^ {2} f (x) u _ {1, i} ^ {2} + \sum_ {i \neq j} \nabla_ {i} f (x) \nabla_ {j} f (x) u _ {1, i} u _ {1, j}\right) \\ = d _ {2} \| \nabla f (x) \| ^ {2} \\ \end{array}
$$
where the first inequality is due to Lemma 1 in [25].
Proof of Variance.
where the inequality is due to Lemma C.1 and C.2.
D proof of Lemma 5.6
Proof. For the convenience of analysis, we denote  $t' = t - K + 1$ , then the update rule of algorithm 1 can be rewritten as
$$
w _ {\mathcal {G} (k)} ^ {t ^ {\prime} + 1} = w _ {\mathcal {G} (k)} ^ {t ^ {\prime}} - \gamma_ {t ^ {\prime}} \left(\hat {o} _ {L} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}}\right)
$$
Take expectation with respect to  $u_{w_{\mathcal{G}(k)}}^{t'}$ , we have