SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
w _ {\mathcal {G} (k)} ^ {t ^ {\prime} + 1} = w _ {\mathcal {G} (k)} ^ {t ^ {\prime}} - \gamma_ {t ^ {\prime}} \mathbb {E} _ {u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}}} \left(\hat {\sigma} _ {L} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}}\right) = w _ {\mathcal {G} (k)} ^ {t ^ {\prime}} - \nabla f _ {\mathcal {G} (k), x _ {i (t ^ {\prime})}} \left(w _ {\mathcal {G} (k)} ^ {t ^ {\prime} - K + k + 1}\right)
$$
Define diagonal matrices  $\mathbf{I}_0, \dots, \mathbf{I}_k, \dots, \mathbf{I}_{K-1} \in \mathbb{R}^{d \times d}$  such that all the principle diagonal elements of  $\mathbf{I}_k$  in  $\mathcal{G}(k)$  are 1, and all the principle diagonal elements of  $I_k$  in other than  $\mathcal{G}(k)$  are 0. Then we have
$$
\hat {\sigma} _ {L} ^ {t ^ {\prime}} \cdot u _ {w} ^ {t ^ {\prime}} = \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {\sigma} _ {L} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}}
$$
$$
\nabla f _ {x _ {i \left(t ^ {\prime}\right)}} \left(w ^ {t ^ {\prime} - K + k + 1}\right) = \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla f _ {\mathcal {G} (k), x _ {i \left(t ^ {\prime}\right)}} \left(w _ {\mathcal {G} (k)} ^ {t ^ {\prime} - K + k + 1}\right)
$$
Since  $f(\cdot)$  is  $L$ -smooth, the following inequality holds that:
$$
f \left(w ^ {t ^ {\prime} + 1}\right) \leq f \left(w ^ {t ^ {\prime}}\right) + \left\langle \nabla f \left(w ^ {t ^ {\prime}}\right), w ^ {t ^ {\prime} + 1} - w ^ {t ^ {\prime}} \right\rangle + \frac {L}{2} \| w ^ {t ^ {\prime} + 1} - w ^ {t ^ {\prime}} \| ^ {2}
$$
From the update rule of Algorithm 1, we take expectation with respect to all random variables on both sides and obtain:
$$
\begin{array}{l} \mathbb {E} \left[ f \left(w ^ {t ^ {\prime} + 1}\right) \right] \leq f \left(w ^ {t ^ {\prime}}\right) - \gamma_ {t ^ {\prime}} \mathbb {E} \left[ \nabla f \left(w ^ {t ^ {\prime}}\right) ^ {\intercal} \left(\sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {\sigma} _ {L} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}}\right) \right] + \frac {L \gamma_ {t ^ {\prime}} ^ {2}}{2} \mathbb {E} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {\sigma} _ {L _ {K - 1}} ^ {t ^ {\prime}} u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}} \right\| ^ {2} \\ = f \left(w ^ {t ^ {\prime}}\right) - \gamma_ {t ^ {\prime}} \sum_ {k = 0} ^ {K - 1} \nabla f \left(w ^ {t ^ {\prime}}\right) ^ {\intercal} \mathbf {I} _ {k} \left(\nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime}}\right) + \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime}}\right)\right) \\ + \frac {L \gamma_ {t ^ {\prime}} ^ {2}}{2} \mathbb {E} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {\sigma} _ {L _ {K - 1}} ^ {t ^ {\prime}} u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}} - \nabla f \left(w ^ {t ^ {\prime}}\right) + \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} \\ = f \left(w ^ {t ^ {\prime}}\right) - \gamma_ {t ^ {\prime}} \left\| \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} - \gamma_ {t ^ {\prime}} \sum_ {k = 0} ^ {K - 1} \nabla f \left(w ^ {t ^ {\prime}}\right) ^ {\intercal} \mathbf {I} _ {k} \left(\nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime}}\right)\right) \\ + \frac {L \gamma_ {t ^ {\prime}} ^ {2}}{2} \left\| \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} + \frac {L \gamma_ {t ^ {\prime}} ^ {2}}{2} \mathbb {E} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {\sigma} _ {L _ {K - 1}} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}} - \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} \\ + L \gamma_ {t ^ {\prime}} ^ {2} \sum_ {k = 0} ^ {K - 1} \nabla f \left(w ^ {t ^ {\prime}}\right) ^ {\intercal} \mathbf {I} _ {k} \left(\nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime}}\right)\right) \\ = f \left(w ^ {t ^ {\prime}}\right) - \left(\gamma_ {t ^ {\prime}} - \frac {L \gamma_ {t ^ {\prime}} ^ {2}}{2}\right) \left\| \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} + \underbrace {\frac {L \gamma_ {t ^ {\prime}} ^ {2}}{2} \mathbb {E} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {o} _ {L _ {K - 1}} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}} - \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2}} _ {Q _ {1}} \\ \underbrace {- \left(\gamma_ {t ^ {\prime}} - L \gamma_ {t ^ {\prime}} ^ {2}\right) \sum_ {k = 0} ^ {K - 1} \nabla f (w ^ {t ^ {\prime}}) ^ {\intercal} \mathbf {I} _ {k} \left(\nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime}}\right)\right)} _ {Q _ {2}}, \\ \end{array}
$$
Using the fact that  $\| x + y\| ^2\leq 2\| x\| ^2 +2\| y\| ^2$  and  $xy\leq \frac{1}{2}\| x\| ^2 +\frac{1}{2}\| y\| ^2$ , we have
$$
\begin{array}{l} Q _ {1} = \frac {L \gamma_ {t ^ {\prime}} ^ {2}}{2} \mathbb {E} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {\sigma} _ {L _ {K - 1}} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}} - \nabla f \left(w ^ {t ^ {\prime}}\right) - \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) \right. \\ + \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) \Bigg \| ^ {2} \\ \leq L \gamma_ {t ^ {\prime}} ^ {2} \underbrace {\mathbb {E} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {o} _ {L K - 1} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}} - \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) \right\| ^ {2}} _ {Q _ {3}} + \\ \end{array}
$$
$$
\begin{array}{l} + L \gamma_ {t ^ {\prime}} ^ {2} \underbrace {\left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f (w ^ {t ^ {\prime}}) \right\| ^ {2}} _ {Q _ {4}} \\ Q _ {2} = - \left(\gamma_ {t ^ {\prime}} - L \gamma_ {t ^ {\prime}} ^ {2}\right) \sum_ {k = 0} ^ {K - 1} \nabla f (w ^ {t ^ {\prime}}) ^ {\intercal} \mathbf {I} _ {k} \left(\nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime}}\right)\right) \\ \leq \frac {\gamma_ {t ^ {\prime}} - L \gamma_ {t ^ {\prime}} ^ {2}}{2} \left\| \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} + \frac {\gamma_ {t ^ {\prime}} - L \gamma_ {t ^ {\prime}} ^ {2}}{2} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla_ {\mathcal {G} (k)} f \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} \\ \end{array}
$$
Using  $\mathbb{E}\| \xi -\mathbb{E}[\xi ]\| ^2\leq \mathbb{E}\| \xi \| ^2$  , we have
$$
\begin{array}{l} Q _ {3} = \mathbb {E} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {\sigma} _ {L K - 1} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}} - \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) \right\| ^ {2} \\ \leq \mathbb {E} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {\sigma} _ {L _ {K - 1}} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}} \right\| ^ {2} \\ \leq (d + 4) \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla f _ {\mathcal {G} (k), x _ {i (t ^ {\prime})}} \left(w ^ {t ^ {\prime} - K + k + 1}\right) \right\| ^ {2} \\ = (d + 4) \sum_ {k = 0} ^ {K - 1} \left\| \nabla f _ {\mathcal {G} (k), x _ {i (t ^ {\prime})}} \left(w ^ {t ^ {\prime} - K + k + 1}\right) \right\| ^ {2} \\ \leq (d + 4) K M, \\ \end{array}
$$
where the second inequality is due to Lemma 5.3. Then we bound  $Q_{4}$ ,
$$
\begin{array}{l} Q _ {4} = \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} = \sum_ {k = 0} ^ {K - 1} \left\| \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f _ {\mathcal {G} (k)} \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} \\ \leq \sum_ {k = 0} ^ {K - 1} \left\| \nabla f \left(w ^ {t ^ {\prime} - K + k + 1}\right) - \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} \\ \leq L ^ {2} \sum_ {k = 0} ^ {K - 1} \left\| w ^ {t ^ {\prime}} - w ^ {t ^ {\prime} - K + k + 1} \right\| ^ {2} \\ = L ^ {2} \sum_ {k = 0} ^ {K - 1} \left\| \sum_ {j = \max  \{0, t ^ {\prime} - K + k + 1 \}} ^ {t ^ {\prime} - 1} \left(w ^ {j + 1} - w ^ {j}\right) \right\| ^ {2} = L ^ {2} \sum_ {k = 0} ^ {K - 1} \left\| \sum_ {j = \max  \{0, t ^ {\prime} - K + k + 1 \}} ^ {t ^ {\prime} - 1} \gamma_ {j} \left(\hat {o} _ {L _ {K - 1}} ^ {j} \cdot u _ {w} ^ {j}\right) \right\| ^ {2} \\ \leq L ^ {2} \gamma_ {\max  \{0, t ^ {\prime} - K + 1 \}} ^ {2} \sum_ {k = 0} ^ {K - 1} K \sum_ {j = \max  \{0, t ^ {\prime} - K + k + 1 \}} ^ {t ^ {\prime} - 1} (d + 4) \left\| \sum_ {k = 0} ^ {K - 1} \nabla f _ {\mathcal {G} (k), x _ {(j)}} (w ^ {t ^ {\prime} - K + k + 1}) \right\| ^ {2} \\ \leq (d + 4) K L \gamma_ {t ^ {\prime}} \frac {\gamma_ {\max  \{0 , t ^ {\prime} - K + 1 \}}}{\gamma_ {t ^ {\prime}}} \sum_ {k = 0} ^ {K - 1} \sum_ {j = \max  \{0, t ^ {\prime} - K + k + 1 \}} ^ {t ^ {\prime} - 1} \left\| \sum_ {k = 0} ^ {K - 1} \nabla f _ {\mathcal {G} (k), x _ {(j)}} \left(w ^ {t ^ {\prime} - K + k + 1}\right) \right\| ^ {2} \\ \leq (d + 4) L \gamma_ {t ^ {\prime}} \sigma K ^ {4} M, \\ \end{array}
$$
where the second inequality is from Assumption 5.1, the third inequality is due to Lemma 5.3, the fourth inequality follows from  $L\gamma_{t'} < 1$ , the last inequality follows from the inequality of arithmetic and geometric means, Assumption 5.2 and  $\sigma \coloneqq \max_{t'} \frac{\gamma_{\max\{0,t' - K + 1\}}}{\gamma_{t'}}$ . Integrating the upper bound together, we have
$$
\mathbb {E} \left[ f (w ^ {t ^ {\prime} + 1}) - f (w ^ {\prime}) \right] \leq - \frac {\gamma_ {t ^ {\prime}}}{2} \left\| \nabla f (w ^ {t ^ {\prime}}) \right\| ^ {2} + (d + 4) L \gamma_ {t ^ {\prime}} ^ {2} K M + \frac {\gamma_ {t ^ {\prime}} + L \gamma_ {t ^ {\prime}} ^ {2}}{2} (d + 4) L \gamma_ {t ^ {\prime}} \sigma K ^ {4} M
$$
$$
\begin{array}{l} \leq - \frac {\gamma_ {t ^ {\prime}}}{2} \left\| \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} + 2 (d + 4) L \gamma_ {t ^ {\prime}} ^ {2} \left(K M + \sigma K ^ {4} M\right) \\ = - \frac {\gamma_ {t ^ {\prime}}}{2} \left\| \nabla f \left(w ^ {t ^ {\prime}}\right) \right\| ^ {2} + 2 (d + 4) L \gamma_ {t ^ {\prime}} ^ {2} M _ {K}, \\ \end{array}
$$
where we let  $M_K = KM + \sigma K^4 M$ .
E Proof of Theorem 5.6
Proof. Let  $\gamma_t = \gamma$  be a constant, taking total expectation in Lemma 5.5, we have
$$
\mathbb {E} \left[ f \left(w ^ {t ^ {\prime} + 1}\right) \right] - \mathbb {E} \left[ f \left(w ^ {t ^ {\prime}}\right) \right] \leq - \frac {\gamma}{2} \mathbb {E} \| \nabla f \left(w ^ {t ^ {\prime}}\right) \| ^ {2} + 2 (d + 4) L \gamma^ {2} M _ {K},
$$
where  $\sigma = 1$  and  $M_{k} = KM + K^{4}M$ . summing the above inequality from  $t' = 0$  to  $T - 1$  we have
$$
\mathbb {E} [ f (w ^ {T}) ] - f (w ^ {0}) \leq - \frac {\gamma}{2} \sum_ {t ^ {\prime} = 0} ^ {T - 1} \mathbb {E} \| \nabla f (w ^ {t ^ {\prime}}) \| ^ {2} + 2 T (d + 4) \gamma^ {2} L M _ {K}
$$
Then we have
$$
\frac {1}{T} \sum_ {t ^ {\prime} = 0} ^ {T - 1} \mathbb {E} \| \nabla f (w ^ {t ^ {\prime}}) \| ^ {2} \leq \frac {2 (f (w ^ {0}) - f (w ^ {*}))}{\gamma T} + 4 (d + 4) L \gamma M _ {K}.
$$
F Proof of Theorem 5.7
Proof. Let  $\{\gamma_{t'}\}$  be a diminishing sequence and  $\gamma_{t'} = \frac{\gamma_0}{t' + 1}$ , such that  $\sigma < K$  and  $M_K = KM + K^5 M$ . Taking expectation in Lemma 5.5 and summing it from  $t' = 0$  to  $T - 1$ , we have
$$
\mathbb {E} \left[ f \left(w ^ {T}\right) \right] - f \left(w ^ {0}\right) \leq - \frac {1}{2} \sum_ {t ^ {\prime} = 0} ^ {T - 1} \gamma_ {t} \mathbb {E} \| \nabla f \left(w ^ {t ^ {\prime}}\right) \| ^ {2} + \sum_ {t ^ {\prime} = 0} ^ {T - 1} 2 (d + 4) \gamma_ {t ^ {\prime}} ^ {2} L M _ {K}.
$$
Letting  $\Gamma_T = \sum_{t' = 0}^{T - 1}\gamma_{t'}$ , then we have
$$
\frac {1}{\Gamma_ {T}} \sum_ {t ^ {\prime} = 0} ^ {T - 1} \gamma_ {t ^ {\prime}} \mathbb {E} \| \nabla f (w ^ {t ^ {\prime}}) \| ^ {2} \leq \frac {2 (f (w ^ {0}) - f (w ^ {*}))}{\Gamma_ {T}} + \frac {\sum_ {t ^ {\prime} = 0} ^ {T - 1} 4 (d + 4) \gamma_ {t ^ {\prime}} ^ {2} L M _ {K}}{\Gamma_ {T}}
$$
G Details of AsyncFGD
G.1 Working with Adam
We provide example for AsyncFGD working with Adam in Algorithm 2. Minimal changes are made on Adam by substituting the gradient the estimator using Forward Gradient.
G.2 Execution Details
Details are presented in Figure 5. By pipelining over time dimension, we can preserve buffer for input in only one timestamp and still achieve parallel computation.
Algorithm 2 AsyncFGD-Adam
Initialize: Stepsize sequence  $\{\gamma_t\}_{t = K - 1}^{T - 1}$  weight  $w^0 = \left[w_{\mathcal{G}(0)}^0,\dots ,w_{\mathcal{G}(K - 1)}^0\right]\in \mathbb{R}^d,m_{\mathcal{G}(k)} =$ $0,v_{\mathcal{G}(k)} = 0,\beta_1 = 0.9,\beta_2 = 0.999,\eta = 1e - 8$    
1: for  $t = 0,1,\dots ,T - 1$  do   
2: for  $k = 0,1,\dots ,K - 1$  in parallel do   
3: Read  $\hat{h}_{L_k - 1}^{t - k},\hat{o}_{L_k - 1}^{t - k}$  from storage if  $k\neq 0$    
4: Compute  $\hat{h}_{L_k}^{t - k},\hat{o}_{L_k}^{t - k}$    
5: Send  $\hat{h}_{L_k}^{t - k},\hat{o}_{L_k}^{t - k}$  to next worker's storage if  $k\neq K - 1$    
6: end for   
7: Broadcast  $\hat{o}_{LK - 1}^{t - K + 1}$    
8: for  $k = 0,1,\dots ,K - 1$  in parallel do   
9: Compute  $\Delta w_{\mathcal{G}(k)}^{t - K + 1} = \hat{o}_{LK - 1}^{t - K + 1}u_{w_{\mathcal{G}(k)}}^{t - K + 1}$    
10: Update  $m_{\mathcal{G}(k)} = \beta_1\Delta w_{\mathcal{G}(k)}^{t - K + 1} + (1 - \beta_1)m_{\mathcal{G}(k)}$    
11: Update  $v_{\mathcal{G}(k)} = \beta_2\Delta w_{\mathcal{G}(k)}^{t - K + 1} + (1 - \beta_2)v_{\mathcal{G}(k)}$    
12: Compute  $\hat{m}_{\mathcal{G}(k)} = m_{\mathcal{G}(k)} / \beta_1^t$    
13: Compute  $\hat{v}_{\mathcal{G}(k)} = v_{\mathcal{G}(k)} / \beta_2^t$    
14: Update  $w_{\mathcal{G}(k)}^{t - K + 2} = w_{\mathcal{G}(k)}^{t - K + 1} - \gamma_{t - K + 1}\hat{m}_{\mathcal{G}(k)} / \hat{v}_{\mathcal{G}(k)}$    
15: end for   
16: end for
Figure 5: Details in execution of AsyncFGD-RNN with 3 modules. In the skip stage, only the host accumulate loss and its jvp value and other workers will jump right into the next state.
G.3 Extension: AsyncFGD-Recursive
In this section, we extend the potential of AsyncFGD by exploring the parallelization of sequential inputs in RNNs with reduced memory footprint, necessitating the preservation of input for only a single timestamp.
We adopt a one-to-many RNN network for ease of illustration and denote the equal length of each sequence as  $n$ . We begin by refactoring the original loss for RNNs in terms of cumulative loss and new activation. Here,  $s_l^t$  signifies the hidden state at timestamp  $t$  on layer  $l$ . At timestamp  $t$ , each layer ingests  $(s_{l-1}^t, s_l^{t-1})$  as input, generating  $s_l^t = F_l(s_{l-1}^t, s_l^{t-1}, w_l)$ . We represent the stacked latent states passed from  $t-1$  as  $s^{t-1} = [s_1^{t-1}, s_2^{t-1}, \ldots, s_L^{t-1}]$  and the output as  $y_t = F(s_0^t, s^{t-1}; w)$ , where  $s_0^t$  symbolizes the input data  $x_t$ . The cumulative loss from timestamp  $1 \sim T$  is given by:
$$
\sum_ {t = 1} ^ {T} f \left(F \left(x _ {t}, s ^ {t - 1}; w\right), y _ {t}\right) \tag {15}
$$
We next refactor equation 2 for the  $i_{th}$  sequential input in iteration  $i, i \geq 0$  as:
$$
w _ {l} ^ {i + 1} = w _ {l} ^ {i} - \gamma_ {i} \frac {\partial \mathcal {L} _ {i}}{\partial w _ {l} ^ {i}}, \quad \forall l \in 1, 2, \dots , L \tag {16}
$$
where  $\mathcal{L}i\coloneqq \sum t = in + 1^{(i + 1)n}f(F(x_t,s^{t - 1};w),y_t)$  represents the loss for the  $i_{th}$  sequence. We break the dependency between timestamps and iterations by employing dynamic staleness in AsynFGD. Specifically, the computation in module  $k\in 1,2,\dots ,K$  at timestamp  $t$  is defined as follows:
$$
\hat {s} L _ {k} ^ {t - k} = f _ {k} \left(\hat {s} L _ {k - 1} ^ {t - k}, \hat {s} \mathcal {G} (k) ^ {t - k - 1}; w \mathcal {G} (k) ^ {t - 2 K + k + 2}\right) \tag {17}
$$
$$
\partial L _ {k} ^ {t - k} = J f _ {k} \left(\hat {s} L k - 1 ^ {t - k}, \hat {s} \mathcal {G} (k) ^ {t - k - 1}; w \mathcal {G} (k) ^ {t - 2 K + k + 2}\right) u _ {\mathcal {G} (k)} ^ {t - k}, \tag {18}
$$
where  $u_{\mathcal{G}(k)}^{t - k} = [\hat{o} Lk - 1^{t - k\mathsf{T}},\hat{o}\mathcal{G}(k)^{t - k - 1\mathsf{T}},uw_{\mathcal{G}(k)}^{t - k\mathsf{T}}]\mathsf{T}$
Given that tasks belonging to the same iteration use identical parameters, we use  $\delta(k,t,i) = t - ni - k - 1$ ,  $t \in [in + 1, (i + 1)n]$  to quantify this difference for the  $i_{th}$  sequential. If  $\delta(k,t,i) \leq 0$ , then module  $k$  uses stale parameters from iteration  $i - 1$  at timestamp  $t$ . AsyncFGD-RNN only updates the parameter upon the completion of the last computation in the sequence. Specifically, we use:
$$
w ^ {t - K + 2} = \left\{ \begin{array}{l l} w ^ {t - K + 1}, & \text {i f} \frac {t - K + 1}{n} \notin \mathbb {N} ^ {*} \\ w ^ {t - K + 1} - \gamma_ {\lfloor \frac {t - K}{n} \rfloor} \mathbb {E} u w ^ {t} \left(\left(\frac {\partial \mathcal {L} \left\lfloor \frac {t - K}{n} \right\rfloor}{\partial s L _ {K} ^ {t - K}} o _ {L _ {K}} ^ {t - K}\right) u _ {w} ^ {(t - K)}\right), & \text {o t h e r w i s e} \end{array} \right.
$$
Refer to figure 5 for detailed execution instructions. By combining training and prediction, we can process data from different timestamps of sequential input, maintain a buffer for just a single timestamp, and still achieve parallelization among various workers.
H Training Details
In this section, we explain some details in the training process.
H.1 Random Seed
The seeds for all experiments are fixed to 0.
H.2 Description of Network Architecture
H.2.1 Models Deployed in Section 6.2.
The network structures of ConvS, ConvL, FCS and FCL are enumerated in Tables 8, 11, 9, and 10 correspondingly. In these designations, the subscripts  $ReLU$  and  $Tanh$  signify the particular activation function used in the model. Moreover, the larger models, denoted as counterparts, incorporate batch normalization layers for enhanced performance.
H.2.2 Models used in Section 6.3.
We delve into the specific models employed in Section 6.3. For MobileNet, we utilize the structure of MobileNet_V3_Small. The ResNet-18 structure is used when implementing ResNet. The model EfficientNet_B0 signifies the EfficientNet architecture. The MNASNet0_5 is used for MNasNet. Lastly, we adopt ShuffleNet_V2_X0_5 for ShuffleNet.
H.3 Model Splitting
In this section, we provide details for how to split the model into consecutive moduels and distribute them in different worker, we will first provide how to split models in Section 6.2, then we provide how to split model in 6.3.
H.3.1 Model Splitting in Section 6.2
In Section 6.2, all models are split with  $K = 3$ . Details for how to split the ConvS, ConvL, FCS, FCL are repented in Table 5 Table 6, Table3 and Table 4, respectively.
Table 3: Details for model splitting for ConvS, definition for layers can be found in Table 8
K	Layers
1	conv1, act
2	pool1, fc1
3	act2, fc2
Table 4: Details for model splitting for ConvL, definition for layers can be found in Table 11
K	Layers
1	conv1, bn1, act1, pool1, conv2, bn2, act2, pool2
2	conv3, bn3, act3, pool3,conv4, bn4, act4, pool4
3	conv5, bn5, act5, pool5, fc1
Table 5: Details for model splitting for FCS, definition for layers can be found in Table 9
K	Layers
1	fc1,ac1
2	fc2,ac2
3	fc3,ac3
Table 6: Details for model splitting for FSL, definition for layers can be found in Table10
K	Layers
1	fc1, bn1, ac1, fc2, bn2, ac2
2	fc3, bn3, ac3 ,fc4, bn4, ac4
3	fc5,bn5,ac5,fc5
H.3.2 Model Splitting in Section 6.3
In section 6.3, all models are divided into four parts ( $K = 4$ ). Detailed descriptions of how each model is split are provided below. Note that 'head' and 'tail' refer to the layers before and after the main blocks of each architecture, respectively, which are assigned to the first and the last worker:
- ResNet-18: The core of ResNet-18 consists of 4 Residual Blocks, each distributed to one of the four workers.
- EfficientNet: The core of EfficientNet consists of 7 Mobile Inverted Bottlenecks (MBConv). The first worker handles MBConv 1, the second handles MBConv 2 to 3, the third manages MBConv 4 to 6, and the last one manages MBConv 7.
- MoblieNet: The core of MoblieNetV3-small includes 13 layers of bottlenecks. The first worker handles layers 1 to 3, the second manages layers 4 to 7, the third manages layers 8 to 11, and the last worker handles layers 12 to 13.
- MnasNet: The core of MnasNet consists of 6 blocks of inverted residuals. Blocks 1 to 2, 3 to 5, and 6 are assigned to workers 2, 3, and 4 respectively, while the first worker only handles the head.
- ShuffleNet: The core of ShuffleNet consists of 3 stages, each assigned to workers 2, 3, and 4, respectively.
I Additional Experimental Results
I.1 Ablation study in  $\alpha$
We have incremented the value of  $\alpha_{bias}$  gradually, with a step size of 0.0075, over 20 epochs. This process can be generalized using the following equations:
$$
\alpha_ {b i a s} = \left\{ \begin{array}{l l} t \times \frac {\alpha_ {b i a s} ^ {*}}{2 0}, & t <   = 2 0 \\ \alpha_ {b i a s} ^ {*}, & \text {o t h e r w i s e} \end{array} \right.
$$
Here,  $\alpha_{bias}^{*}$  control the rate of increase and the maximum attainable value of  $\alpha_{bias}$ , respectively. The ablation study with respect to  $\alpha_{bias}^{*}$  is presented in Table 7.
We observe that reducing  $\alpha_{bias}^{*}$  to 0, which corresponds to only updating the classifier, still results in performance gains compared to updating the full model. This improvement can be attributed to reduced variance. As  $\alpha_{bias}^{*}$  increases, we generally see better results, since the norm of the gradient approximation increases. However, when  $\alpha_{bias}^{*}$  exceeds 0.25, we sometimes observe a performance drop due to the corresponding increase in variance.
Table 7: Ablation study on different value of  ${\alpha }_{bias}^{ * }$
Dataset	Model	α*bias
0.00	0.03	0.06	0.09	0.12	0.15	0.20	0.25
CIFAR10	Res-18	0.838	0.838	0.845	0.855	0.865	0.878	0.872	0.885
Mobile	0.898	0.912	0.911	0.910	0.913	0.911	0.914	0.909
Efficient	0.892	0.900	0.902	0.903	0.902	0.902	0.887	0.895
Shuffle	0.788	0.805	0.808	0.812	0.820	0.820	0.822	0.825
Mnas	0.782	0.790	0.788	0.789	0.788	0.789	0.777	0.782
FMNIST	Res-18	0.866	0.869	0.871	0.873	0.875	0.880	0.882	0.884
Mobile	0.890	0.908	0.906	0.906	0.906	0.906	0.899	0.901
Efficient	0.889	0.904	0.906	0.902	0.905	0.904	0.908	0.897
Shuffle	0.849	0.854	0.857	0.860	0.864	0.870	0.870	0.877
Mnas	0.854	0.868	0.870	0.870	0.870	0.870	0.864	0.864
I.2 Acceleration across Various Platforms and Architectures
In Section 6.5, we examined the acceleration of AsyncFGD in comparison to vanilla FGD on ResNet-18, using two hardware platforms: 1) NVIDIA AGX Orin, an embedded device, and 2) a cluster of four NVIDIA 1080 Ti GPUs. These platforms were chosen to reflect real-world edge device scenarios and to simulate situations of ample computational power, such as in the case of stacked chips.
In this section, we expand our scope of investigation by incorporating two additional devices: 1) NVIDIA A100, and 2) Intel(R) Xeon(R) CPU E5-2678 v3 @2.50GHz. These additions allow us to further examine acceleration under various conditions. We also provide supplementary results on acceleration with respect to different batch sizes to reflect variable input streams. Moreover, to emulate streamlined input, the mini-batch size of the synchronized pipeline is set to 1.