SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaohan Zhao<sup>1</sup>
xiaotuzxh@gmail.com
Hualin Zhang
zhanghualin98@gmail.com
Zhouyuan Huo*
huozhouyuan@gmail.com
Bin Gu2,3†
jsbugin@gmail.com
<sup>1</sup> Nanjing University of Information Science and Technology, China
$^{2}$  Mohamed bin Zayed University of Artificial Intelligence, UAE
<sup>3</sup> School of Artificial Intelligence, Jilin University, China
On-device learning faces memory constraints when optimizing or fine-tuning on edge devices with limited resources. Current techniques for training deep models on edge devices rely heavily on backpropagation. However, its high memory usage calls for a reassessment of its dominance. In this paper, we propose forward gradient descent (FGD) as a potential solution to overcome the memory capacity limitation in on-device learning. However, FGD's dependencies across layers hinder parallel computation and can lead to inefficient resource utilization. To mitigate this limitation, we propose AsyncFGD, an asynchronous framework that decouples dependencies, utilizes module-wise stale parameters, and maximizes parallel computation. We demonstrate its convergence to critical points through rigorous theoretical analysis. Empirical evaluations conducted on NVIDIA's AGX Orin, a popular embedded device, show that AsyncFGD reduces memory consumption and enhances hardware efficiency, offering a novel approach to on-device learning.
Deep learning models have increasingly gained attraction in a multitude of applications, showcasing exceptional predictive capabilities. Nevertheless, their rapidly expanding size [19] poses a formidable challenge for resource-limited edge devices, such as mobile phones and embedded systems. These devices are pervasive in our society and continuously generate new data. To attain model customization, user privacy, and low latency, these devices necessitate on-device learning, involving training and fine-tuning models on freshly gathered data [46]. However, the restricted memory capacity of these devices emerges as a significant hindrance. For example, the Raspberry Pi Model A, introduced in 2013, only featured  $256\mathrm{MB}$  of memory [17], while the more recent Raspberry Pi 400, released in 2020, modestly increased this capacity to a mere 4 GB.
Various techniques have been proposed to address this issue, encompassing quantized training, efficient transfer learning, and rematerialization. Quantized training curtails memory consumption by utilizing low-precision network representation [22, 15, 42, 43, 8]. Efficient transfer learning diminishes training costs by updating merely a subset of the model [20, 5, 2, 44]. Lastly, rematerialization
*Author Zhouyuan Huo is currently at Google. No work performed at Google.
† Corresponding author.
Figure 1: Comparison of Vanilla FGD and AsyncFGD, where A, B, and C signify workers in the system. Through the allocation of tasks from varying iterations, AsyncFGD breaks forward locking in FGD, thereby maximizing worker utilization.
conserves memory at the expense of time by discarding and recomputing intermediate variables [3, 7, 16, 35].
Although most of these techniques can be applied irrespective of the optimization strategy, they are frequently employed in tandem with backpropagation in deep learning. Due to the reciprocal nature of backpropagation (i.e., activations from the forward pass are preserved for the subsequent backward pass), the training of deep models utilizing backpropagation typically commands a memory footprint that is  $3 - 4 \times$  larger than the number of parameters involved. Consequently, reexamining backpropagation within the context of surpassing the memory capacity barrier is crucial due to its elevated memory consumption.
The recent revival of interest in the forward-forward algorithm proposed by [10], along with other algorithms predicated on forward computation [39, 28, 1, 33], has prompted a reevaluation of backpropagation. The question persists: is it essential to store intermediate variables and pause to propagate gradients? An alternative, forward gradient descent (FGD), leverages the Jacobian-vector product (JVP) under automatic differentiation [37, 1, 33] to assess the effect of a stochastic perturbation in conjunction with the forward pass, yielding an unbiased approximation of the true gradient. FGD offers benefits in terms of memory consumption and biological plausibility[31], as it solely employs forward passes and substantially reduces the storage of intermediate variables [31]. Moreover, FGD can be combined with other existing techniques, such as quantized training and efficient transfer learning, potentially further diminishing memory costs.
Nonetheless, the intrinsic locking mechanism within FGD, more specifically, the layer-by-layer sequential computation during the forward pass, poses an impediment to parallel computation across disparate layers as shown in Fig. 1. In response to these challenges, this paper aims at training deep models on edge devices with memory constraints from an optimization perspective, while improving resource utilization by breaking the lock inside the forward pass with provable convergence. Thus, we propose AsyncFGD, an asynchronous version of Forward Gradient Descent with module-wise asynchronism in the forward pass to disentangle its dependencies, allowing simultaneous computation on different workers. The induced module-wise decreasing and bounded staleness in the parameters not only accelerates prediction and training but also provides theoretical guarantees in convergence. We empirically validate our method across multiple architectures and devices, including CPUs, GPUs, and embedded devices. Our results indicate that AsyncFGD reduces memory consumption and improves hardware efficiency, achieving efficient training on edge devices.
2.1 Forward Gradient with Reinforcement Learning
This research builds upon the concept of forward-mode automatic differentiation (AD) initially introduced by [37]. It has since been applied to learning recurrent neural networks [39] and calculating
Hessian vector products [28]. However, exact gradient computation using forward-mode AD requires the complete and computationally expensive Jacobian matrix. Recently, Baydin et al. [1] and Silver et al. [31] proposed an innovative technique for weight updating based on directional gradients along randomly perturbed directions. These algorithms have connections to both reinforcement learning (RL) and evolution strategies (ES), since the network receives global rewards in each instance. RL and ES have been successfully utilized in specific continuous control and decision-making tasks in neural networks [38, 34, 32, 4]. Clark et al. [4] observed that global credit assignment performs well in vector neural networks with weights between vectorized neuron groups. However, in forward-model AD methods, much like in evolution strategies, the random sampling approach can lead to high variance in the estimated gradient, particularly when optimizing over a large-dimensional parameter space [31].
2.2 Parallel Strategies and Asynchronous Approaches
Various parallel frameworks have been developed to accelerate computations by utilizing multiple workers. These frameworks include data parallelism [21], pipeline parallelism [12, 30, 23, 14], and tensor parallelism [29, 24]. Each approach leverages different dimensions of data - batch dimension for data parallelism, layer dimension for pipeline parallelism, and feature dimension for tensor parallelism - to improve computational efficiency.
However, in edge computing environments, data is often collected and processed frame-by-frame rather than in batches. This is particularly true for scenarios requiring real-time adaptation and streaming data learning. In such contexts, edge models must quickly adapt to newly acquired data in real-time, leaving limited scope for collecting, fetching, and batching historical data. Therefore, algorithms that do not depend on batch size are preferable. The ability to align with this pipeline and train the model on the fly with incoming data is crucial. Therefore, our research specifically focuses on pipeline parallelism.
Pipeline parallelism can be categorized into synchronous and asynchronous pipelines, which divide computations into sequential stages and align well with the data stream on edge devices. However, in edge devices with limited computational power, potential idle workers due to synchronization in synchronous pipelines like [12] are not optimal.
Asynchronous parallelism, which allows non-simultaneous task execution to enhance resource utilization, is particularly relevant to our method. [23, 14] parallelize tasks by executing them from different iterations. However, the additional memory overhead of storing replicate copies to handle staleness in [23] and the use of multiple activations for backpropagation make the algorithm memory inefficient. This memory overhead is mitigated in strategies such as "rematerialization" [40, 13] and weight estimation [41]. However, these strategies were originally designed for backpropagation, while our focus is on FGD, which has distinct computational characteristics, rendering the existing "1B1F" strategy for work scheduling in [40, 13, 41, 23] unsuitable.
Gradient-based Method. We commence with a succinct introduction to gradient-based methods deployed in neural networks. Envision the training of a  $L$ -layer feedforward neural network, where each layer  $l \in 1,2,\ldots,L$  accepts  $h_{l-1}$  as an input, generating an activation  $h_l = F_l(h_{l-1};w_l)$  with weight  $w_l \in \mathbb{R}^{d_l}$ . We represent all parameters by  $w = [w_1^{\intercal}, w_2^{\intercal}, \ldots, w_L^{\intercal}]^{\intercal} \in \mathbb{R}^d$  and the output of the network by  $h_L = F(h_0, w)$ , where  $h_0$  symbolizes the input data  $x$ . Given a loss function  $f$  and targets  $y$ , the objective problem is as follows:
$$
\min  _ {w} f (F (x; w), y) \tag {1}
$$
Here, we use  $f(x;w)$  in subsequent sections for the sake of brevity.
Gradient-based methods are widely used to optimize deep learning problems. At iteration  $t$ , we feed data sample  $x_{i(t)}$  into the network, where  $i(t)$  signifies the sample's index. As per the principles of stochastic gradient descent (SGD), we define the network parameters as follows:
$$
w _ {l} ^ {t + 1} = w _ {l} ^ {t} - \gamma_ {t} \nabla f _ {l, x _ {i (t)}} \left(w ^ {t}\right), \quad \forall l \in \{1, 2, \dots , L \} \tag {2}
$$
Here,  $\gamma_t \in \mathbb{R}$  is the stepsize and  $\nabla f_{l,x_{i(t)}}(w^t) \in \mathbb{R}^{d_l}$  is the gradient of the loss function with respect to the weights at layer  $l$  and data sample  $x_{i(t)}$ .
The crux of the matter is obtaining  $\nabla f_{l,x_{i(t)}}(w^t)$ . Both scientific and industrial sectors lean towards backpropagation underpinned by automatic differentiation (AD). However, FGD could be another alternative, because it can approximate the true gradient without bias by using forward-mode AD, and importantly has low memory consumption as it only preserves the intermediate variable passed from previous layer, while backpropagation need to store intermediate variable in each layer.
Forward Gradient Descent. Let  $J_{F_l}$  represent the Jacobian matrix of layer  $l$  while  $u_{w_l}$  represent the random perturbation on  $w_l$  (more precisely, tangent vector around  $w_l$ ), we can calculate the JVP value  $o_l$ , in each layer sequentially by
$$
h _ {l} = F _ {l} \left(h _ {l - 1}; w _ {l}\right) \in \mathbb {R} ^ {d _ {h _ {l}}}, \tag {3}
$$
$$
o _ {l} = J _ {F _ {l}} \left(h _ {l - 1}, w _ {l}\right) u _ {l} \in \mathbb {R} ^ {d _ {h _ {l}}}, \quad \text {w h e r e} \quad u _ {l} = \left[ o _ {l - 1} ^ {\intercal}, u _ {w _ {l}} ^ {\intercal} \right] ^ {\intercal} \in \mathbb {R} ^ {d _ {h _ {l - 1}} + d _ {l}} \tag {4}
$$
Mathematically,  $o_l$  is the directional derivative of  $F_l$  along  $u_l$ ; intuitively,  $o_l$  can be interpreted as the influence of the perturbation  $u_{w_l}$  on function value  $h_l$  (we set  $o_0$  to be 0 since we don't need to calculate the JVP value with respect to the input data). This process can be arranged within the forward pass such that (3), (4) are computed in the same pass. Also, since we don't need to propagate backward,  $h_{l-1}$  and  $o_{l-1}$  are thrown away right after the computation of  $h_l$  and  $o_l$ . Then we can approximate the true gradient  $\nabla f(x; w)$  unbiasedly (Lemma 3.1) by  $\left( \frac{\partial f}{\partial h_L} o_L \right) u$ , where  $u = [u_{w_1}, \ldots, u_{w_L}]^T$ .
Lemma 3.1. Let  $u_{w_l} \in \mathbb{R}^{d_l}$ ,  $l \in \{1,2,\dots,L\}$  be a normally distributed random Gaussian vectors, then the forward gradient computed through Eq.(3), (4) is an unbiased estimate of the true gradient
$$
\nabla f (x; w) = \mathbb {E} _ {u} \left[ \left(\frac {\partial f}{\partial h _ {L}} o _ {L}\right) u \right].
$$
More specifically, each layer  $l$  receives  $o_{L}$  from the last layer and updates its own parameters locally by using  $\mathbb{E}_{u_{w_l}}(u_{w_l} \cdot o_L)$ . We can then rewrite (2) as:
$$
w _ {l} ^ {t + 1} = w _ {l} ^ {t} - \gamma_ {t} \left(\left(\frac {\partial f}{\partial h _ {L}} o _ {L} ^ {t}\right) u _ {w _ {l}} ^ {t}\right) \tag {5}
$$
Forward Locking. It is evident from Eq.(4) that the computation in layer  $l$  depends on the activation and JVP value  $h_{l-1}, o_{l-1}$  from layer  $l-1$ . This dependency creates a "lock" that prevents all layers from updating before receiving the output from dependent layers, thus serializing the computation in the forward pass (refer to Figure 1 for illustration).
4 AsyncFGD
In this section, we propose an innovative approach that utilizes module-wise staleness to untether the dependencies inherent in FGD. This method, which we've named AsyncFGD, facilitates the simultaneous execution of tasks originating from disparate iterations. Suppose a  $L$ -layer feedforward neural network is divided into  $K$  modules, with each module comprising a set of consecutive layers and their respective parameters. Consequently, we have a configuration such that  $w = [w_{\mathcal{G}(0)}^{\top}, w_{\mathcal{G}(1)}^{\top}, \dots, w_{\mathcal{G}(K-1)}^{\top}]^{\top}$ , with  $\mathcal{G}(k)$  denoting the layer indices in group  $k$ .
Now, let's delve into the details of how AsyncFGD untethers iteration dependencies and expedites the training process.
4.1 Detaching Iteration Dependency
Forward. At the timestamp  $t$ , the data sample  $x_{i}(t)$  is pumped to the network. In contrast to sequential FGD [1], AsyncFGD permits modules to concurrently compute tasks, each originally belonging to a distinct iteration. All modules, with the exception of the last, operate using delayed parameters. We designate the activation and its JVP value originally ascribed to iteration  $t$  in Eq.(3), (4) as  $\hat{o}_l^t, \hat{h}_l^t$  respectively. Though the superscript  $t$  is no longer time-dependent, it maintains its role in indicating sequential order. Consider  $L_{k} \in \mathcal{G}(k)$  as the final layer in module  $k$  and  $f_{k}$  as the activation of this last layer. The computation in module  $k \in 0,1,\dots ,K - 1$  at timestamp  $t$  is defined recursively as follows:
$$
\hat {h} _ {L _ {k}} ^ {t - k} = f _ {k} \left(\hat {h} _ {L _ {k - 1}} ^ {t - k}; w _ {\mathcal {G} (k)} ^ {t - 2 K + k + 2}\right) \tag {6}
$$
$$
\hat {o} _ {L _ {k}} ^ {t - k} = J _ {f _ {k}} \left(\hat {h} _ {L _ {k - 1}} ^ {t - k}; w _ {\mathcal {G} (k)} ^ {t - 2 K + k + 2}\right) u _ {\mathcal {G} (k)} ^ {t - k}, \text {w h e r e} u _ {\mathcal {G} (k)} ^ {t - k} = \left[ \hat {o} _ {L _ {k - 1}} ^ {t - k \mathsf {T}}, u _ {w _ {\mathcal {G} (k)}} ^ {t - k \mathsf {T}} \right] ^ {\mathsf {T}}. \tag {7}
$$
Concurrently, each module receives and stores output from its dependent module for future computations.
Update. The update phase in AsyncFGD parallels that in FGD [1]. The final module broadcasts its JVP value, triggering each module to perform local updates to their parameters. To summarize, at timestamp  $t$ , we execute the following update:
$$
w ^ {t - K + 2} = w ^ {t - K + 1} - \gamma_ {t - K + 1} \left(\hat {\partial} _ {L _ {K - 1}} ^ {t - K + 1} u _ {w} ^ {t - K + 1}\right) \tag {8}
$$
Staleness. We measure the time delay in Eq.(6), (7) with  $g(k) = K - 1 - k$ , and  $g(K - 1) = 0$  indicates that last module employs up-to-date parameters.
This approach effectively disrupts the "lock-in" characteristic of FGD, facilitating a parallel forward pass. A comparative illustration of the execution details in sequential FGD and AsyncFGD is provided in Figure 1.
4.2 Stochastic AsyncFGD Algorithm
To better illustrate the working state of AsyncFGD, we make the following definitions:
$$
w ^ {t - K + 1} := \left\{ \begin{array}{l l} w ^ {0}, & t - K + 1 <   0 \\ w ^ {t - K + 1}, & \text {o t h e r w i s e} \end{array} ; \right. \text {a n d} \hat {h} _ {L _ {k}} ^ {t - k}, \hat {o} _ {L _ {k}} ^ {t - k} := \left\{ \begin{array}{l l} 0, 0, & t - k <   0 \\ \hat {h} _ {L _ {k}} ^ {t - k}, \hat {o} _ {L _ {k}} ^ {t - k}, & \text {o t h e r w i s e} \end{array} \right. \tag {9}
$$
Unlike FGD, AsyncFGD forwards the JVP value and activations by parameters in different time delays, which can be concluded as  $f\left(x_{i(t - 2K + 2)};w_{\mathcal{G}(0)}^{t - 2K + 2};w_{\mathcal{G}(1)}^{t - 2K + 3};\dots ;w_{\mathcal{G}(K - 1)}^{t - K + 1}\right)$ . A detailed illustration of the AsyncFGD with  $K = 3$  is shown in Appendix G.2. We summarize the proposed algorithm in Algorithm 1 by example of sampling one tangent vector per iteration.
Algorithm 1 AsyncFGD-SGD
Initialize: Stepsize sequence  $\{\gamma_t\}_{t = K - 1}^{T - 1}$  weight  $w^0 = \left[w_{\mathcal{G}(0)}^0,\dots ,w_{\mathcal{G}(K - 1)}^0\right]\in \mathbb{R}^d$    
1: for  $t = 0,1,\dots ,T - 1$  do   
2: for  $k = 0,1,\dots ,K - 1$  in parallel do   
3: Read  $\hat{h}_{L_{k - 1}}^{t - k},\hat{o}_{L_{k - 1}}^{t - k}$  from storage if  $k\neq 0$    
4: Compute  $\hat{h}_{L_k}^{t - k},\hat{o}_{L_k}^{t - k}$    
5: Send  $\hat{h}_{L_k}^{t - k},\hat{o}_{L_k}^{t - k}$  to next worker's storage if  $k\neq K - 1$    
6: end for   
7: Broadcast  $\hat{o}_{L_K - 1}^{t - K + 1}$    
8: for  $k = 0,1,\dots ,K - 1$  in parallel do   
9: Compute  $\Delta w_{\mathcal{G}(k)}^{t - K + 1} = \hat{o}_{L_{K - 1}}^{t - K + 1}u_{w_{\mathcal{G}(k)}}^{t - K + 1}$    
10: Update  $w_{\mathcal{G}(k)}^{t - K + 2} = w_{\mathcal{G}(k)}^{t - K + 1} - \gamma_{t - K + 1}\Delta w_{\mathcal{G}(k)}^{t - K + 1}$    
11: end for   
12: end for
Additionally, the procedure in line 5 could overlap with the operations in line 7,9 and 10. We can also apply this approximation algorithm to gradient-based methods like Adam [18] with little modification to the original. Details can be found in Appendix G.1.
Tangent checkpoint. However, some workers, especially those which are closer to the input, store duplicated tangent vectors. To tackle this issue, we use tangent checkpoint,i.e., storing the seed of tangent vectors and reproducing them in the update stage.
Integration with Efficient Transfer Learning Although AsyncFGD offers several advantages over backpropagation, it shares a common challenge with random Zeroth-Order Optimization and Evolution Strategy methods: the variance of the approximation increases with the dimension of random perturbations[25]. Reducing the learning rate can help but may result in slower convergence. However, we observe that deploying models on edge devices typically involves fine-tuning rather
than training from scratch. Our method can flexibly incorporate the principles of efficient transfer learning by introducing a scaling factor  $\alpha \in [0,1]$  to the randomly sampled tangent:
$$
u _ {w l} ^ {\prime} = u _ {w l} \cdot \alpha_ {w l}
$$
The modified  $u_{w_l}'$  still supports an approximation of the true gradient, with the expectation of the modified estimator being  $\alpha_{w_l}^2 \nabla f(x; w_l)$ . When  $\alpha_{w_l}$  is set to 0, the corresponding parameter is "frozen," resulting in no perturbation or updates and a transparent transmission of JVP values. By adjusting  $\alpha$  to various values, we can either fix or reduce the influence and learning of specific layers, aligning our approach with the objectives of efficient transfer learning.
4.3 Acceleration of AsyncFGD
When  $K = 1$  the AsyncFGD is reduced to vanilla FGD without any time delay in parameters. When  $K \geq 2$ , we can distribute the network across multiple workers. Figure 2 shows the computational time of different algorithms in ideal conditions (i.e. the network is evenly distributed and the communication is overlapped by computation and update).  $T_{F}$  denotes the time for forward pass and  $T_{U}$  denotes the time for updates. It is easy to see that AsyncFGD can fully utilize the computation resources, thus achieving considerable speedup.
Method	Training Progress (K=3)	Computation Time
FGD	TF+TU
Sync FGD	TF+TU
Async FGD	Forward Update Idle	TF/K+TU
Figure 2: Comparison of computation time and training process when the network is deployed across  $K$  workers. Since  $T_{F}$  is much larger than  $T_{U}$ , AsyncFGD can achieve considerable speedup.
5 Convergence Analysis
In this section, we provide the convergence guarantee of Algorithm 1 to critical points in a non-convex setup. We first make the following basic assumptions for nonconvex optimization:
Assumption 5.1. The gradient of  $f(w)$  is Lipschitz continuous with Lipschitz constant  $L > 0$ , i.e.,
$$
\forall x, y \in \mathbb {R} ^ {d}, \| \nabla f (x) - \nabla f (y) \| \leq L \| x - y \|.
$$
Assumption 5.2. The second moment of stochastic gradient is bounded, i.e., there exist a constant  $M \geq 0$ , for any sample  $x_{i}$  and  $\forall w \in \mathbb{R}^{d}$ :
$$
\| \nabla f _ {x _ {i}} (w) \| ^ {2} \leq M.
$$
Lemma 5.3 (Mean and Variance). Let  $t' = t - K + 1$  and diagonal matrices  $\mathbf{I}_0, \dots, \mathbf{I}_k, \dots, \mathbf{I}_{K-1} \in \mathbb{R}^{d \times d}$  such that all the principle diagonal elements of  $\mathbf{I}_k$  in  $\mathcal{G}(k)$  are 1, and all the principle diagonal elements of  $I_k$  in other than  $\mathcal{G}(k)$  are 0. Then we can obtain the mean value and the variance of the forward gradient as follows,
$$
\mathbb {E} _ {u _ {w _ {g} (k)} ^ {t ^ {\prime}}} \left(o _ {L _ {K - 1}} ^ {t ^ {\prime}} \cdot u _ {w _ {g (k)}} ^ {t ^ {\prime}}\right) = \nabla f _ {g (k), x _ {i (t ^ {\prime})}} \left(w ^ {t ^ {\prime} - K + k + 1}\right), \tag {10}
$$
$$
\mathbb {E} _ {u _ {w} ^ {t ^ {\prime}}} \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \cdot \hat {\sigma} _ {L _ {K - 1}} ^ {t ^ {\prime}} \cdot u _ {w _ {\mathcal {G} (k)}} ^ {t ^ {\prime}} \right\| ^ {2} \leq (d + 4) \left\| \sum_ {k = 0} ^ {K - 1} \mathbf {I} _ {k} \nabla f _ {\mathcal {G} (k), x _ {i \left(t ^ {\prime}\right)}} \left(w ^ {t ^ {\prime} - K + k + 1}\right) \right\| ^ {2}. \tag {11}
$$
Remark 5.4. Note that, from modules 0 to  $K - 1$ , the corresponding time delays are from  $K - 1$  to 0. Specifically, in timestamp  $t'$ , when  $k = K - 1$ ,  $\mathbb{E}_{u_{w_{\mathcal{G}(K - 1)}}^{t'}}(o_L^{t'} \cdot u_{w_{\mathcal{G}(K - 1)}}^{t'}) = \nabla f_{\mathcal{G}(K - 1),x_{i(t')}}(w^{t'})$  indicates that the forward gradient in module  $K - 1$  is an unbiased estimation of the up-to-date gradient.
Under Assumption 5.1 and 5.2, we obtain the following descent lemma about the objective function value:
Lemma 5.5. Assume that Assumption 5.1 and 5.2 hold. In addition, let  $t' = t - K + 1$ ,  $\sigma := \max_{t'} \frac{\gamma_{\max\{0,t' - K + 1\}}}{\gamma_{t'}}$ ,  $M_K = KM + \sigma K^4 M$  and choose  $\gamma_{t'} \leq \frac{1}{L}$ . The iterations in Algorithm 1 satisfy the following descent property in expectation,  $\forall t' \in \mathbb{N}$ :
$$
\mathbb {E} \left[ f \left(w ^ {t ^ {\prime} + 1}\right) \right] - f \left(w ^ {t ^ {\prime}}\right) \leq - \frac {\gamma t ^ {\prime}}{2} \| \nabla f \left(w ^ {t ^ {\prime}}\right) \| ^ {2} + 4 (d + 4) L \gamma_ {t ^ {\prime}} ^ {2} M _ {K}, \tag {12}
$$