SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since we are hiding samples when computing the overall loss function  $F(\cdot)$ , we assume each of the functions  $f_{i}(.)$  shares the same minimum value  $\min_{\mathbf{w}}f_{i}(\mathbf{w}) = \min_{\mathbf{w}}f_{j}(\mathbf{w})\forall i,j$ . We extend the proof of the theorem on the guarantees for a linear rate of convergence for smooth functions with strong convexity [35] to the non-convex landscape obtained when training with hidden samples (proof in Appendix A)
Lemma 1. Let  $F(\mathbf{w}) = \mathbb{E}[f_i(\mathbf{w})]$  be non-convex. Set  $\sigma^2 = \mathbb{E}[\|\nabla f_i(\mathbf{w}_\mathbf{M})\|^2]$  with  $\mathbf{w}^* := \arg \min F(\mathbf{w})$ . Suppose  $\eta \leq \frac{1}{\sup_i L_i}$ . Let  $\Delta_t = \mathbf{w}_t - \mathbf{w}$ . After  $T$  iterations, SGD satisfies:
$$
\mathbb {E} \left[ \| \Delta_ {T} \| ^ {2} \right] \leq (1 - 2 \eta \hat {C}) ^ {T} \| \Delta_ {0} \| ^ {2} + \eta R _ {\sigma} \tag {9}
$$
where  $\hat{C} = \lambda (1 - \eta \sup_{i}L_{i})$  and  $R_{\sigma} = \frac{\sigma^2}{\hat{C}}$
Since the losses  $f_{i}(\mathbf{w})$  are effectively dropping for individual samples, driven by the weight update, we thus drop the maximum fraction that can be hidden to satisfy Eq. 9. Specifically, we suggest selecting a reasonable number that is not too high at the first epoch, e.g.,  $F = 0.3$ . We then adjust the maximum fraction per epoch (denoted as  $F_{e}$ ) to achieve  $F_{e}$ . We suggest using step scheduling, i.e., to reduce the maximum hiding fraction gradually with a factor of  $\alpha$  by the number of epochs increases. For example, we set  $\alpha$  as [1, 0.8, 0.6, 0.4] at epoch [0, 30, 60, 80] for ImageNet-1K and [0, 60, 120, 180] for CIFAR-100, respectively.
3.4 Update Loss and Prediction
Our technique is inspired by an observation that the importance of each sample of the local data does not change abruptly across multiple SGD iterations [22]. We propose to reuse the loss and historical prediction confidence, computed during the training process, and only recompute those metrics for samples from the hidden list. Specifically, the loss and historical prediction confidence of samples are computed only one time at each epoch, i.e., when the samples are fed to the forward pass. It is not re-calculated at the end of each epoch based on the latest model. Therefore, only samples of the last training iteration of a given epoch have an up-to-date loss. Furthermore, if we re-calculate the loss of hidden samples, i.e., only skip the backward and weight update pass of these samples, the loss of hidden samples is also up-to-date. For instance, if we cut off  $20\%$  of samples, we have nearly  $20\%$  up-to-date losses and  $80\%$  of not-up-to-date losses at the end of each epoch. As the result, in comparison to the baseline scenario, KAKURENBO helps to reduce the total backward and weight update time by a fraction of  $F_{e}$  while it does not require any extra forward time
We evaluate KAKURENBO using several models on various datasets. We measure the effectiveness of our proposed method on two large datasets. We use Resnet50 [36] and EfficientNet [37] on ImageNet-1K [19], and DeepCAM [4], a scientific image segmentation model with its accompanying dataset. To confirm the correctness of the baseline algorithms we also use WideResNet-28-10 on the CIFAR-100 dataset. Details of experiment settings and additional experiments such as ablation studies and robustness evaluation are reported in Appendix-B and Appendix-C. We compare the following training strategies:
- **Baseline:** We follow the original training regime and hyper-parameters suggested by their authors using uniform sampling without replacement.
- Importance Sampling With Replacement [11] (ISWR): In each iteration, each sample is chosen with a probability proportional to its loss. The with-replacement strategy means that a sample may be selected several times during an epoch, and the total number of samples fed to the model is the same as the baseline implementation.
- FORGET is an online version of a pruning technique [13]: instead of fully training the model using the whole dataset before pruning, we train it for 20 epochs, and a fraction  $F$  of forgettable samples (i.e. samples that are always correctly classified) are pruned from the dataset<sup>1</sup>. The training then restarts from epoch 0. We report the total training time that includes the 20 epochs of training with the whole dataset, and the full training with the pruned dataset.
- Selective Backprop (SB) [17] prioritizes samples with high loss at each iteration. It performs the forward pass on the whole dataset, but only performs backpropagation on a subset of the dataset.
- Grad-Match [18] trains using a subset of the dataset. Every  $R$  epoch, a new subset is selected so that it would minimize the gradient matching error.
- KAKURENBO: our proposed method where samples are hidden dynamically during training.
It is worth noting that we follow the hyper-parameters reported in [38] for training ResNet-50, [39] for training WideResNet-28-10, [37] for training EfficientNet-b3, and [4] for DeepCAM. We show the detail of our hyper-parameters in Appendix B. We configure ISWR, and FORGET to remove the same fraction  $F$  as KAKURENBO. For SB, we use the  $\beta = 1$  parameter that results in removing  $50\%$  of samples. Unless otherwise mentioned, our default setting for the maximum hidden fraction
<sup>1</sup>We choose the samples to remove by increasing number of forgetting events as in [13].
Figure 2: Convergence and speedup of KAKURENBO and importance sampling (ISWR).
Table 2: Max testing accuracy (Top-1) in percentage of KAKURENBO in the comparison with those of the Baseline and other SOTA methods. Diff. represent the gap to the Baseline.
Setting	CIFAR-100 WRN-28-10	ImageNet-1K ResNet-50	EfficientNet-b3	DeepCAM
Acc.	Diff.	Acc.	Diff.	Acc.	Diff.	Acc.	Diff.
Baseline	77.49	74.89	76.63	78.14
ISWR	76.51	(-0.98)	74.91	(+0.02)	N/A	75.75	(-2.39)
FORGET	76.14	(-1.35)	73.70	(-1.20)	N/A	N/A
SB	77.03	(-0.46)	71.37	(-3.52)	N/A	N/A
KAKURENBO	77.21	(-0.28)	75.15	(+0.26)	76.23	(-0.5)	77.42	(-0.9)
Table 3: Comparison with GradMatch in a single GPU (cutting fraction is set to 0.3.
Setting	CIFAR-100 ResNet-18
Acc.	Time (sec)
Baseline	77.98	8556
Grad-Match-0.3	76.87	8104
(-1.11)	(-5.3%)
KAKURENBO-0.3	77.05	8784
(-0.93)	(+2.7%)
$F$  for KAKURENBO is  $30\%$ , except for the CIFAR-100 small dataset, for which we use  $10\%$  (see below).
To maintain fairness in comparisons between KAKURENBO and other state-of-the-art methods, we use the same model and dataset with the same hyper-parameters. This would mean we are not capable of using state-of-the-art hyper-parameters tuning methods to improve the accuracy of ResNet-50/ImageNet (e.g., as in [40]). That is since the state-of-the-art hyper-parameters tuning methods are not applicable to some of the methods we compare with. Particularly, we can not apply GradMatch for training with a large batch size on multiple GPUs. Thus, we compare KAKURENBO with GradMatch using the setting reported in [18], i.e., CIFAR-100 dataset, ResNet-18 model.
4.1 Accuracy
The progress in the top-1 test accuracy with a maximum hiding fraction of 0.3 is shown in Figure 2. Table 2 summarizes the final accuracy for each experiment. We present data on the small dataset of CIFAR-100 to confirm the correctness of our implementation of ISWR, FORGET, and SB. Table 3 reports the single GPU accuracy obtained with Grad-Match because it cannot work on distributed systems. For CIFAR-100, we report similar behavior as reported in the original work on ISWR [11], SB [17], FORGET [13], and Grad-Match [18]: ISWR, FORGET, and Grad-Match degrade accuracy by approximately  $1\%$ , while SB and KAKURENBO roughly perform as the baseline. KAKURENBO on CIFAR-100 only maintains the baseline accuracy for small fractions (e.g.  $F = 0.1$ ). When hiding a larger part of the dataset, the remaining training set becomes too scarce, and the model does not generalize well.
On the contrary, on large datasets such as ImageNet-1K, ISWR and KAKURENBO slightly improve accuracy (by 0.2) in comparison to the baseline, while FORGET and SB degrade accuracy by  $1.2\%$  and  $3.5\%$ , respectively. On DeepCAM, KAKURENBO does not affect the accuracy while ISWR degrades it by  $2.4\%$  in comparison to the baseline<sup>2</sup>. Table 4 reports the accuracy obtained for transfer learning. We do not report Grad-Match results because we could not apply it to this application. Using SB significantly degrades accuracy compared to the baseline, while ISWR, FORGET, and KAKURENBO maintains the same accuracy as the baseline. Especially, as reported in Figure 3, the testing accuracy obtained by KAKURENBO are varied when changing the maximum hiding fraction. We observe that for small hiding fractions, KAKURENBO achieves the same accuracy as
2We confirm the same behaviors of KAKURENBO and other methods with different hyper-parameters (as shown in Appendix-C).
Figure 3: Test accuracy vs. epoch of KAKURENBO with different maximum hiding fractions  $F$ .
Table 4: Impact of KAKURENBO in transfer learning with DeiT-Tiny-224 model.
Dataset	Metrics	Baseline	ISWR	FORGET	SB	KAKUR.
Up stream	Fractal-3K	Loss	3.26	3.671	3.27	4.18	3.59
Time (min)	623	719	533	414	529
Impr.	-	(+15.4%)	(-14.4%)	(-33.5%)	(-15.1%)
Down stream	CIFAR-10	Acc. (%)	95.03	95.79	95.85	93.59	95.28
Diff.	-	(+0.76)	(+0.82)	(-1.44)	(+0.25)
CIFAR-100	Acc. (%)	79.69	79.62	79.95	76.98	79.35
Diff.	-	(-0.07)	(+0.26)	(-2.71)	(-0.34)
Table 5: Impact of  $\tau$  (prediction confidence threshold) on the performance of KAKURENBO.
Setting	CIFAR-100
WRN-28-10
Acc.	Time (sec)
τ = 0.5	76.37	753.9
τ = 0.7	76.81	758.9
τ = 0.9	76.92	760.7
the baseline. When increasing hiding fractions, as expected, the degradation of the testing accuracy becomes more significant.
4.2 Convergence Speedup and Training Time
Here we discuss KAKURENBO's impact on training time. Figure 2 reports test accuracy as the function of elapsed time (note the X-axis), and reports the training time to a target accuracy. Table 4 reports the upstream training time of DeiT-Tiny-224. The key observation of these experiments is that KAKURENBO reduces the training time of Wide-ResNet by  $21.7\%$ , of ResNet-50 by  $23\%$ , of EfficientNet by  $13.7\%$ , of DeepCAM by  $22.4\%$ , and of DeiT-Tiny by  $15.1\%$  in comparison to the baseline training regime.
Surprisingly, Importance Sampling With Replacement (ISWR) [11] introduces an overhead of  $34.8\%$  on WideNet, of  $41\%$  on ImageNet-1K and offers only a slight improvement of  $2.5\%$  on DeepCAM. At each epoch, ISWR processes the same number of samples as the baseline. Yet, it imposes an additional overhead of keeping track of the importance (i.e., the loss) of all input samples. While on DeepCAM it achieves a modest speedup due to its faster convergence, these experiments reveal that ISWR's behavior is widely different on large datasets than on the smaller ones previously reported [11, 17].
FORGET increases the training time of WideResNet by  $46.1\%$  because of the additional 20 epochs training on the whole dataset needed for pruning the samples. When the number of epoch is large, such as for ResNet50 that runs for 600 epochs, FORGET decreases the training time by  $17.9\%$ , and for DeiT by  $14.4\%$ . However, this reduction of training time comes at the cost of degradation of the test accuracy. On WideResNet and ResNet, SB performs similarly to KAKURENBO by reducing the training time without altering the accuracy. However, SB significantly degrades accuracy compared to the baseline for ImageNet and DeiT.
It is worth noting that KAKURENBO has computation overheads for updating the loss and prediction (Step D in Figure 1), and sorting the samples based on the loss (Step A in Figure 1). For example, Figure 4 reports the measured speedup per epoch as compared to the baseline epoch duration. The speedup follows the same trend as the hiding rate. This is because reducing the number of samples in the training set impacts the speed of the training. The measured speedup does not reach the maximum hiding rate because of the computation overhead. The performance gain from hiding samples will be limited if the maximum hiding fraction  $F$  is small, or potentially less than the overhead to compute the importance score of samples. In experiments using multiple GPUs, those operations are performed in parallel to reduce the running time overhead. When using a single GPU on CIFAR-100 with ResNet-18 (Table 3), the computational overhead is bigger than the speedup gained from hiding
Figure 4: Reduction of hiding fraction, per epoch, and the resulting speedup.
Table 6: The impact of different components of KAKURENBO on testing accuracy including HE: Hiding  $F\%$  lowest-loss examples, MB: Moving Back, RF: Reducing the Fraction by epoch, LR: Adjusting Learning Rate. Numbers inside the (. ) indicate the gap in percentage compared to the full version of KAKURENBO.
Component	Accuracy
HE	MB	RF	LR
Baseline	×	×	×	×	73.68
v1000	✓	×	×	×	72.25 (-1.8%)
v1001	✓	×	×	✓	73.08 (-0.7%)
v1010	✓	×	✓	×	72.81 (-1.1%)
v1011	✓	×	✓	✓	73.27 (-0.4%)
v1100	✓	✓	×	×	72.37 (-1.7%)
v1101	✓	✓	×	✓	73.09 (-0.7%)
v1110	✓	✓	✓	×	72.96 (-0.9%)
KAKUR. (v1111)	✓	✓	✓	✓	73.6
samples. Thus, KAKURENBO takes more training time in this case. In short, KAKURENBO is optimized for large-scale training and provides more benefits when running on multiple GPUs.
Impact of prediction confidence threshold  $\tau$ . Higher prediction confidence threshold  $\tau$  leads to a higher number of samples being moved back to the training set, i.e., fewer hidden samples at the beginning of the training process. At the end of the training process, when the model has is well-trained, more samples are predicted correctly with high confidence. Thus the impact of the prediction confidence threshold on the number of moved-back samples becomes less (as shown in Figure 4). The result in Table 5 shows that when we increase the threshold  $\tau$ , we obtain better accuracy (fewer hidden samples), but at the cost of smaller performance gain. We suggest to set  $\tau = 0.7$  in all the experiments as a good trade-off between training time and accuracy.
Impact of different components of KAKURENBO. We evaluate how KAKURENBO's individual internal strategies, and their combination, affect the testing accuracy of a neural network. Table 6 reports the results we obtained when training ResNet-50 on ImageNet-1K<sup>3</sup> with a maximum hiding fraction of  $40\%$ . The results show that when only HE (Hiding Examples) of the  $40\%$  lowest loss samples is performed, accuracy slightly degrades. Combining HE with other strategies, namely MB (Move-Back), RF (Reducing Fraction), and LR (Learning Rate adjustment) gradually improves testing accuracy. In particular, all combinations with RF achieve higher accuracy than the ones without it. For example, the accuracy of v1110 is higher than that of v1100 by about  $0.59\%$ . We also observe that using LR helps to improve the training accuracy by a significant amount, i.e., from  $0.46\%$  to  $0.83\%$ . The MB strategy also improves accuracy. For example, the accuracy of v1010 is  $72.81\%$ , compared to v1110 which is  $72.96\%$ . This small impact of MB on the accuracy is due to moving back samples at the beginning of the training, as seen in Appendix C.3. By using all the strategies, KAKURENBO achieves the best accuracy of  $73.6\%$ , which is very close to the baseline of  $73.68\%$ .
We have proposed KAKURENBO, a mechanism that adaptively hides samples during the training of deep neural networks. It assesses the importance of samples and temporarily removes the ones that would have little effect on the SGD convergence. This reduces the number of samples to process at each epoch without degrading the prediction accuracy. KAKURENBO combines the knowledge of historical prediction confidence with loss and moves back samples to the training set when necessary. It also dynamically adapts the learning rate in order to maintain the convergence pace. We have demonstrated that this approach reduces the training time without significantly degrading the accuracy on large datasets.
3 We use the ResNet-50 (A) configuration in this evaluation as shown in Appendix-B
This work was supported by JSPS KAKENHI under Grant Numbers JP21K17751 and JP22H03600. This paper is based on results obtained from a project, JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO). This work was supported by MEXT as "Feasibility studies for the next-generation computing infrastructure" and JST PRESTO Grant Number JPMJPR20MA.
We thank Rio Yokota and Hirokatsu Kataoka for their support on the Fractal-3K dataset.
[33] L. Wu, C. Ma, and W. E., “How SGD Selects the Global Minima in Over-Parameterized Learning: A Dynamical Stability Perspective,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, ser. NIPS'18, 2018, p. 8289–8298. 6
[34] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour,” CoRR, vol. abs/1706.02677, 2017. 6, 16
[35] D. Needell, N. Srebro, and R. Ward, "Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm," in Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1, ser. NIPS'14, 2014, p. 1017-1025. 6
[36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770–778. 7, 15
[37] M. Tan and Q. Le, "EfficientNet: Rethinking model scaling for convolutional neural networks," in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09-15 Jun 2019, pp. 6105-6114. [Online]. Available: https://proceedings.mlr.press/v97/tan19a.html 7, 15
[38] V. Vryniotis, "How to train state-of-the-art models using torchvision's latest primitives." [Online]. Available: https://torch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/ 7, 15
[39] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in BMVC, September 2016, pp. 87.1–87.12. 7, 15
[40] R. Wightman, H. Touvron, and H. Jégou, “Resnet strikes back: An improved training procedure in timm,” 2021. 8
[41] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Master's thesis, Dep. of Comp. Sci. Univ. of Toronto, 2009. 15
[42] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou, “Training data-efficient image transformers & distillation through attention,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18-24 Jul 2021, pp. 10347-10357. [Online]. Available: https://proceedings.mlr.press/v139/touvron21a.html 15
[43] Y. Yang, H. Kang, and B. Mirzasoleiman, “Towards sustainable learning: Coresets for data-efficient deep learning,” in Proceedings of the 40th International Conference on Machine Learning, ser. ICML'23. JMLR.org, 2023. 20
[44] Y. Wu, Z. Wang, Y. Shi, and J. Hu, "Enabling on-device cnn training by self-supervised instance filtering and error map pruning," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 39, no. 11, pp. 3445-3457, 2020. 23
[45] Y. Wang, Z. Jiang, X. Chen, P. Xu, Y. Zhao, Y. Lin, and Z. Wang, “E2-train: Training state-of-the-art cnns with over  $80\%$  energy savings,” Advances in Neural Information Processing Systems, vol. 32, 2019. 23
Appendix A. Proof of Lemma 1
Lemma 1. Let  $F(\mathbf{w}) = \mathbb{E}[f_i(\mathbf{w})]$  be non-convex. Set  $\sigma^2 = \mathbb{E}[\|\nabla f_i(\mathbf{w}_\mathbf{M})\|^2]$  with  $\mathbf{w}^* := \text{argmin}F(\mathbf{w})$ . Suppose  $\eta \leq \frac{1}{\sup_i L_i}$ . Let  $\Delta_t = \mathbf{w}_t - \mathbf{w}$ . After  $T$  iterations, SGD satisfies:
$$
\mathbb {E} \left[ \| \Delta_ {T} \| ^ {2} \right] \leq (1 - 2 \eta \hat {C}) ^ {T} \| \Delta_ {0} \| ^ {2} + \eta R _ {\sigma}
$$
Proof.  $\| \nabla f_i(\mathbf{w})\| = 0$  in the noiseless setting, and so  $\sigma \coloneqq 0$ . For  $\pmb{x}_k$  being the input at  $i$  random index for iteration  $k$ , there exists a parameter  $\lambda_{\mathbf{w_t}}$  for  $\lambda_{max}$  (Eq. 7), and  $w = w_{\lambda}$ , we have for step size  $\gamma$
$$
\begin{array}{l} \mathbb {E} \left[ \| \Delta_ {T} \| ^ {2} \right] = \| \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} - \gamma \nabla f _ {i} (\boldsymbol {x} _ {k}) \| ^ {2} \\ = \left\| \left(\boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star}\right) - \gamma \left(\nabla f _ {i} (\boldsymbol {x} _ {k}) - \nabla f _ {i} (\boldsymbol {x} _ {\star})\right) - \gamma \nabla f _ {i} (\boldsymbol {x} _ {\star}) \right\| ^ {2} \\ = \| \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} \| ^ {2} - 2 \gamma \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} * \nabla f _ {i} (\boldsymbol {x} _ {k}) + \gamma^ {2} \| \nabla f _ {i} (\boldsymbol {x} _ {k}) - \nabla f _ {i} (\boldsymbol {x} _ {\star}) + \nabla f _ {i} (\boldsymbol {x} _ {\star}) \| ^ {2} \\ \leq \left\| \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} \right\| ^ {2} - 2 \gamma \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} * \nabla f _ {i} (\boldsymbol {x} _ {k}) + 2 \gamma^ {2} \left\| \nabla f _ {i} (\boldsymbol {x} _ {k}) - \nabla f _ {i} (\boldsymbol {x} _ {\star}) \right\| ^ {2} + 2 \gamma^ {2} \left\| \nabla f _ {i} (\boldsymbol {x} _ {\star}) \right\| ^ {2} \\ \leq \left\| \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} \right\| ^ {2} - 2 \gamma \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} * \nabla f _ {i} (\boldsymbol {x} _ {k}) \\ + 2 \gamma^ {2} L _ {i} \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} + \nabla f _ {i} (\boldsymbol {x} _ {k}) - \nabla f _ {i} (\boldsymbol {x} _ {\star}) + 2 \gamma^ {2} \| \nabla f _ {i} (\boldsymbol {x} _ {\star}) \| ^ {2} \\ \end{array}
$$
where we employ Jensen's inequality in the first inequality for  $\sigma^2 = \mathbb{E}[\| \nabla f_i(\mathbf{w_M})\|^2]$ . Then  $\mathbb{E}[\nabla f_i(\pmb{x})] = F(\pmb{x})$ , and we obtain
$$
\begin{array}{l} \mathbb {E} \left[ \| \Delta_ {T} \| ^ {2} \right] \leq \| \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} \| ^ {2} - 2 \gamma \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} * F (\boldsymbol {x} _ {k}) + 2 \gamma^ {2} \mathbb {E} \left[ L _ {i} \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star}, \nabla f _ {i} (\boldsymbol {x} _ {k}) - \nabla f _ {i} (\boldsymbol {x} _ {\star}) \right] \\ + 2 \gamma^ {2} \mathbb {E} \| \nabla f _ {i} (\boldsymbol {x} _ {\star}) \| ^ {2} \\ \leq \| \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} \| ^ {2} - 2 \gamma \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} * F (\boldsymbol {x} _ {k}) + 2 \gamma^ {2} \sup  _ {i} L _ {i} \mathbb {E} \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star}, \nabla f _ {i} (\boldsymbol {x} _ {k}) - \nabla f _ {i} (\boldsymbol {x} _ {\star}) \\ + 2 \gamma^ {2} \mathbb {E} \| \nabla f _ {i} (\boldsymbol {x} _ {\star}) \| ^ {2} \\ = \| \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} \| ^ {2} - 2 \gamma \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star} * F (\boldsymbol {x} _ {k}) + 2 \gamma^ {2} \sup  L \boldsymbol {x} _ {k} - \boldsymbol {x} _ {\star}, F (\boldsymbol {x} _ {k}) - F (\boldsymbol {x} _ {\star}) + 2 \gamma^ {2} \sigma^ {2} \\ \end{array}
$$
when  $\gamma \leq \frac{1}{\sup L}$ . Recursively applying this bound over the first  $k$  iterations yields the desired result
$$
\begin{array}{l} \mathbb {E} \left[ \| \Delta_ {T} \| ^ {2} \right] \leq \Big (1 - 2 \gamma \mu (1 - \gamma) \Big) \Big) ^ {k} \| \boldsymbol {x} _ {0} - \boldsymbol {x} _ {\star} \| ^ {2} + 2 \sum_ {j = 0} ^ {k - 1} \Big (1 - 2 \gamma \mu (1 - \gamma) \Big) \Big) ^ {j} \gamma^ {2} \sigma^ {2} \\ \left. \leq \left(1 - 2 \gamma \mu (1 - \gamma)\right)\right) ^ {k} \| \boldsymbol {x} _ {0} - \boldsymbol {x} _ {\star} \| ^ {2} + \frac {\gamma \sigma^ {2}}{\mu (1 - \gamma)}. \\ \end{array}
$$
Appendix B. Experiments Details
B.1. System detail