SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This is a star-  $M$  -smooth-convex function as in Definition 2 with  $M = 2L$
For the analysis of shuffling type algorithm in this paper, we consider the general assumption called the generalized star-smooth-convex condition for shuffling algorithms:
Assumption 5. Using Algorithm 1, let us assume that there exist some constants  $M > 0$  and  $N \geq 0$  such that at each epoch  $t = 1, \dots, T$ , we have for  $i = 1, \dots, n$ :
$$
\begin{array}{l} \left\| \nabla f (w _ {i - 1} ^ {(t)}; \pi^ {(t)} (i)) - \nabla f (w _ {*}; \pi^ {(t)} (i)) \right\| ^ {2} \leq M \langle \nabla f (w _ {i - 1} ^ {(t)}; \pi^ {(t)} (i)) - \nabla f (w _ {*}; \pi^ {(t)} (i)), w _ {i - 1} ^ {(t)} - w _ {*} \rangle \\ + N \frac {1}{n} \sum_ {i = 1} ^ {n} \| w _ {i} ^ {(t)} - w _ {0} ^ {(t)} \| ^ {2}, \tag {13} \\ \end{array}
$$
where  $w_{*}$  is a global solution of  $F$ .
We note that it is sufficient for our analysis to assume the case  $N = 0$ , i.e. the individual function  $f(\cdot ;i)$  is star- $M$ -smooth-convex with respect to  $w_{*}$  for every  $i = 1,\dots ,n$  as in Definition 2. In that case, the assumption does not depend on the algorithm progress.
Assumption 5 is more flexible than the star-  $M$ -smooth-convex one in (10) because an additional term  $N \frac{1}{n} \sum_{i=1}^{n} \| w_i^{(t)} - w_0^{(t)} \|^2$  for some constant  $N > 0$  allows for extra flexibility in our setting, where the right-hand side term  $\langle \nabla f(w; i) - \nabla f(w_*, i), w - w_* \rangle$  could be negative for some  $w \in \mathbb{R}^d$ .
Note that we do not impose any assumptions on bounded weights or bounded gradients. Therefore, the term  $\frac{1}{n}\sum_{i = 1}^{n}\| w_i^{(t)} - w_0^{(t)}\|^2$  cannot be uniformly bounded by any universal constant.
3 New Framework for Convergence to a Global Solution
In this section, we present our theoretical results. Our Lemma 2 first provides a recursion to bound the squared distance term  $\| \tilde{w}_t - w_*\|^2$ :
Lemma 2. Assume that Assumptions 1, 2, 3, and 5 hold. Let  $\{\tilde{w}_t\}_{t=1}^T$  be the sequence generated by Algorithm 1 with  $0 < \eta_t \leq \min \left\{\frac{n}{2M}, \frac{1}{2L}\right\}$ . For every  $\gamma > 0$  we have
$$
\left\| \tilde {w} _ {t} - w _ {*} \right\| ^ {2} \leq \left(1 + C _ {1} \eta_ {t} ^ {3}\right) \left\| \tilde {w} _ {t - 1} - w _ {*} \right\| ^ {2} + C _ {2} \eta_ {t} \sigma_ {*} ^ {2} - C _ {3} \eta_ {t} [ F (\tilde {w} _ {t - 1}) - F _ {*} ]. \tag {14}
$$
where  $w_{*}$  is a global solution of  $F$ ,  $F_{*} = \min_{w\in \mathbb{R}^{d}}F(w)$ , and
$$
\left\{ \begin{array}{l} C _ {1} = \frac {8 L ^ {2}}{3} + \frac {1 4 N L ^ {2}}{M} + \frac {4 \gamma L ^ {4}}{6 M}, \\ C _ {2} = \frac {2}{M} + 1 + \frac {5}{6 L ^ {2}} + \frac {8 N}{3 M L ^ {2}} + \frac {5 \gamma}{1 2 M}, \\ C _ {3} = \frac {\gamma}{\gamma + 1} \frac {\mu}{M}. \end{array} \right. \tag {15}
$$
Rearranging the results of Lemma 2, we have
$$
F \left(\tilde {w} _ {t - 1}\right) - F _ {*} \leq \frac {1}{C _ {3}} \left(\frac {1}{\eta_ {t}} + C _ {1} \eta_ {t} ^ {2}\right) \| \tilde {w} _ {t - 1} - w _ {*} \| ^ {2} - \frac {1}{C _ {3} \eta_ {t}} \| \tilde {w} _ {t} - w _ {*} \| ^ {2} + \frac {C _ {2}}{C _ {3}} \sigma_ {*} ^ {2}. \tag {16}
$$
Therefore, with an appropriate choice of learning rate that guarantee  $\left(1 / \eta_t + C_1\eta_t^2\right)\leq 1 / \eta_{t - 1}$ , we can unroll the recursion from Lemma 2. Thus we have our main result in the next Theorem.
Theorem 2. Assume that Assumptions 1, 2, 3, and 5 hold. Let  $\{\tilde{w}_t\}_{t=1}^T$  be the sequence generated by Algorithm 1 with the learning rate  $\eta_i^{(t)} = \frac{\eta_t}{n}$  where  $0 < \eta_t \leq \min \left\{\frac{n}{2M}, \frac{1}{2L}\right\}$ .
Let the number of iterations  $T = \frac{\lambda}{\varepsilon^{3/2}}$  for some  $\lambda > 0$  and  $\varepsilon > 0$ . Constants  $C_1, C_2,$  and  $C_3$  are defined in (15) for any  $\gamma > 0$ . We further define  $K = 1 + C_1 D^3 \varepsilon^{3/2}$  and specify the learning rate  $\eta_t = K \eta_{t-1} = K^t \eta_0$  and  $\eta_0 = \frac{D \sqrt{\varepsilon}}{K \exp(\lambda C_1 D^3)}$  such that  $\frac{D \sqrt{\varepsilon}}{K} \leq \min \left\{ \frac{n}{2M}, \frac{1}{2L} \right\}$  for some constant  $D > 0$ . Then we have
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} [ F (\tilde {w} _ {t - 1}) - F _ {*} ] \leq \frac {K \exp \left(\lambda C _ {1} D ^ {3}\right)}{C _ {3} D \lambda} \| \tilde {w} _ {0} - w _ {*} \| ^ {2} \cdot \varepsilon + \frac {C _ {2}}{C _ {3}} \sigma_ {*} ^ {2}, \tag {17}
$$
where  $F_{*} = \min_{w\in \mathbb{R}^{d}}F(w)$  and  $\sigma_{*}^{2}$  is defined in (8).
Our analysis holds for arbitrarily constant values of the parameters  $\gamma, \lambda$  and  $D$ . In addition, we show our current analysis for every shuffling scheme. An interesting research question arises: whether the convergence results can be improved if one chooses to analyze a randomized shuffling scheme in this framework. However, we leave that question to future works.
Using Assumption 4, we can show the total complexity of Algorithm 1 for our setting.
Table 1: Comparisons of computational complexity (the number of individual gradient evaluations) needed by SGD algorithm to reach an  $\hat{\varepsilon}$ -accurate solution  $w$  that satisfies  $F(w) - F(w_{*})\leq \hat{\varepsilon}$  (or  $\| \nabla F(w)\| ^2\leq \hat{\varepsilon}$  in the non-convex case).
Settings	References	Complexity	Shuffling Schemes	Global Solution
Convex	Nemirovski et al. [2009], Shamir and Zhang [2013] (1)	O\(\left(\frac{\Delta_0^2+G^2}{\hat{\varepsilon}^2}\right)\)	X	✓
Mishchenko et al. [2020], Nguyen et al. [2021] (2)	O\(\left(\frac{n}{\hat{\varepsilon}^{3/2}}\right)\)	✓	✓
PL condition	Nguyen et al. [2021]	\(\tilde{O}\left(\frac{n\sigma^2}{\hat{\varepsilon}^{1/2}}\right)\)	✓	✓
Star-convex related	Gower et al. [2021] (3)	O\(\left(\frac{1}{\hat{\varepsilon}^2}\right)\)	X	✓
Non-convex	Ghadimi and Lan [2013] (5)	O\(\left(\frac{\sigma^2}{\hat{\varepsilon}^2}\right)\)	X	X
Nguyen et al. [2021], Mishchenko et al. [2020] (5)	O\(\left(\frac{n\sigma}{\hat{\varepsilon}^{3/2}}\right)\)	✓	X
Our setting (non-convex)	This paper, Corollary 1(4)	O\(\left(\frac{n(N \vee 1)^{3/2}}{\hat{\varepsilon}^{3/2}}\right)\)	✓	✓
(0) We note that the assumptions in this table are not comparable and we only show the roughly complexity in terms of  $\hat{\varepsilon}$ . In addition, to make fair comparisons, we only report the complexity of unified shuffling schemes.
(1) Standard results for SGD in convex literature often use a different set of assumptions from the one in this paper (e.g. bounded domain that  $\| w - w_{*}\|^{2} \leq \Delta_{0}$  for each iterate  $w$  and/or bounded gradient that  $\mathbb{E}[\|\nabla f(w;i)\|] \leq G^{2}$ ). We report the corresponding complexity for a rough comparison.
(2) [Mishchenko et al., 2020] shows a bound for Incremental Gradient while [Nguyen et al., 2021] has a unified setting. We translate these results for unified shuffling schemes from these references to the convex setting.
(3) Since we cannot find a reference containing the convergence rate for vanilla SGD and star-convex functions, we adapt the reference Gower et al. [2021] here. This paper shows a result for  $L$ -smooth and quasar convex function with an additional Expected Residual (ER) assumption, which is weaker than assuming smoothness for  $f(\cdot; i)$  and interpolation property. The star-convex results hold when the quasar-convex parameter is 1.
(4) Since we use a different set of assumptions than the other references, we only report the rough comparison in  $n$ ,  $N$  and  $\hat{\varepsilon}$ , where  $N$  is the constant from Assumption 5 and  $N \vee 1 = \max(N, 1)$ . Note that  $N = 0$  in the framework of star-smooth-convex function. In addition, we need  $\sigma_{*}^{2} = 0$  so that the complexity holds with arbitrary  $\hat{\varepsilon}$ . We explain the detailed complexity below and in the Appendix.
(5) Standard literature for SGD in non-convex setting assumes a bounded variance that  $\mathbb{E}_i\left[\| f(w;i) - \nabla F(w)\| ^2\right] \leq \sigma^2$ , we report the rough comparison.
Corollary 1. Suppose that the conditions in Theorem 2 and Assumption 4 hold. Choose  $C_1D\lambda = 1$  and  $\varepsilon = \hat{\varepsilon} / G$  such that  $0 < \hat{\varepsilon} \leq G$  with the constants
$$
G = \frac {2 C _ {1} D ^ {2} e}{C _ {3}} \| \tilde {w} _ {0} - w _ {*} \| ^ {2} + \frac {C _ {2} P}{C _ {3}}, \text {w h e r e}
$$
$$
\left\{ \begin{array}{l} C _ {1} = \frac {8 L ^ {2}}{3} + \frac {1 4 N L ^ {2}}{M} + \frac {4 L ^ {2}}{3 M}, \\ C _ {2} = \frac {2}{M} + 1 + \frac {5}{6 L ^ {2}} + \frac {8 N}{3 M L ^ {2}} + \frac {5}{1 2 M L}, \\ C _ {3} = \frac {1}{L ^ {2} + 1} \frac {\mu}{M}. \end{array} \right.
$$
Then, the we need  $T = \frac{\lambda G^{3/2}}{\hat{\varepsilon}^{3/2}}$  epochs to guarantee
$$
\min  _ {1 \leq t \leq T} [ F (\tilde {w} _ {t - 1}) - F _ {*} ] \leq \frac {1}{T} \sum_ {t = 1} ^ {T} [ F (\tilde {w} _ {t - 1}) - F _ {*} ] \leq \hat {\varepsilon}.
$$
Computational complexity. Our global convergence result in this Corollary holds for a fixed value of  $\hat{\varepsilon}$  in Assumption 4. Thus, when  $\varepsilon \rightarrow 0$ , this assumption is equivalent to assuming  $\sigma_{*}^{2} = 0$ . The total complexity of Corollary 1 is  $\mathcal{O}\left(\frac{n}{\hat{\varepsilon}^{3/2}}\right)$ . This rate matches the best known rate for unified sampling schemes for SGD in convex setting [Mishchenko et al., 2020, Nguyen et al., 2021]. However, our result holds for a broader class of functions that are possibly non-convex. Comparing to the non-convex setting, current literature [Mishchenko et al., 2020, Nguyen et al., 2021] also matches our rate to the order of  $\hat{\varepsilon}$ , however, we can only prove that SGD converges to a stationary point with a weaker criteria  $\| \nabla F(w) \|^2 \leq \hat{\varepsilon}$  for general non-convex functors. Table 1 shows these comparisons in various settings. Note that when using a randomized shuffling scheme, SGD often performs a better rate in terms of the data  $n$  in various settings with and without (strongly) convexity. For example, in strongly convex and/or PL setting, the convergence rate of RR is  $\tilde{\mathcal{O}}(\sqrt{n}/\sqrt{\hat{\varepsilon}})$ , which is better than unified schemes with  $\tilde{\mathcal{O}}(n/\sqrt{\hat{\varepsilon}})$  [Ahn et al., 2020]. However, for a fair comparison, we do not report these results in Table 1 as our theoretical analysis is derived for unified shuffling scheme.
If we further assume that  $L, M, N > 1$ , the detailed complexity with respect to these constants is
$$
\mathcal {O} \left(\frac {L ^ {4} (M + N) ^ {3 / 2}}{\mu^ {3 / 2}} \cdot \frac {n}{\hat {\varepsilon} ^ {3 / 2}}\right).
$$
We present all the detailed proofs in the Appendix. Our theoretical framework is new and adapted to the finite-sum minimization problem. Moreover, it utilizes the choice of shuffling sample schemes to show a better complexity in terms of  $\hat{\varepsilon}$  than the complexity of vanilla i.i.d. sampling scheme.
In this section, we show some experiments for shuffling-type SGD algorithms to demonstrate our theoretical results of convergence to a global solution. Following the setting of Theorem 1, we consider the non-convex regression problem with squared loss function. We choose fully connected neural networks in our implementation. We experiment with different regression datasets: the Diabetes dataset from sklearn library [Efron et al., 2004, Pedregosa et al., 2011] with 442 samples in dimension 10; the Life expectancy dataset from WHO [Repository, 2016] with 1649 trainable data points and 19 features. In addition, we test with the California Housing data from StatLib repository [Repository, 1997, Pedregosa et al., 2011] with a training set of 16514 samples and 8 features.
For the small Diabetes dataset, we use the classic LeNet-300-100 model [LeCun et al., 1998]. For other larger datasets, we use similar fully connected neural networks with an additional starting layer of 900 neurons. We apply the randomized reshuffling scheme using PyTorch framework [Paszke et al., 2019]. This shuffling scheme is the common heuristic in training neural networks and is implemented in many deep learning platforms (e.g. TensorFlow, PyTorch, and Keras [Abadi et al., 2015, Paszke et al., 2019, Chollet et al., 2015]).
For each dataset  $\{x_i, y_i\}$ , we preprocess and modify the initial data lightly to guarantee the overparameterized setting in our experiment. We do this by using a pre-training stage: firstly we use GD/SGD algorithm to find a weight  $w$  that yields a sufficiently small value for the loss function (for Diabetes dataset we train to  $10^{-8}$  and for other datasets we train to  $10^{-2}$ ). Letting the input data  $x_i$  be fixed, we change the label data to  $\hat{y}_i$  such that the weight  $w$  yields a small loss function  $\mathcal{O}(\epsilon)$  for the optimization associated with data  $\{x_i, \hat{y}_i\}$ , and the distance between  $\hat{y}_i$  and  $y_i$  is small. Then the modified data is ready for the next stage. We summarize the data (after modification) in our experiments in Table 2.
Table 2: Datasets used in our experiments
Data name	# Samples	# Features	Networks layers	Sources
Diabetes	442	10	300-100	Efron et al. [2004]
Life Expectancy	1649	19	900-300-100	Repository [2016]
California Housing	16514	8	900-300-100	Repository [1997]
For each dataset, we first tune the step size using a coarse grid search [0.0001, 0.001, 0.01, 0.1, 1] for 100 epochs. Then, for example, if 0.01 performs the best, we test the second grid search [0.002, 0.005, 0.01, 0.02, 0.05] for 5000 epochs. Finally, we progress to the training stage with  $10^{6}$  epochs and repeat that experiment for 5 random seeds. We report the average results with confidence intervals in Figure 1.
Figure 1: The train loss produced by Shuffling SGD algorithm for three datasets: Diabetes, Life Expectancy and California Housing.
For California Housing data, Shuffling SGD fluctuates toward the end of the training process. Nevertheless, for all three datasets it converges steadily to a small value of loss function. In summary, this experiment confirms our theoretical guarantee that demonstrates a convergence to global solution for shuffling-type SGD algorithm in neural network settings.
5 Conclusion and Future Research
In this paper, our focus is on investigating the global convergence properties of shuffling-type SGD methods. We consider a more relaxed set of assumptions in the framework of star-smooth-convex functions. We demonstrate that our approach achieves a total complexity of  $\mathcal{O}\left(\frac{n}{\hat{\varepsilon}^{3/2}}\right)$  to attain an  $\hat{\varepsilon}$ -accurate global solution. Notably, this result aligns with the previous computational complexity of unified shuffling methods in non-convex settings, while ensuring that the algorithm converges to a global solution. Our theoretical framework revolves around the shuffling sample schemes for finite-sum minimization problems in machine learning.
We also provide insights into the relationships between our framework and well-established overparameterized settings, as well as the existing literature on the star-convexity class of functions. Furthermore, we establish connections with neural network architectures and explore how these learning models align with our theoretical optimization frameworks.
This paper prompts several intriguing research questions, including practical network designs and more relaxed theoretical settings that can support the global convergence of shuffling SGD methods. Additionally, extending the global convergence framework to other stochastic gradient methods [Duchi et al., 2011, Kingma and Ba, 2014] and variance reduction methods [Le Roux et al., 2012, Defazio et al., 2014, Johnson and Zhang, 2013, Nguyen et al., 2017], all with shuffling sampling schemes, as well as the exploration of momentum shuffling methods [Tran et al., 2021, 2022], represents a promising direction.
An interesting research question that arises is whether the convergence results can be further enhanced by exploring the potential of a randomized shuffling scheme within this framework [Mishchenko et al., 2020]. However, we leave this question for future research endeavors.
Yi Zhou, Junjie Yang, Huishuai Zhang, Yingbin Liang, and Vahid Tarokh. Sgd converges to global minimum in deep learning via star-convex path. arXiv preprint arXiv:1901.00451, 2019.
Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 2053-2062, 2019.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks, 2018.
On the Convergence to a Global Solution of Shuffling-Type Gradient Algorithms Supplementary Material, NeurIPS 2023
A Theoretical settings: Proof of Theorem 1
Proof. Let us use the notation  $f(w; i) = \phi_i(h(w; i)) = \frac{1}{2} \| h(w; i) - y^{(i)} \|^2$ . We consider an architecture  $h(w; i)$  with  $w$  be the vectorized weight and  $h$  consists of a final bias layer  $b$ :
$$
h (w; i) = W ^ {T} z (\theta ; i) + b \in \mathbb {R} ^ {c},
$$
where  $w = \mathbf{vec}(\{\theta, W, b\})$  and  $z(\theta; i)$  are some inner architecture, which can be chosen arbitrarily.
Firstly, we compute the gradient of  $f(\cdot ;i)$  with respect to  $b\in \mathbb{R}^c$ . For  $j = 1,\dots ,c$ , we have
$$
\frac {\partial f (w ; i)}{\partial b _ {j}} = \frac {\partial \phi_ {i} (h (w ; i))}{\partial b _ {j}} = \sum_ {k = 1} ^ {c} \frac {\partial h (w ; i) _ {k}}{\partial b _ {j}} \cdot \left. \frac {\partial \phi_ {i} (x)}{\partial x _ {k}} \right| _ {x = h (w; i)} = \left. \frac {\partial \phi_ {i} (x)}{\partial x _ {j}} \right| _ {x = h (w; i)}, i = 1, \dots , n. \tag {18}
$$
The last equality follows since  $\frac{\partial h(w;i)_k}{\partial b_j} = 0$  for every  $k \neq j$  and  $\frac{\partial h(w;i)_k}{\partial b_j} = 1$  for  $k = j$ . In other words, it is the identity matrix.
Let us denote that  $f_{i}^{*} = \min_{w}f(w;i)$  and  $\phi_i^* = \min_x\phi_i(x)$ . We prove the following statement for  $\mu = 1$ :
$$
\left\| \nabla_ {w} f (w; i) \right\| ^ {2} \geq \left\| \nabla_ {x} \phi_ {i} (x) \right| _ {x = h (w; i)} \| ^ {2} \geq 2 \mu [ \phi_ {i} (h (w; i)) - \phi_ {i} ^ {*} ] \geq 2 \mu [ f (w; i) - f _ {i} ^ {*} ],
$$
for every  $w\in \mathbb{R}^d$  , and  $i = 1,\ldots ,n$
We begin with the first inequality:
$$
\begin{array}{l} \| \nabla_ {w} f (w; i) \| ^ {2} = \sum_ {j = 1} ^ {d} \left(\frac {\partial f (w ; i)}{\partial w _ {j}}\right) ^ {2} \geq \sum_ {j = d - c + 1} ^ {d} \left(\frac {\partial f (w ; i)}{\partial w _ {j}}\right) ^ {2} = \sum_ {j = 1} ^ {c} \left(\frac {\partial f (w ; i)}{\partial b _ {j}}\right) ^ {2} \\ \stackrel {(1 8)} {=} \sum_ {j = 1} ^ {c} \left(\frac {\partial \phi_ {i} (x)}{\partial x _ {j}} \Big | _ {x = h (w; i)}\right) ^ {2} = \| \nabla_ {x} \phi_ {i} (x) | _ {x = h (w; i)} \| ^ {2}. \\ \end{array}
$$
Now let us prove the PL condition for each function  $\phi_i(x)$ , i.e., there exists a constant  $\mu > 0$  such that:
$$
\left\| \nabla_ {x} \phi_ {i} (x) \right\| ^ {2} \geq 2 \mu \left[ \phi_ {i} (x) - \phi_ {i} ^ {*} \right] \forall x \in \mathbb {R} ^ {c}, i = 1, \dots , n.
$$
Recall the squared loss that  $\phi_i(x) = \frac{1}{2} \| x - y^{(i)}\|^2$  and  $\nabla_{x}\phi_{i}(x) = x - y^{(i)}$ . We can see that the constant  $\mu = 1$  satisfies the following inequality for every  $x\in \mathbb{R}^c$ ,  $i = 1,\ldots ,n$ :
$$
\| \nabla_ {x} \phi_ {i} (x) \| ^ {2} = \| x - y ^ {(i)} \| ^ {2} = 2 \frac {1}{2} \| x - y ^ {(i)} \| ^ {2} = 2 \mu \phi_ {i} (x) \geq 2 \mu [ \phi_ {i} (x) - \phi_ {i} ^ {*} ],
$$
where the last inequality follows since  $\phi_i^* \geq 0$ .
The PL condition for  $\phi_i$  directly implies the second inequality. The last inequality follows from the facts that  $f(w;i) = \phi_i(h(w;i))$  and  $f_{i}^{*} = \min_{w}f_{i}\geq \min_{x}\phi_{i}(x) = \phi_{i}^{*}$ . Hence, Theorem 1 is proved.
B Preliminary results for SGD Shuffling Algorithm
In this section, we present the preliminary results for Algorithm 1. Firstly, from the choice of learning rate  $\eta_i^{(t)}\coloneqq \frac{\eta_t}{n}$  and the update  $w_{i + 1}^{(t)}\coloneqq w_i^{(t)} - \eta_i^{(t)}\nabla f(w_i^{(t)};\pi^{(t)}(i + 1))$  in Algorithm 1, for  $i\in [n]$ , we have
$$
w _ {i} ^ {(t)} = w _ {i - 1} ^ {(t)} - \frac {\eta_ {t}}{n} \nabla f \left(w _ {i - 1} ^ {(t)}; \pi^ {(t)} (i)\right) = w _ {0} ^ {(t)} - \frac {\eta_ {t}}{n} \sum_ {j = 0} ^ {i - 1} \nabla f \left(w _ {j} ^ {(t)}; \pi^ {(t)} (j + 1)\right). \tag {19}
$$
$$
w _ {0} ^ {(t + 1)} = w _ {n} ^ {(t)} = w _ {0} ^ {(t)} - \frac {\eta_ {t}}{n} \sum_ {j = 0} ^ {n - 1} \nabla f \left(w _ {j} ^ {(t)}; \pi^ {(t)} (j + 1)\right). \tag {20}
$$
Next, we refer to a Lemma in [Nguyen et al., 2021] to bound the updates of shuffling SGD algorithms.
Lemma 3 (Lemma 5 in Nguyen et al. [2021]). Suppose that Assumption 2 holds for (1). Let  $\{w_i^{(t)}\}$  be generated by Algorithm 1 with the learning rate  $\eta_i^{(t)} := \frac{\eta_t}{n} > 0$  for a given positive sequence  $\{\eta_t\}$ . If  $0 < \eta_t \leq \frac{1}{2L}$  for all  $t \geq 1$ , we have
$$
\frac {1}{n} \sum_ {j = 0} ^ {n - 1} \| w _ {j} ^ {(t)} - w _ {*} \| ^ {2} \leq 4 \| w _ {0} ^ {(t)} - w _ {*} \| ^ {2} + 8 \sigma_ {*} ^ {2} \cdot \eta_ {t} ^ {2}, \tag {21}
$$
$$
\frac {1}{n} \sum_ {j = 0} ^ {n - 1} \| w _ {j} ^ {(t)} - w _ {0} ^ {(t)} \| ^ {2} \leq \eta_ {t} ^ {2} \cdot \frac {8 L ^ {2}}{3} \| w _ {0} ^ {(t)} - w _ {*} \| ^ {2} + \frac {1 6 L ^ {2} \sigma_ {*} ^ {2}}{3} \cdot \eta_ {t} ^ {4} + 2 \sigma_ {*} ^ {2} \cdot \eta_ {t} ^ {2}. \tag {22}
$$
Now considering the term  $\| w_n^{(t)} - w_0^{(t)}\|^2$ , we get that
$$
\begin{array}{l} \left\| w _ {n} ^ {(t)} - w _ {0} ^ {(t)} \right\| ^ {2} \stackrel {(2 0)} {\leq} \frac {\eta_ {t} ^ {2}}{n} \left\| \frac {1}{n} \sum_ {j = 0} ^ {n - 1} \nabla f \left(w _ {j} ^ {(t)}; \pi^ {(t)} (j + 1)\right) \right\| ^ {2} \\ = \frac {\eta_ {t} ^ {2}}{n} \left\| \frac {1}{n} \sum_ {j = 0} ^ {n - 1} (\nabla f (w _ {j} ^ {(t)}; \pi^ {(t)} (j + 1)) - \nabla f (w _ {*}; \pi^ {(t)} (j + 1))) \right\| ^ {2} \\ \leq \frac {\eta_ {t} ^ {2}}{n} \frac {1}{n} \sum_ {j = 0} ^ {n - 1} \left\| \nabla f (w _ {j} ^ {(t)}; \pi^ {(t)} (j + 1)) - \nabla f (w _ {*}; \pi^ {(t)} (j + 1)) \right\| ^ {2} \\ \stackrel {(2)} {\leq} \frac {L ^ {2} \eta_ {t} ^ {2}}{n} \frac {1}{n} \sum_ {j = 0} ^ {n - 1} \| w _ {j} ^ {(t)} - w _ {*} \| ^ {2} \\ \stackrel {(2 1)} {\leq} \frac {4 L ^ {2} \eta_ {t} ^ {2}}{n} \| w _ {0} ^ {(t)} - w _ {*} \| ^ {2} + \frac {8 L ^ {2} \eta_ {t} ^ {4}}{n} \sigma_ {*} ^ {2}. \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{n} \sum_ {j = 0} ^ {n} \| w _ {j} ^ {(t)} - w _ {0} ^ {(t)} \| ^ {2} = \frac {1}{n} \sum_ {j = 0} ^ {n - 1} \| w _ {j} ^ {(t)} - w _ {0} ^ {(t)} \| ^ {2} + \frac {1}{n} \| w _ {n} ^ {(t)} - w _ {0} ^ {(t)} \| ^ {2} \\ \leq \eta_ {t} ^ {2} \cdot \frac {8 L ^ {2}}{3} \| w _ {0} ^ {(t)} - w _ {*} \| ^ {2} + \frac {1 6 L ^ {2} \sigma_ {*} ^ {2}}{3} \cdot \eta_ {t} ^ {4} + 2 \sigma_ {*} ^ {2} \cdot \eta_ {t} ^ {2} \\ + \frac {4 L ^ {2} \eta_ {t} ^ {2}}{n} \| w _ {0} ^ {(t)} - w _ {*} \| ^ {2} + \frac {8 L ^ {2} \eta_ {t} ^ {4}}{n} \sigma_ {*} ^ {2}. \tag {23} \\ \end{array}
$$
C Main results: Proofs of Lemma 4, Lemma 2, Theorem 2, and Corollary 1
C.1 Proof of Lemma 4
Lemma 4. Let  $\{w_i^{(t)}\}_{t = 1}^T$  be the sequence generated by Algorithm 1 with  $\eta_i^{(t)} = \frac{\eta_t}{n}$ , with  $0 < \eta_t \leq \frac{n}{2M}$  for  $\eta_t \leq \frac{1}{2L}$ . Then, under Assumptions 1, 2, and 5, we have
$$
\left\| w _ {0} ^ {(t + 1)} - w _ {*} \right\| ^ {2} \leq \left(1 + B _ {1} \eta_ {t} ^ {3}\right) \left\| w _ {0} ^ {(t)} - w _ {*} \right\| ^ {2} - \frac {\eta_ {t}}{2 M} \frac {1}{n} \sum_ {i = 1} ^ {n} \| \nabla f \left(w _ {i - 1} ^ {(t)}; \pi^ {(t)} (i)\right) \| ^ {2} + B _ {2} \eta_ {t} \sigma_ {*} ^ {2}, \tag {24}
$$