SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark A.2. We give an experimental exploration of the  $\mathrm{gen}^{nl}(\mathbf{J}_{\mathbf{T}})$  in Appendix C.3. We discover that if the optimizer doesn't enter the EoS (Edge of Stability) regime [9], we have  $\mathrm{gen}^{nl}(\mathbf{J}_{\mathbf{T}})\approx 0$
Proof. Analyzing of  $\mathrm{gen}^{lin}(\mathbf{J}_{\mathbf{T}})$
Because of  $\epsilon (\mathbf{w}) = C(\mathbf{w})^{\frac{1}{2}}\epsilon^{\prime}$  and  $\mathbb{E}[\epsilon^{\prime}] = 0$  (detail in Equation (3) and Equation (5)d), we can get:
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ \epsilon_ {t} ^ {\mathrm {T}} \left(\nabla F _ {\mu} (\mathbf {J} _ {\mathbf {t}}) - \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}})\right) \right] \\ = \mathbb {E} _ {t - 1} \left[ \left(\epsilon^ {\prime}\right) ^ {\mathrm {T}} \left(C \left(\mathbf {J} _ {\mathbf {t}}\right) ^ {\frac {1}{2}}\right) ^ {\mathrm {T}} \left(\nabla F _ {\mu} \left(\mathbf {J} _ {\mathbf {t}}\right) - \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right)\right) \right] \tag {20} \\ = \mathbb {E} _ {t - 1} [ \epsilon^ {\prime} ] ^ {\mathrm {T}} \mathbb {E} _ {t - 1} \left[ \left(C \left(\mathbf {J} _ {\mathbf {t}}\right) ^ {\frac {1}{2}}\right) ^ {\mathrm {T}} \left(\nabla F _ {\mu} (\mathbf {J} _ {\mathbf {t}}) - \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}})\right) \right] \\ = 0. \\ \end{array}
$$
Combining with Formula (3), we have
$$
\mathbb {E} \left[ \operatorname {g e n} ^ {l i n} \left(\mathbf {J} _ {\mathbf {T}}\right) \right] = \mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} \eta_ {t} \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) ^ {\mathrm {T}} \left(\nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) - \nabla F _ {\mu} \left(\mathbf {J} _ {\mathbf {t}}\right)\right) \right]. \tag {21}
$$
Analyzing of  $\mathrm{gen}^{nl}(\mathbf{J}_{\mathbf{T}})$
Here, we denote  $M \triangleq \max_{t} \| \nabla F_S(\mathbf{J}_{\mathbf{t}}) \|$ . According to the definition of  $\mathrm{gen}^{nl}(\mathbf{J}_{\mathbf{T}})$ ,
$$
\begin{array}{l} | \mathrm {g e n} ^ {n l} (\mathbf {J _ {T}}) | \leq | F _ {\mu} (\mathbf {J _ {T}}) - F _ {\mu} ^ {l i n} (\mathbf {J _ {T}}) | + | F _ {S} (\mathbf {J _ {T}}) - F _ {S} ^ {l i n} (\mathbf {J _ {T}}) | \\ = \left| \sum_ {t = 1} ^ {T} \mathcal {O} \left(\left\| \mathbf {J} _ {\mathbf {t} + \mathbf {1}} - \mathbf {J} _ {\mathbf {t}} \right\| ^ {2}\right) \right| + \left| \sum_ {t = 1} ^ {T} \mathcal {O} \left(\left\| \mathbf {J} _ {\mathbf {t} + \mathbf {1}} - \mathbf {J} _ {\mathbf {t}} \right\| ^ {2}\right) \right| \\ = \sum_ {t = 1} ^ {T} \mathcal {O} \left(\eta_ {t} ^ {2} \| \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) \| ^ {2}\right) \tag {22} \\ = \mathcal {O} (T \eta_ {m} ^ {2} M ^ {2}) \\ = \mathcal {O} \left(\frac {1}{\eta_ {m}} \eta_ {m} ^ {2} M ^ {2}\right) \\ = \mathcal {O} (\eta_ {m}) \\ \end{array}
$$
Because all the elements of training set  $S$  are sampled from distribution  $\mu$ , we have  $\lim_{n\to \infty}\nabla F_S(\mathbf{w}) = F_\mu (\mathbf{w})$ , Therefore:
$$
\lim  _ {n \rightarrow \infty} (i i) _ {t} ^ {l i n} = \lim  _ {n \rightarrow \infty} \left(\mathbf {J} _ {\mathbf {t}} - \mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right) ^ {\mathrm {T}} \left(\nabla F _ {\mu} \left(\mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right) - \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right)\right) = 0. \tag {23}
$$
What's more, we also have:
$$
\lim  _ {n \rightarrow \infty} \left[ F _ {\mu} \left(\mathbf {J} _ {\mathbf {T}}\right) - F _ {S} \left(\mathbf {J} _ {\mathbf {T}}\right)\right] = 0. \tag {24}
$$
Because  $F_{\mu}(\mathbf{J}_{\mathbf{T}}) - F_{S}(\mathbf{J}_{\mathbf{T}}) = \sum_{t=1}^{T}(ii)_{t}^{lin} + \sum_{t=1}^{T}(ii)_{t}^{nl}$ , we have:
$$
\lim  _ {n \rightarrow \infty} | \operatorname {g e n} ^ {n l} (\mathbf {J} _ {\mathbf {T}}) | = \lim  _ {n \rightarrow \infty} \left| \sum_ {t = 1} ^ {T} (i i) _ {t} ^ {n l} \right| = \lim  _ {n \rightarrow \infty} \left| F _ {\mu} (\mathbf {J} _ {\mathbf {T}}) - F _ {S} (\mathbf {J} _ {\mathbf {T}}) - \sum_ {t = 1} ^ {T} (i i) _ {t} ^ {l i n} \right| = | 0 - 0 | = 0 \tag {25}
$$
According to the Equation (17), we analyze the generalization error of  $\mathcal{F}_{\mathbf{J}|S} \triangleq \{\sum_{t=0}^{T-1} \mathbf{w}_t^{\mathrm{T}} \nabla f(\mathbf{J}_t) \mid \mathbf{w}_t = \delta_t \frac{\nabla f(\mathbf{J}_t)}{\|\nabla f(\mathbf{J}_t)\|}\}$  as a proxy for analyzing generalization error of the function trained using SGD or GD algorithm. The value of  $\mathrm{gen}^{lin}(\mathbf{J}_{\mathbf{T}})$  is equal to the generalization error of  $\mathcal{F}_{\mathbf{J}|S}$ . To analyze  $\mathcal{F}_{\mathbf{J}|S}$ , we construct an additive linear space as  $\mathcal{L}_{\mathbf{J}|S} \triangleq \{\sum_{t=0}^{T-1} \mathbf{w}_t^{\mathrm{T}} \nabla f(\mathbf{J}_t) \mid \| \mathbf{w}_t \| \leq \delta_t\}$ , where  $\delta_t \triangleq \| \eta_t \nabla F_S(\mathbf{J}_t)\|$ . Here, we use  $\mathbf{J}|S$  to emphasize that  $\mathbf{J}$  depends on  $S$ .
Under Assumption 3.4 (that is introduced in the main paper), we can have the following lemma.
Lemma A.3. Under Assumption 3.4, given  $S \sim \mu^n$ , let  $\mathbf{J} = \mathcal{A}(S)$ , we have:
$$
\mathbb {E} \left[ \operatorname {g e n} ^ {l i n} \left(\mathbf {J} _ {\mathbf {T}}\right) \right] \leq 2 \gamma^ {\prime} \mathbb {V} _ {m} \mathbb {E} R _ {S} \left(\mathcal {L} _ {\mathbf {J} | S}\right), \tag {26}
$$
where  $\mathbb{V}(\mathbf{w}) = \frac{\|\nabla F_S(\mathbf{w})\|}{\mathbb{E}_{U\subset S}\|\frac{|U|}{n}\nabla F_U(\mathbf{w}) - \frac{|S| - |U|}{n}\nabla F_{S / U}(\mathbf{w})\|},\quad \mathbb{V}_m = \max_t\mathbb{V}(\mathbf{J}_t)$  and  $\gamma^{\prime} =$ $\max \{1,\max_{U\subset S;t}\frac{|U||\nabla F_U(\mathbf{J}_t)||}{n\|\nabla F_S(\mathbf{J}_t)\|}\} \gamma .$
Proof. For a function  $h$ , we define that  $h_{\mu} \triangleq \mathbb{E}_{z \sim \mu}[g(z)]$  and  $h_S = \frac{1}{n} \sum_{z_i \in S} h(z_i)$ . Given a function space, the maximum generalization error of the space can be defined as:  $\Phi(S, H) \triangleq \sup_{h \in H} (h_{\mu} - h_S)$
$$
\begin{array}{l} \Phi (S, H | S) = \sup  _ {h \in H | S} \left(h _ {\mu} - h _ {S}\right) \\ = \sup  _ {h \in H | S} \left(\mathbb {E} _ {S ^ {\prime}} h _ {S ^ {\prime}} - h _ {S}\right) \\ \leq \mathbb{E}_{S^{\prime}}\sup_{h\in H|S}\left(h_{S^{\prime}} - h_{S}\right) \\ = \mathbb {E} _ {S ^ {\prime}, \sigma} \sup  _ {h \in H | S} \left(\frac {1}{n} \sum_ {i} ^ {n} \sigma_ {i} \left(h \left(z _ {i} ^ {\sigma_ {i}}\right) - h \left(z _ {i} ^ {- \sigma_ {i}}\right)\right)\right) \tag {27} \\ \leq \mathbb {E} _ {S ^ {\prime}, \sigma} \sup _ {h \in H | S} (\frac {1}{n} \sum_ {i} ^ {n} \sigma_ {i} (h (z _ {i} ^ {\sigma_ {i}}))) + \mathbb {E} _ {S ^ {\prime}, \sigma} \sup _ {h \in H | S} (\frac {1}{n} \sum_ {i} ^ {n} \sigma_ {i} (h (z _ {i} ^ {\sigma_ {i}}))) \\ = 2 \mathbb {E} _ {S ^ {\prime}, \sigma} \sup  _ {h \in H | S} \left(\frac {1}{n} \sum_ {i} ^ {n} \sigma_ {i} (h (z _ {i} ^ {\sigma_ {i}}))\right), \\ \end{array}
$$
where  $S'$  is another i.i.d sample set drawn from  $\mu^n$  and  $\sigma$  denotes the Rademacher variable. The  $\sigma_i$  in  $z_i^{\sigma_i}$  denotes  $z_i^{\sigma_i}$  that belongs to  $S$  or  $S'$ . If  $\sigma_i = -1$ $z_i^{\sigma_i} \in S$ , otherwise,  $z_i^{\sigma_i} \in S'$ .
$$
\begin{array}{l} R _ {S} (\mathcal {L} _ {\mathbf {J} | S}) \triangleq \mathbb {E} _ {\sigma} \sup _ {h \in \mathcal {L} _ {\mathbf {J} | S}} (\frac {1}{n} \sum_ {i} ^ {n} \sigma_ {i} h (z _ {i})) \\ = \mathbb {E} _ {\sigma} \sup  _ {h \in \mathcal {L} _ {\mathbf {J} | S}} \left(\frac {1}{n} \left(\sum_ {z \in S _ {+}} h (z) - \sum_ {z \in S _ {-}} h (z)\right)\right) \tag {28} \\ = \mathbb {E} _ {\sigma} \left(\frac {1}{n} \sum_ {t = 0} ^ {\mathrm {T}} \delta_ {t} \| g _ {S _ {+}} (\mathbf {J} _ {\mathbf {t}}) - g _ {S _ {-}} (\mathbf {J} _ {\mathbf {t}}) \|\right), \\ \end{array}
$$
where  $S_{+}\triangleq \{z_{i}|\sigma_{i} = +1\}$  and  $S_{-}\triangleq \{z_{i}|\sigma_{i} = -1\}$ , and  $g_{S}(\mathbf{w})\triangleq |S|\nabla F_{S}(\mathbf{w})$
$$
\begin{array}{l} \mathbb {E} _ {S ^ {\prime}, \sigma} \sup  _ {h \in \mathcal {F} _ {\mathbf {J} | S}} (\frac {1}{n} \sum_ {i} ^ {n} \sigma_ {i} (h (z _ {i} ^ {\sigma_ {i}}))) = \mathbb {E} _ {S ^ {\prime}, \sigma} \sup  _ {h \in \mathcal {F} _ {\mathbf {J} | S}} (\frac {1}{n} (\sum_ {z \in S _ {+} ^ {\prime}} h (z) - \sum_ {z \in S _ {-}} h (z))) \\ = \mathbb {E} _ {S ^ {\prime}, \sigma} \left(\frac {1}{n} \sum_ {t = 0} ^ {T - 1} \delta_ {t} \frac {g _ {S} (\mathbf {J} _ {\mathbf {t}})}{\| g _ {S} (\mathbf {J} _ {\mathbf {t}}) \|} \left(g _ {S _ {+} ^ {\prime}} \left(\mathbf {J} _ {\mathbf {t}}\right) - g _ {S _ {-}} \left(\mathbf {J} _ {\mathbf {t}}\right)\right)\right) \tag {29} \\ = \mathbb {E} _ {S ^ {\prime}, \sigma} \left(\frac {1}{n} \sum_ {t = 0} ^ {T - 1} \delta_ {t} \frac {g _ {S} (\mathbf {J} _ {\mathbf {t}})}{\| g _ {S} (\mathbf {J} _ {\mathbf {t}}) \|} \left(| S _ {+} | \nabla F _ {\mu} (\mathbf {J} _ {\mathbf {t}}) - g _ {S _ {-}} (\mathbf {J} _ {\mathbf {t}})\right)\right) \\ \end{array}
$$
where  $S_{+}^{\prime}$  is a subset of  $S^{\prime}$  with  $|S_{+}^{\prime}| = |S_{+}|$ . Defining  $k \triangleq \gamma' \mathbb{V}_m$ , we have:
$$
\begin{array}{l} k \mathbb {E} _ {\sigma} \sup  _ {h \in \mathcal {L} _ {\mathbf {J} | S}} (\frac {1}{n} \sum_ {i} ^ {n} \sigma_ {i} h (z _ {i})) - \mathbb {E} _ {S ^ {\prime}, \sigma} \sup  _ {h \in \mathcal {F} _ {\mathbf {J} | S}} (\frac {1}{n} \sum_ {i} ^ {n} \sigma_ {i} (h (z _ {i}))) \\ = k \mathbb {E} _ {\sigma} \left(\frac {1}{n} \sum_ {t = 0} ^ {T} \delta_ {t} \| g _ {S _ {+}} (\mathbf {J} _ {\mathbf {t}}) - g _ {S _ {-}} (\mathbf {J} _ {\mathbf {t}}) \|\right) - \mathbb {E} _ {S ^ {\prime}, \sigma} \left(\frac {1}{n} \sum_ {t = 0} ^ {T} \delta_ {t} \frac {g _ {S} (\mathbf {J} _ {\mathbf {t}})}{\| g _ {S} (\mathbf {J} _ {\mathbf {t}}) \|} \left(g _ {S _ {+} ^ {\prime}} (\mathbf {J} _ {\mathbf {t}}) - g _ {S _ {-}} (\mathbf {J} _ {\mathbf {t}})\right)\right) \\ = k \mathbb {E} _ {\sigma} \left(\frac {1}{n} \sum_ {t = 0} ^ {T} \delta_ {t} \| g _ {S _ {+}} (\mathbf {J} _ {\mathbf {t}}) - g _ {S _ {-}} (\mathbf {J} _ {\mathbf {t}}) \| - \frac {1}{n} \sum_ {t = 0} ^ {T} \delta_ {t} \frac {g _ {S} (\mathbf {J} _ {\mathbf {t}})}{\| g _ {S} (\mathbf {J} _ {\mathbf {t}}) \|} \left(| S _ {+} | \nabla F _ {\mu} (\mathbf {J} _ {\mathbf {t}}) - g _ {S _ {-}} (\mathbf {J} _ {\mathbf {t}})\right)\right) \tag {30} \\ \geq k \mathbb {E} _ {\sigma} \left(\frac {1}{n} \sum_ {t = 0} ^ {T} \delta_ {t} \| g _ {S _ {+}} (\mathbf {J} _ {\mathbf {t}}) - g _ {S _ {-}} (\mathbf {J} _ {\mathbf {t}}) \| - \frac {1}{n} \sum_ {t = 0} ^ {T} \delta_ {t} \| | S _ {+} | \nabla F _ {\mu} (\mathbf {J} _ {\mathbf {t}}) - g _ {S _ {-}} (\mathbf {J} _ {\mathbf {t}}) |\right) \\ \geq k \frac {1}{n} \sum_ {t = 0} ^ {T} \delta_ {t} \mathbb {E} _ {\sigma} \left(\left\| g _ {S _ {+}} \left(\mathbf {J} _ {\mathbf {t}}\right) - g _ {S _ {-}} \left(\mathbf {J} _ {\mathbf {t}}\right) \right\|\right) - \sum_ {t = 0} ^ {T} \delta_ {t} \gamma^ {\prime} \| \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) \| \\ \geq 0 \\ \end{array}
$$
Therefore, combining Equation (27) and (30), we have  $\mathbb{E}[\mathrm{gen}^{lin}(\mathbf{J}_{\mathbf{T}})] \leq 2\gamma' \mathbb{V}_m \mathbb{E}R_S(\mathcal{L}_{\mathbf{J}|S})$ .
Lemma A.4. Given  $\mathbf{J} = \mathcal{A}(S)$ , the formula  $R_{S}(\mathcal{L}_{\mathbf{J}|S})$  can be upper bounded with:
$$
R _ {S} \left(\mathcal {L} _ {\mathbf {J} | S}\right) \leq - \mathbb {E} \int_ {t} \frac {d F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right)}{\sqrt {n}} \sqrt {1 + \frac {\operatorname {T r} \left(\Sigma (\mathbf {w})\right)}{\| \nabla F _ {S} (\mathbf {w}) \| _ {2} ^ {2}}}. \tag {31}
$$
Proof. Let us start with the calculation of  $R_{S}(\mathbf{w}^{\mathrm{T}}\nabla f(\mathbf{J}_{\mathbf{t}}))$ :
$$
\begin{array}{l} R _ {S} (\left\{\mathbf {w} ^ {\mathrm {T}} \nabla f \left(\mathbf {J} _ {\mathbf {t}}\right) \mid \| \mathbf {w} \| \leq \delta \right\}) = \frac {1}{n} \mathbb {E} _ {\sigma} \left(\sup  _ {\| \mathbf {w} \| \leq \delta} \mathbf {w} ^ {\mathrm {T}} \sum_ {i = 1} ^ {n} \sigma_ {i} \nabla f \left(\mathbf {J} _ {\mathbf {t}}, z _ {i}\right)\right) \\ = \frac {\delta}{n} \mathbb {E} _ {\sigma} \left(\sqrt {\left\| \sum_ {i = 1} ^ {n} \sigma_ {i} \nabla f (\mathbf {J} _ {\mathbf {t}} , z _ {i}) \right\| ^ {2}}\right) \\ \leq \frac {\delta}{n} \left(\sqrt {\mathbb {E} _ {\sigma} \left\| \sum_ {i = 1} ^ {n} \sigma_ {i} \nabla f \left(\mathbf {J} _ {\mathbf {t}} , z _ {i}\right) \right\| ^ {2}}\right) \tag {32} \\ \triangleq \frac {\delta}{n} \left(\sqrt {\mathbb {E} _ {\sigma} \sum_ {i = 1} ^ {n} \left\| \sigma_ {i} \nabla f (\mathbf {J} _ {\mathbf {t}} , z _ {i}) \right\| ^ {2}}\right) \\ = \frac {\delta}{n} \sqrt {\sum_ {i = 1} ^ {n} \left\| \nabla f (\mathbf {J _ {t}} , z _ {i}) \right\| ^ {2}}, \\ \end{array}
$$
where  $\triangleleft$  represents using the relation that for  $i,j$  satisfying  $i\neq j$ , we have  $\mathbb{E}\sigma_i\sigma_j = 0$
Because  $w_{i}$  is independent of  $w_{j}$  if  $i\neq j$  , we have:
$$
\begin{array}{l} R _ {S} \left(\mathcal {L} _ {\mathbf {J} | S}\right) = R _ {S} \left(\left\{f \left(\mathbf {J} _ {\mathbf {0}}\right) + \sum_ {t = 0} ^ {T - 1} \mathbf {w} _ {\mathbf {t}} ^ {\mathrm {T}} \nabla f \left(\mathbf {J} _ {\mathbf {t}}\right) \mid \| \mathbf {w} _ {\mathbf {t}} \| \leq \delta_ {t} \right\}\right) \\ = \sum_ {t = 0} ^ {T - 1} R _ {S} \left( \right.\left\{ \right.\mathbf {w} _ {\mathbf {t}} ^ {\mathrm {T}} \nabla f \left(\mathbf {J} _ {\mathbf {t}}\right)\left. \right\|\left. \mathbf {w} _ {\mathbf {t}} \right\| \leq \delta_ {t} \left. \right\}\left. \right) \tag {33} \\ \leq \sum_ {t = 0} ^ {T - 1} \frac {\delta_ {t}}{n} \sqrt {\sum_ {i = 1} ^ {n} \| \nabla f (\mathbf {J} _ {\mathbf {t}} , z _ {i}) \| ^ {2}}. \\ \end{array}
$$
The covariance of gradient noise can be calculated as:
$$
\begin{array}{l} \operatorname {T r} [ \Sigma (\mathbf {w}) ] = \operatorname {T r} [ \frac {1}{n} \sum_ {i = 1} ^ {n} \nabla f (\mathbf {w}, z _ {i}) \nabla f (\mathbf {w}, z _ {i}) ^ {\mathrm {T}} - \nabla F _ {S} (\mathbf {w}) \nabla F _ {S} (\mathbf {w}) ^ {\mathrm {T}} ] \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \operatorname {T r} \left[ \nabla f (\mathbf {w}, z _ {i}) \nabla f (\mathbf {w}, z _ {i}) ^ {\mathrm {T}} \right] - \operatorname {T r} \left[ \nabla F _ {S} (\mathbf {w}) \nabla F _ {S} (\mathbf {w}) ^ {\mathrm {T}} \right] \tag {34} \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \| \nabla f (\mathbf {w}, z _ {i}) ^ {2} \| - \| \nabla F _ {S} (\mathbf {w}) \| ^ {2} \\ \end{array}
$$
Taking Equation (33) and  $\delta_t \triangleq \|\eta_t \nabla F_S(\mathbf{J}_t)\|$  into Equation (34), we have:
$$
\begin{array}{l} R _ {S} \left(\mathcal {L} _ {\mathbf {J} | S}\right) \leq \sum_ {t = 0} ^ {T - 1} \frac {\delta_ {t}}{n} \sqrt {\sum_ {i = 1} ^ {n} \| \nabla f \left(\mathbf {J} _ {\mathbf {t}} , z _ {i}\right) \| ^ {2}} \\ = \sum_ {t = 0} ^ {T - 1} \frac {\delta_ {t}}{\sqrt {n}} \sqrt {\operatorname {T r} [ \Sigma (\mathbf {J} _ {\mathbf {t}}) ] + \| \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) \| ^ {2}} \tag {35} \\ = \sum_ {t = 0} ^ {T - 1} \frac {\eta_ {t} \| \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) \|}{\sqrt {n}} \sqrt {\operatorname {T r} [ \Sigma (\mathbf {J} _ {\mathbf {t}}) ] + \| \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) \| ^ {2}} \\ \end{array}
$$
When  $\eta_{t}$  is small,  $\delta_t\approx -\mathbb{E}_\epsilon \frac{(\mathbf{J}_{\mathbf{t} + 1} - \mathbf{J}_{\mathbf{t}})^{\mathrm{T}}\nabla F_S(\mathbf{J}_{\mathbf{t}})}{\|\nabla F_S(\mathbf{J}_{\mathbf{t}})\|}\approx -\mathbb{E}_\epsilon \frac{F_S(\mathbf{J}_{\mathbf{t} + 1}) - F_S(\mathbf{J}_{\mathbf{t}})}{\|\nabla F_S(\mathbf{J}_{\mathbf{t}})\|}$  holds, therefore we have:
$$
\begin{array}{l} \mathbb {E} R _ {S} \left(\mathcal {L} _ {\mathbf {J} | S}\right) \leq \mathbb {E} \sum_ {t = 0} ^ {T - 1} \frac {\delta_ {t}}{n} \sqrt {\sum_ {i = 1} ^ {n} \left\| \nabla f \left(\mathbf {J} _ {\mathbf {t}} , z _ {i}\right) \right\| _ {2} ^ {2}} \\ \approx - \mathbb {E} \sum_ {t = 0} ^ {T - 1} \frac {F _ {S} (\mathbf {J} _ {\mathbf {t} + \mathbf {1}}) - F _ {S} (\mathbf {J} _ {\mathbf {t}})}{\sqrt {n}} \sqrt {1 + \frac {\operatorname {T r} (\Sigma (\mathbf {w}))}{\| \nabla F _ {S} (\mathbf {w}) \| _ {2} ^ {2}}} \tag {36} \\ \approx - \mathbb {E} \int_ {t} \frac {d F _ {S} (\mathbf {J _ {t}})}{\sqrt {n}} \sqrt {1 + \frac {\mathrm {T r} (\Sigma (\mathbf {w}))}{\| \nabla F _ {S} (\mathbf {w}) \| _ {2} ^ {2}}} \\ \end{array}
$$
Theorem A.5. Under Assumption 3.4, given  $S \sim \mu^n$ , let  $\mathbf{J} = \mathcal{A}(S)$ , where  $\mathcal{A}$  denotes the SGD or GD algorithm training with  $T$  steps, we have:
$$
\mathbb {E} \left[ F _ {\mu} \left(\mathbf {J} _ {\mathbf {T}}\right) - F _ {S} \left(\mathbf {J} _ {\mathbf {T}}\right) \right] \leq - 2 \gamma^ {\prime} \mathbb {V} _ {m} \mathbb {E} \int_ {t} \frac {d F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right)}{\sqrt {n}} \sqrt {1 + \frac {\operatorname {T r} \left(\Sigma \left(\mathbf {J} _ {\mathbf {t}}\right)\right)}{\| \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) \| _ {2} ^ {2}}} + \mathcal {O} (\eta_ {m}), \tag {37}
$$
where  $\mathbb{V}(\mathbf{w}) = \frac{\|\nabla F_S(\mathbf{w})\|}{\mathbb{E}_{U\subset S}\|\frac{|U|}{n}\nabla F_U(\mathbf{w}) - \frac{|S| - |U|}{n}\nabla F_{S / U}(\mathbf{w})\|},\mathbb{V}_m = \max_t\mathbb{V}(\mathbf{J}_t)$  and  $\gamma^{\prime} =$
max{1, max  $\max_{U\subset S;t}\frac{|U||\nabla F_U(\mathbf{J}_t)||}{n\|\nabla F_S(\mathbf{J}_t)\|}\} \gamma .$
Proof. We rewrite Equation 2 of the update of SGD with batchsize  $b$  here:
$$
\mathbf {J} _ {\mathbf {t}} = \mathbf {J} _ {\mathbf {t} - \mathbf {1}} - \eta_ {t} \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right) + \eta_ {t} \epsilon_ {t} \tag {38}
$$
where we simplify the  $\epsilon (\mathbf{w}_{\mathbf{t}})$  as  $\epsilon_{t}$ , then we can expand the function at  $f(\mathbf{J_T})$  as:
$$
\begin{array}{l} f ^ {l i n} \left(\mathbf {J} _ {\mathbf {T}}\right) \triangleq f \left(\mathbf {J} _ {\mathbf {0}}\right) + \sum_ {t = 0} ^ {T - 1} \left(\eta_ {t} \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) + \epsilon\right) ^ {\mathrm {T}} \nabla f \left(\mathbf {J} _ {\mathbf {t}}\right) \tag {39} \\ = f \left(\mathbf {J} _ {\mathbf {0}}\right) + \sum_ {t = 0} ^ {K - 1} \eta_ {t} \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) ^ {\mathrm {T}} \nabla f \left(\mathbf {J} _ {\mathbf {t}}\right) + \sum_ {t = 0} ^ {T - 1} \epsilon_ {t} ^ {\mathrm {T}} \nabla f \left(\mathbf {J} _ {\mathbf {t}}\right) \\ \end{array}
$$
Note that when the learning rate is small, we have  $f(\mathbf{J}_{\mathbf{T}}) \approx f^{lin}(\mathbf{J}_{\mathbf{T}})$ .
The difference between the distributional value and the empirical value of the linear function can be calculated as:
$$
\begin{array}{l} \mathbb {E} \left[ F _ {\mu} (\mathbf {J _ {0}}) + \sum_ {t = 0} ^ {T - 1} \left(\eta_ {t} \nabla F _ {S} (\mathbf {J _ {t}}) + \epsilon\right) ^ {\mathrm {T}} \nabla F _ {\mu} (\mathbf {J _ {t}}) \right] - \mathbb {E} \left[ F _ {S} (\mathbf {J _ {0}}) + \sum_ {t = 0} ^ {T - 1} \left(\eta_ {t} \nabla F _ {S} (\mathbf {J _ {t}}) + \epsilon\right) ^ {\mathrm {T}} \nabla F _ {S} (\mathbf {J _ {t}}) \right] \\ = \mathbb {E} \left[ F _ {\mu} (\mathbf {J _ {0}}) - F _ {S} (\mathbf {J _ {0}}) + \sum_ {t = 0} ^ {T - 1} \left(\eta_ {t} \nabla F _ {S} (\mathbf {J _ {t}}) + \epsilon\right) ^ {\mathrm {T}} \nabla F _ {\mu} (\mathbf {J _ {t}}) - \sum_ {t = 0} ^ {T - 1} \left(\eta_ {t} \nabla F _ {S} (\mathbf {J _ {t}}) + \epsilon\right) ^ {\mathrm {T}} \nabla F _ {S} (\mathbf {J _ {t}}) \right] \\ = \mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} \eta_ {t} \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) ^ {\mathrm {T}} \left(\nabla F _ {\mu} (\mathbf {J} _ {\mathbf {t}}) - \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}})\right) + \sum_ {t = 0} ^ {T - 1} \epsilon_ {t} ^ {\mathrm {T}} \left(\nabla F _ {\mu} (\mathbf {J} _ {\mathbf {t}}) - \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}})\right) \right] \\ \triangleq \mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} \eta_ {t} \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) ^ {\mathrm {T}} \left(\nabla F _ {\mu} \left(\mathbf {J} _ {\mathbf {t}}\right) - \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right)\right) \right] \\ \leq \Phi (S, \mathcal {F} _ {\mathbf {J} | S}), \tag {40} \\ \end{array}
$$
where  $\triangleleft$  using the equation that  $\mathbb{E}[\epsilon_t^{\mathrm{T}}(\nabla F_\mu (\mathbf{J}_t) - \nabla F_S(\mathbf{J}_t))] = 0$ , according to Equation 20.
Because of  $\mathbb{E}[F_{\mu}(\mathbf{J}_{\mathbf{T}}) - F_S(\mathbf{J}_{\mathbf{T}})] = \mathbb{E}[F_{\mu}^{lin}(\mathbf{J}_{\mathbf{T}}) + \mathcal{O}(\eta_m) - F_S^{lin}(\mathbf{J}_{\mathbf{T}}) - \mathcal{O}(\eta_m)] = \mathbb{E}[F_{\mu}^{lin}(\mathbf{J}_{\mathbf{T}}) - F_S^{lin}(\mathbf{J}_{\mathbf{T}})] + \mathcal{O}(\eta_m)$  (from Proposition A.1), by applying Lemma A.3 and Lemma A.4, the theorem is proved.
Corollary A.6. If function  $f(\cdot)$  is  $\beta$ -smooth, under Assumption 3.4 given  $S \sim \mu^n$ , let  $\mathbf{J} = \mathcal{A}(S)$ ,  $\eta_t = \frac{c}{\beta(t + 1)}$ ,  $M_2^2 = \max_t \mathbb{E}_{t - 1}(\| \nabla F_S(\mathbf{J}_t) + \epsilon (\mathbf{J}_t)\|^2)$  and  $M_4^4 = \max_t \mathbb{E}_{t - 1}(\| \nabla F_S(\mathbf{J}_t) + \epsilon (\mathbf{J}_t)\|^4)$ , where  $\mathcal{A}$  denoted the SGD or GD algorithm training with  $T$  steps, we have:
$$
\begin{array}{l} \mathbb {E} [ F _ {\mu} (\mathbf {J _ {T}}) - F _ {S} (\mathbf {J _ {T}}) ] \leq - 2 \gamma^ {\prime} \mathbb {V} _ {m} \mathbb {E} \int_ {t} \frac {d F _ {S} (\mathbf {J _ {t}})}{\sqrt {n}} \sqrt {1 + \frac {\mathrm {T r} (\Sigma (\mathbf {J _ {t}}))}{\| \nabla F _ {S} (\mathbf {J _ {t}}) \| _ {2} ^ {2}}} \\ + 2 c ^ {2} \gamma^ {\prime} \mathbb {V} _ {m} M _ {4} ^ {2} \sqrt {\mathbb {E} \int_ {t} \frac {d t}{n \beta^ {2} (t + 1) ^ {4}} \left(1 + \frac {\operatorname {T r} (\Sigma (\mathbf {J} _ {\mathbf {t}}))}{\| \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) \| _ {2} ^ {2}}\right)} \tag {41} \\ + 2 c ^ {2} \frac {M _ {2} ^ {2}}{\beta}. \\ \end{array}
$$
$$
\text {w h e r e} \quad \mathbb {V} (\mathbf {w}) = \frac {\| \nabla F _ {S} (\mathbf {w}) \|}{\mathbb {E} _ {U \subset S} \| \frac {| U |}{n} \nabla F _ {U} (\mathbf {w}) - \frac {| S | - | U |}{n} \nabla F _ {S / U} (\mathbf {w}) \|}, \quad \mathbb {V} _ {m} = \max  _ {t} \mathbb {V} (\mathbf {J} _ {\mathbf {t}}) \text {a n d} \gamma^ {\prime} =
$$
$$
\max \{1, \max  _ {U \subset S; t} \frac {| U | \| \nabla F _ {U} (\mathbf {J} _ {\mathbf {t}}) \|}{n \| \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) \|} \} \gamma .
$$
Proof. If  $f(\cdot)$  is  $\beta$ -smooth, we have:
$$
f \left(\mathbf {J} _ {\mathbf {t} + \mathbf {1}}\right) - f \left(\mathbf {J} _ {\mathbf {t}}\right) \leq \left(\mathbf {J} _ {\mathbf {t} + \mathbf {1}} - \mathbf {J} _ {\mathbf {t}}\right) ^ {\mathrm {T}} \nabla f \left(\mathbf {J} _ {\mathbf {t}}\right) + \frac {1}{2} \beta \| \mathbf {J} _ {\mathbf {t} + \mathbf {1}} - \mathbf {J} _ {\mathbf {t}} \| ^ {2} \tag {42}
$$
$$
f \left(\mathbf {J} _ {\mathbf {t} + 1}\right) - f \left(\mathbf {J} _ {\mathbf {t}}\right) \geq \left(\mathbf {J} _ {\mathbf {t} + 1} - \mathbf {J} _ {\mathbf {t}}\right) ^ {\mathrm {T}} \nabla f \left(\mathbf {J} _ {\mathbf {t}}\right) - \frac {1}{2} \beta \left\| \mathbf {J} _ {\mathbf {t} + 1} - \mathbf {J} _ {\mathbf {t}} \right\| ^ {2}. \tag {43}
$$
Combining the two equations, we obtain:
$$
\begin{array}{l} | R _ {\mu} (\mathbf {J _ {T}}) - R _ {S} (\mathbf {J _ {T}}) | \leq | F _ {\mu} (\mathbf {J _ {T}}) - F _ {\mu} ^ {l i n} (\mathbf {J _ {T}}) | + | F _ {S} (\mathbf {J _ {T}}) - F _ {S} ^ {l i n} (\mathbf {J _ {T}}) | \\ \leq \frac {\beta}{2} \sum_ {t = 0} ^ {T - 1} \| \mathbf {J} _ {\mathbf {t} + \mathbf {1}} - \mathbf {J} _ {\mathbf {t}} \| ^ {2} + \frac {\beta}{2} \sum_ {t = 0} ^ {T - 1} \| \mathbf {J} _ {\mathbf {t} + \mathbf {1}} - \mathbf {J} _ {\mathbf {t}} \| ^ {2} \tag {44} \\ = \beta \sum_ {t = 0} ^ {T - 1} \left\| \mathbf {J} _ {\mathbf {t} + \mathbf {1}} - \mathbf {J} _ {\mathbf {t}} \right\| ^ {2}. \\ \end{array}
$$
The generalization error can be divided into three parts: