SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparison of the generalization bounds with stability based method for SGD learning algorithms. T.R.T is an abbreviation for the term related to trajectory. T.R.T is defined as the term that 1) varies based on different learning trajectories, and 2) don't rely on the extra information of data distribution  $\mu$  except from training data  $S$ . We can infer that the proposed bound incorporates a greater amount of information pertaining to the trajectory. Other related works are discussed in Appendix D.
Method	β-Smooth	L-Lipschitz	Convex	Small LR	Other Conditions	Generalization Bound	T.R.T
Hardt et al. [11]	✓	✓	✓	✓	2Lctn∑t=1Tηt	∑t=1Tηt
Hardt et al. [11]	✓	✓	✓	f∈[0,1],ηt<ct	O(1/Lπct+1T3/2c+1)	T3c+1
Zhang et al. [43]	✓	✓	✓	T>nt,ηt=c/βt	16L2Tc/n1+c	Tc
Zhou et al. [44]	✓	✓	✓	Ez∈S||∇f(w,z)−∇FS(w)||2≤B2	O(√1nL√2βFμ(J0)+1/2EB2logT)	√logT
Bassily et al. [5]	✓	✓	Projected SGD	2L2√∑t=1T-1ηt+4L2n∑t=1T-1ηt	∑t=1T-1ηt
Lei and Ying [16]	✓	✓	Projected SGD	O((1+T/n2)∑t=1Tηt)	T∑t=1Tηt2
Ours (Theorem 3.6)	✓	||∇Fμ(w)||≤γ||∇FS(w)||	Theorem 3.6	∫tdFs(Jt)√1+Tr(Σ(Jt))||∇F(Sj)(Jt)||
Step 3 Finally, we compute the upper bound of  $R_{S}(\mathcal{L}_{\mathbf{J}|S})$ , which follows same techniques used in Radermacher Complexity theory. By combining this with Proposition A.1, we establish the theorem.
Technical Novel Directly applying the Rademacher complexity to calculate the generalization error bound fails because the large complexity of neural network's function space leads to trivial bound[41]. In this work, we want to calculate the complexity of the function space that can be explored during the training process. However, there are two challenges here. First, the trajectory of neural network is a "line", instead of a function space that can be calculated the complexity. To solve this problem, we introduce the addictive linear space  $\mathcal{L}_{\mathbf{J}|S}$ . This space contains the local information of learning trajectory, and can serve as the pseudo function space. Second, the function space  $\mathcal{L}_{\mathbf{J}|S}$  has a dependent on the sample set  $S$ , while the theory of Rademacher complexity requires that the function space is independent with training samples. To decouple this dependence, we adapt the Rademacher complexity and we obtain that  $\mathbb{E}[\mathrm{gen}^{lin}(\mathbf{J}_{\mathbf{T}})] \leq 2\gamma' \mathbb{V}_m \mathbb{E}R_S(\mathcal{L}_{\mathbf{J}|S})$ . Here,  $\gamma'$  is introduced to decouple the dependent fact mentioned above.
Next, in order to draw a clearer comparison with the stability-based method, we present the following corollary. This corollary employs the  $\beta$ -smooth assumption to bound  $\mathrm{gen}^{nl}(\mathbf{J}_{\mathbf{T}})$  and leverages a similar learning rate setting to that found in stability based works.
Corollary 3.8. If function  $f(\cdot)$  is  $\beta$ -smooth, under Assumption 3.4 given  $S \sim \mu^n$ , let  $\mathbf{J} = \mathcal{A}(S)$ ,  $\eta_t = \frac{c}{\beta(t + 1)}$ ,  $M_2^2 = \max_t \mathbb{E}_{t - 1}(\| \nabla F_S(\mathbf{J}_t) + \epsilon (\mathbf{J}_t)\|^2)$  and  $M_4^4 = \max_t \mathbb{E}_{t - 1}(\| \nabla F_S(\mathbf{J}_t) + \epsilon (\mathbf{J}_t)\|^4)$ , where  $\mathcal{A}$  denoted the SGD or GD algorithm training with  $T$  steps, we have:
$$
\begin{array}{l} \mathbb {E} \left[ F _ {\mu} (\mathbf {J} _ {\mathbf {T}}) - F _ {S} (\mathbf {J} _ {\mathbf {T}}) \right] \leq - 2 \gamma^ {\prime} \mathbb {V} _ {m} \mathbb {E} \int_ {t} \frac {d F _ {S} (\mathbf {J} _ {\mathbf {t}})}{\sqrt {n}} \sqrt {1 + \frac {\operatorname {T r} (\Sigma (\mathbf {J} _ {\mathbf {t}}))}{\| \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) \| _ {2} ^ {2}}} \\ + 2 c ^ {2} \gamma^ {\prime} \mathbb {V} _ {m} M _ {4} ^ {2} \sqrt {\mathbb {E} \int_ {t} \frac {d t}{n \beta^ {2} (t + 1) ^ {4}} \left(1 + \frac {\operatorname {T r} (\Sigma (\mathbf {J} _ {\mathbf {t}}))}{\| \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) \| _ {2} ^ {2}}\right)} \tag {10} \\ + 2 c ^ {2} \frac {M _ {2} ^ {2}}{\beta}. \\ \end{array}
$$
$$
\begin{array}{l l} w h e r e \quad \mathbb {V} (\mathbf {w}) & = \frac {\| \nabla F _ {S} (\mathbf {w}) \|}{\mathbb {E} _ {U \subset S} \| \frac {| U |}{n} \nabla F _ {U} (\mathbf {w}) - \frac {n - | U |}{n} \nabla F _ {S / U} (\mathbf {w}) \|}, \quad \mathbb {V} _ {m} \\ & = \max  _ {t} \mathbb {V} (\mathbf {J} _ {\mathbf {t}}) \quad a n d \quad \gamma^ {\prime} \quad = \\ \max  \{1, \max  _ {U \subset S; t} \frac {| U | \| \nabla F _ {U} (\mathbf {J} _ {\mathbf {t}}) \|}{n \| \nabla F _ {S} (\mathbf {J} _ {\mathbf {t}}) \|} \} \gamma . \end{array}
$$
3.3 Further Analysis
3.3.1 Interpreting the Generalization Bounds
We rewrite the obtained generalization bound here:
$$
\mathbb {E} \left[ F _ {\mu} \left(\mathbf {J} _ {\mathbf {T}}\right) - F _ {S} \left(\mathbf {J} _ {\mathbf {T}}\right) \right] \leq \underbrace {\gamma^ {\prime}} _ {\text {B i a s o f T r a i n i n g S e t}} \overbrace {\mathbb {V} _ {m}} ^ {\frac {1}{\text {D i v e r s y o f T r a i n i n g S e t}}} \underbrace {\left(- 2 \mathbb {E} \int_ {t} \frac {d F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right)}{\sqrt {n}} \sqrt {1 + \frac {\operatorname {T r} \left(\Sigma \left(\mathbf {J} _ {\mathbf {t}}\right)\right)}{\| \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) \| _ {2} ^ {2}}}\right)} _ {\text {C o m p l e x i t y o f L e a n i n g T r a j e c t o r y}} + \mathcal {O} (\eta_ {m}) \tag {11}
$$
The "Bias of Training Set" refers to the disparity between the characteristics of the training set and those of the broader population. To measure this difference, we use the distance between the norm of the popular gradient and that of the training set gradient, as specified in Assumption 3.4. The "Diversity of Training Set" can be understood as the variation among the samples in the training set, which in turn affects the quality of the training data. The ratio  $\frac{\text{Bias of Training Set}}{\text{Diversity of Training Set}}$  gives us the property of information conveyed by the training set. It is important to consider the properties of the training set, as the data may not contribute equally to the generalization[36]. The detail version of the equation can be found in Theorem 3.6.
3.3.2 Asymptotic Analysis
We will first analyze the dependent of  $n$  for  $\mathbb{V}$ . The  $\mathbb{V}$  is calculated as  $\mathbb{V}(\mathbf{w}) = \frac{\|\nabla F_S(\mathbf{w})\|}{\mathbb{E}_{U\subset S}\|\frac{|U|}{n}\nabla F_U(\mathbf{w}) - \frac{n - |U|}{n}\nabla F_{S / U}(\mathbf{w})\|}$ . Obviously, the gradient of individual sample is unrelated to the sample size  $n$ . And  $|U| \sim n$ . Therefore,  $\mathbb{V} = \mathcal{O}(1)$ . Similarly, we have  $\mathbb{E}\int_t\frac{dF_S(\mathbf{J_t})}{\sqrt{n}}\sqrt{1 + \frac{\mathrm{Tr}(\Sigma(\mathbf{J_t}))}{\|\nabla F_S(\mathbf{J_t})\|_2^2}} = \mathcal{O}(\frac{1}{\sqrt{n}})$ . As for the  $\mathcal{O}(\eta_m)$  term in Theorem 3.6, we have  $\lim_{n\to \infty}\mathcal{O}(\eta_m) = 0$  according to Proposition A.1. We simply assume that  $\mathcal{O}(\eta_m) = \mathcal{O}(\frac{1}{n^c})$ . Therefore, our bound has  $\mathcal{O}(\frac{1}{n^{\min\{0.5,c\}}})$ .
3.3.3 Comparison with Stability-based methods
We first compare our method with the stability-based methods in terms of the trajectory information. In Table 1, we present a summary of stability-based methods, while other methods are outlined in Appendix D. We focus on generalization bounds from previous works that eliminate terms dependent on extra information about data distribution  $\mu$ , apart from the training data  $S$ , using assumptions such as smoothness or Lipschitz continuity. Analyzing Table 1 reveals that most prior works primarily depend on the learning rate  $\eta$  and the total number of training steps  $T$ . This suggests that we can achieve the same bound by using an identical learning rate schedule and total training steps, which does not align with our practical experience. Our proposed generalization bound considers the evolution of function values, gradient covariance, and gradient norms throughout the training process. As a result, our bounds encompass more comprehensive information about the learning trajectory.
Table 2: Detail comparison with Hardt et al. [11]. The  $\beta$  refers to the  $\beta$ -smooth assumption (see in Definition 3.2).  $S$  denotes the training set.
Uniform Stability[11]	Ours
Assumption	∀w ∈ W ∀z' ∈ Z ||∇f(w, z')|| ≤ L	∀w ∈ {Jt|t ∈ N} ||Ez'~μ∇f(w, z')|| ≤ γ ||Ez∈S∇f(w, z')||
Modelling Method of SGD	Epoch Structure	Full Batch Gradient + Stochastic Noise
Batch Size	1	≤ n
Trajectory Information in Bound	Learning rate and number of training step	Values in Trajectory (gradient norm and covariance)
Perspective	Stability of Algorithm	Complexity of Learning Trajectory
Next, we give a detail comparison with Hardt et al. [11] in Table 2. The concept of uniform stability is commonly used to evaluate the ability of SGD in generalization, by assessing its stability when a single training sample is altered. Our primary point of comparison is with Hardt et al. [11], as their work is considered the most representative in terms of analyzing the stability of SGD. We find that First, the assumption of Uniform Stability requires the gradient norm of all input samples for all weights being bounded by  $L$ , whereas our assumption only limits the expectation of the gradients
for the weights during the learning trajectory. Secondly, Uniform Stability uses an epoch structure to model the stochastic gradient descent, whereas our approach regards each stochastic gradient descent as full batch gradient descent with added stochastic noise. The epoch structure complicates the modelling process because it requires a consideration of sampling. As a result, in Hardt et al. [11], the author only considers the setting with batch size 1. Thirdly, the bound of Uniform Stability only uses hyperparameters setting such as learning rate and number of training step. In contrast, our bound contains more trajectory-related information, such as the gradient norm and covariance. Finally, the Uniform Stability provides the generalization bound based on the stability of the algorithm, while our approach leverages the complexity of the learning trajectory. In summary, there are some notable differences between our approach and Uniform Stability, such as the assumptions made, the modelling process, the type of information used in the bound, and the perspectives.
4.1 Tightness of Our Bounds
Table 3: Numeric comparison with stability-based work on toy examples. The reason for the value of Zhang et al. [43] is large is because that our and Hardt et al. [11] has dependent on  $\frac{L^2}{\beta}$ , while Zhang et al. [43] depends on  $L^2$ .  $L$  and  $\beta$  are usually large numbers.
Gen Error	Ours	Hardt et al. [11]	Zhang et al. [43]
1.49	3.62	4.04	4417
In a toy dataset setting, we compare our generalization bound with stability-based methods.
Reasons for toy examples 1) Some values in the bounds are hard to be calculated. Calculating  $\beta$  (under the  $\beta$ -smooth assumption) and  $L$  (under the  $L$ -Lipschitz assumption) in stability-based work, as well as the values of  $\mathbb{V}$  and  $\gamma$  in our proposed bound, are challenging. 2) Stability-based methods require a batch size of 1. The training is hard for batch size of 1 with learning rate setting  $\eta_t = \frac{1}{t}$  in complex datasets.
**Construction of the toy examples** In the following, we discuss the construction of the toy dataset used to compare the tightness of the generalization bounds. The training data is  $X_{tr} = \{x_i\}_{i=1}^n$ . All the data  $x_i$  is sampled from Gaussian distribution  $\mathcal{N}(0, \mathbf{I}_d)$ . Sampling  $\tilde{\mathbf{w}} \sim \mathcal{N}(0, \mathbf{I}_d)$ , the ground truth is generated by  $y_i = 1$  if  $\tilde{\mathbf{w}}^{\mathrm{T}} x_i > 0$  else 0. The weights for learning is denoted as  $\mathbf{w}$ . The predict  $\tilde{y}$  is calculated as  $\tilde{y}_i = \mathbf{w}^{\mathrm{T}} x_i$ . The loss for a simple data point is  $l_i = \left\| y_i - \mathbf{w}^{\mathrm{T}} x_i \right\|_2$ . The training loss is  $\mathcal{L} = \sum_{i=1}^{n} l_i$ . The test data is  $X_{te} = \{x_i'\}$ , where  $x_i' = \tilde{x}_i'$  and  $\tilde{x}_i' \sim \mathcal{N}(0, \mathbf{I}_d)$ . We use 100 samples for training and 1,000 samples for evaluation. The model is trained using SGD for 200 epochs.
We evaluate the tightness of our bound by comparing our results with those in Hardt et al. [11] and Zhang et al. [43] from the original paper. We set the learning rate as  $\eta_t = \frac{1}{\beta t}$ . Our reasons for comparing with these two papers are: 1) Hardt et al. [11] is a representative study, 2) Both papers have theorems using a learning rate setting of  $\eta_t = \mathcal{O}(\frac{1}{t})$ , which aligns with Corollary 3.8 in our paper, and 3) They do not assume convexity. The generalization bounds we compare include Corollary 3.8 from our paper, Theorem 3.12 from Hardt et al. [11], and Theorem 5 from Zhang et al. [43].
Our results are given in Table 3. Our bound is tighter under this setting.
4.2 Capturing the trend of generalization error
In this section, 1) we conduct the deep learning experiment to verify Assumption 3.4 and 2) Verify whether our proposed generalization bound can capture the changes of generalization error. In this experiment, we mainly consider the term  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})\triangleq -2\int_{i = 0}^{t}\frac{dF_{S}(\mathbf{J}_{i})}{\sqrt{n}}\sqrt{1 + \frac{\mathrm{Tr}(\Sigma(\mathbf{J}_{i}))}{\|\nabla F_{S}(\mathbf{J}_{i})\|_{2}^{2}}}$ . We omit the term  $\gamma^{\prime}$  and  $\mathbb{V}_m$ , because all the trajectory related information that we want to explore is stored in  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$ . Capturing the trend of generalization error is regarded as an important problem in Nagarajan [21]. Unless further specified, we use the default setting of the experiments on CIFAR-10 dataset [14]
Figure 1: Exploration of Assumption 3.4 for different dataset. The  $\widetilde{\gamma}_t$  is stable before training loss reaches a relative small value. Assumption holds if the training is stop before extremely overfitting. A relaxed assumption and its corresponding generalization bound are given in Appendix B for extremely overfitting situation
Figure 2: Exploration of  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  during the training process. Left: The curve  $F_{S}(\mathbf{J}_{\mathbf{t}}) + \mathcal{C}(\mathbf{J}_{\mathbf{t}})$  exhibits a comparable pattern with the curve  $F_{S^{\prime}}(\mathbf{J}_{\mathbf{t}})$ . Center: After the early stage,  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  and  $\nabla F_{S^{\prime}}(\mathbf{J}_{\mathbf{t}}) - \nabla F_{S}(\mathbf{J}_{\mathbf{t}})$  have a similar trend. Right: The value of  $\frac{d\mathcal{C}(\mathbf{J}_{\mathbf{t}})}{dF_{S}(\mathbf{J}_{\mathbf{t}})}$  alone the training process.
with the VGG13 [34] network. The experimental details for each figure can be found in Appendix C.2.
Our observations are:
- Assumption 3.4 is valid when SGD is not exhibiting extreme overfitting.
- The term of  $\mathcal{C}(\mathbf{J}_t)$  can depict how the generalization error varies along the training process. And it can also track the changes in generalization error when adjustments are made to learning rates and label noise levels
Exploring the assumption 3.4 for different dataset during the training process To explore the Assumption 3.4, we define  $\gamma_{t}\triangleq \frac{\|\nabla F_{\mu}(\mathbf{J}_{t})\|}{\|\nabla F_{S}(\mathbf{J}_{t})\|}$  and  $\widetilde{\gamma}_t\triangleq \frac{\|\nabla F_{S'}(\mathbf{J}_t)\|}{\|\nabla F_S(\mathbf{J}_t)\|}$ , where  $S^{\prime}$  is another data set i.i.d sampled from distribution  $\mu$ . Because  $S^{\prime}$  is independent with  $S$ , we have  $\widetilde{\gamma}_t\approx \gamma_t$ . We found that  $\widetilde{\gamma}_t$  is stable around 1 during the early stage of training(Figure 1). When the training loss is reaching a relative small value,  $\widetilde{\gamma}_t$  increases as we continue training. This phenomenon remains consistent across the Cifar10, Cifar100 and SVHN datasets. The  $\gamma$  in Assumption 3.4 can be assigned as  $\gamma = \max_t\widetilde{\gamma}_t$ . We can always find such  $\gamma$  if the optimizer is not extreme overfitting. Under the extremely overfitting case, we can use the relaxed theorem in Appendix B to bound the generalization error.
The bound capturing the trend of generalization error during training process The generalization error and the  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  both changes as the training continues. Therefore, we want to verify whether they correlate with each other during the training process. Here, we use the term  $\nabla F_{S^{\prime}}(\mathbf{J}_{\mathbf{t}}) - \nabla F_{S}(\mathbf{J}_{\mathbf{t}})$  to approximate the generalization error. We find that  $\nabla F_{S^{\prime}}(\mathbf{J}_{\mathbf{t}}) - \nabla F_{S}(\mathbf{J}_{\mathbf{t}})$  has similar trend with  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  (Figure 2 Center). What's more, we also find that the curve of  $\nabla F_{S}(\mathbf{J}_{\mathbf{t}}) + \mathcal{C}(\mathbf{J}_{\mathbf{t}})$  exhibits a comparable pattern with the curve  $F_{S^{\prime}}(\mathbf{J}_{\mathbf{t}})$  (Figure 2 Left). To explore whether  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  reveals influence of the change of  $F_{S}(\mathbf{J}_{\mathbf{t}})$  to the generalization error, we plot  $\frac{d\mathcal{C}(\mathbf{J}_{\mathbf{t}})}{dF_{S}(\mathbf{J}_{\mathbf{t}})}$  (Figure 2 Right) during
Figure 3:  $\mathcal{C}(\mathbf{J}_{\mathbf{T}})$  correlates with  $\nabla F_{S^{\prime}}(\mathbf{J}_{\mathbf{t}}) - \nabla F_{S}(\mathbf{J}_{\mathbf{t}})$ . Left:  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  and the generalization error under different label noise level. Right:  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  and the generalization error under learning rate. The  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  can capture the trend of generalization error caused by learning rate when learning rate is small. Appendix E provides proof that a large learning rate results in a smaller proposed generalization bound. Further discussions on why a small learning rate leads to a larger generalization error can be found in Li et al. [17], Barrett and Dherin [3].
the training process.  $\frac{d\mathcal{C}(\mathbf{J}_t)}{dF_S(\mathbf{J}_t)}$  increases slowly during the early stage of training, but surge rapidly afterward. This discovery is aligned with our intuition about the overfitting.
The complexity of learning trajectory correlates with the generalization error In Figure 3, we carry out experiments under various settings. Each data point in the figure represents the average of three repeated experiments. The results demonstrate that both the generalization error and  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  increase as the level of label noise is raised (Figure 3 Left). The another experiments measure  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  and generalization error for different learning rate and discover that  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$  can capture the trend generalization error. The reasons behind a larger learning rate resulting in a smaller generalization error have been explored in Li et al. [17], Barrett and Dherin [3]. Additionally, Appendix E discusses why a larger learning rate can lead to a smaller  $\mathcal{C}(\mathbf{J}_{\mathbf{t}})$ .
The assumption of small learning rate is required by our method. But this assumption is also common use in previous works. For example, Hardt et al. [11], Zhang et al. [43], Zhou et al. [44] explicitly requires that the learning rate should be small and is decayed with a rate of  $\mathcal{O}\left(\frac{1}{t}\right)$ . Some methods have no explicit requirements about this but show that large learning rate pushes the generalization bounds to a trivial point. For example, the generalization bounds in works [5, 16] have a term  $\sum_{t=1}^{T} \eta_t^2$  that is not decayed as the data size  $n$  increases. The value of this term is ignorable when the learning rate is large. The small learning assumption widens the gap between theory and practice. Eliminating this assumption is crucial for future work.
In this study, we investigate the relation between learning trajectories and generalization capabilities of Deep Neural Networks (DNNs) from a unique standpoint. We show that learning trajectories can serve as reliable predictors for DNNs' generalization performance. To understand the relation between learning trajectory and generalization error, we analyze how each update step impacts the generalization error. Based on this, we propose a novel generalization bound that encompasses extensive information related to the learning trajectory. The conducted experiments validate our newly proposed assumption. Experimental findings reveal that our method effectively captures the generalization error throughout the training process. Furthermore, our approach can also track changes in generalization error when adjustments are made to learning rates and the level of label noises.
We thank all the anonymous reviewers for their valuable comments. The work was supported in part with the National Natural Science Foundation of China (Grant No. 62088102).
A Proof of Theorem 3.6
We rewrite the Equation (7) and Equation (8):
$$
F _ {\mu} (\mathbf {J} _ {\mathbf {T}}) - F _ {S} (\mathbf {J} _ {\mathbf {T}}) = \underbrace {F _ {\mu} (\mathbf {J} _ {\mathbf {0}}) - F _ {S} (\mathbf {J} _ {\mathbf {0}})} _ {(i)} + \sum_ {t = 1} ^ {T} \underbrace {[ (F _ {\mu} (\mathbf {J} _ {\mathbf {t}}) - F _ {\mu} (\mathbf {J} _ {\mathbf {t} - \mathbf {1}})) - (F _ {S} (\mathbf {J} _ {\mathbf {t}}) - F _ {S} (\mathbf {J} _ {\mathbf {t} - \mathbf {1}})) ] ,} _ {(i i) _ {t}} \tag {12}
$$
$$
\mathbb {E} \left[ F _ {\mu} \left(\mathbf {J} _ {\mathbf {T}}\right) - F _ {S} \left(\mathbf {J} _ {\mathbf {T}}\right) \right] = \mathbb {E} \sum_ {t = 1} ^ {T} (i i) _ {t}. \tag {13}
$$
Using Taylor expansion for the function  $f(\cdot)$ , we have:
$$
f \left(\mathbf {J} _ {\mathbf {t}}\right) - f \left(\mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right) = \left(\mathbf {J} _ {\mathbf {t}} - \mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right) ^ {\mathrm {T}} \nabla f \left(\mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right) + \mathcal {O} \left(\left\| \mathbf {J} _ {\mathbf {t} + \mathbf {1}} - \mathbf {J} _ {\mathbf {t}} \right\| ^ {2}\right). \tag {14}
$$
Therefore, we can define  $(ii)_t^{lin}$  as:
$$
\left(i i\right) _ {t} ^ {l i n} \triangleq \left(\mathbf {J} _ {\mathbf {t}} - \mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right) ^ {\mathrm {T}} \left(\nabla F _ {\mu} \left(\mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right) - \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t} - \mathbf {1}}\right)\right). \tag {15}
$$
The  $(ii)_t$  can be decomposed as  $(ii)_t = (ii)_t^{lin} + (ii)_t^{nl}$ , where  $(ii)_t^{nl} \triangleq (ii)_t - (ii)_t^{lin}$ .
Then Equation 13 can be decomposed as:
$$
\mathbb {E} \left[ F _ {\mu} \left(\mathbf {J} _ {\mathbf {T}}\right) - F _ {S} \left(\mathbf {J} _ {\mathbf {T}}\right) \right] = \mathbb {E} \underbrace {\sum_ {t = 1} ^ {T} (i i) _ {t} ^ {l i n}} _ {\text {g e n} ^ {l i n} \left(\mathbf {J} _ {\mathbf {T}}\right)} + \mathbb {E} \underbrace {\sum_ {t = 1} ^ {T} (i i) _ {t} ^ {n l}} _ {\text {g e n} ^ {n l} \left(\mathbf {J} _ {\mathbf {T}}\right)}. \tag {16}
$$
Proposition A.1. For the gradient descent or the stochastic gradient descent algorithm, we have:
$$
\mathbb {E} \left[ \operatorname {g e n} ^ {l i n} \left(\mathbf {J} _ {\mathbf {T}}\right) \right] = \mathbb {E} \left[ \sum_ {t = 0} ^ {T - 1} \eta_ {t} \nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) ^ {\mathrm {T}} \left(\nabla F _ {S} \left(\mathbf {J} _ {\mathbf {t}}\right) - \nabla F _ {\mu} \left(\mathbf {J} _ {\mathbf {t}}\right)\right) \right]. \tag {17}
$$
If  $T = \mathcal{O}\left(\frac{1}{\eta_m}\right)$ , we have:
$$
| \operatorname {g e n} ^ {n l} (\mathbf {J} _ {\mathbf {T}}) | = \mathcal {O} \left(\eta_ {m}\right), \tag {18}
$$
where  $\eta_{m}\triangleq \max_{t}\eta_{t}$  , and we have:
$$
\lim  _ {n \rightarrow \infty} | \operatorname {g e n} ^ {n l} (\mathbf {J} _ {\mathbf {T}}) | = 0. \tag {19}
$$