SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 977-988. IEEE, 2022.
[2] Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q-networks. In 2018 Information Theory and Applications Workshop (ITA), pages 1-9. IEEE, 2018.
[3] Ainesh Bakshi, Rajesh Jayaram, and David P Woodruff. Learning two layer rectified neural networks in polynomial time. In Conference on Learning Theory, pages 195-268. PMLR, 2019.
[4] Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory, pages 1691-1692. PMLR, 2018.
[5] Rajendra Bhatia. Matrix analysis, volume 169. Springer Science & Business Media, 2013.
[6] Steven J Bradtke and Andrew G Barto. Linear least-squares algorithms for temporal difference learning. Machine learning, 22(1):33-57, 1996.
[7] Alon Brutzkus and Amir Globerson. Globally optimal gradient descent for a convnet with gaussian inputs. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 605–614. JMLR.org, 2017.
[8] Alon Brutzkus and Amir Globerson. An optimization and generalization analysis for max-pooling networks. In Uncertainty in Artificial Intelligence, pages 1650-1660. PMLR, 2021.
[9] Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. Sgd learns overparameterized networks that provably generalize on linearly separable data. In International Conference on Learning Representations, 2018.
[10] Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal-difference learning converges to global optima. Advances in Neural Information Processing Systems, 32, 2019.
[11] Yichen Chen, Lihong Li, and Mengdi Wang. Scalable bilinear pi learning using state and action features. In International Conference on Machine Learning, pages 834-843. PMLR, 2018.
[12] Yu Fan Chen, Michael Everett, Miao Liu, and Jonathan P How. Socially aware motion planning with deep reinforcement learning. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1343-1350. IEEE, 2017.
[13] Zaiwei Chen, Sheng Zhang, Thinh T Doan, Siva Theja Maguluri, and John-Paul Clarke. Performance of q-learning with linear function approximation: Stability and finite-time analysis. arXiv preprint arXiv:1905.11425, page 4, 2019.
[14] Girish Chowdhary, Miao Liu, Robert Grande, Thomas Walsh, Jonathan How, and Lawrence Carin. Off-policy reinforcement learning with gaussian processes. IEEE/CAA Journal of Automatica Sinica, 1(3):227-238, 2014.
[15] Mohammed Nowaz Rabbani Chowdhury, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. Patch-level routing in mixture-of-experts is provably sample-efficient for convolutional neural networks. arXiv preprint arXiv:2306.04073, 2023.
[16] Antonio Coronato, Muddasar Naeem, Giuseppe De Pietro, and Giovanni Paragliola. Reinforcement learning for intelligent healthcare applications: A survey. Artificial Intelligence in Medicine, 109:101964, 2020.
Supplementary Materials for:
On the Convergence and Sample Complexity Analysis of Deep Q-Network with Epsilon-Greedy Exploration
The structure of the appendix mainly follows the roadmap of the proof described in Section 4.4.
In Appendix A, we define the characterizable population risk function in (31) to approximate the objective function. Also, some notations to simplify the analysis are introduced in Appendix A, and we recommend the readers to refer to Table 3 for the major notations used in the proofs.
In Appendix B, we provide the proof for Lemma 1 and Theorem 1 following the steps as (1) Characterization of the local convex region of population risk function (Lemma 2), (2) Characterization of the distance between the population risk function and the objective function (Lemma 3), (3) Characterization of the convergence of two consecutive iterations  $\mathbf{W}^{(t,m + 1)}$  and  $\mathbf{W}^{(t,m)}$ , and (4) Mathematical induction over the  $t$  and  $m$  to obtain the error bound between the convergent point  $\mathbf{W}^{(T,0)}$  and the desired point  $\mathbf{W}^{\star}$ .
In Appendix C, we provide the preliminary lemmas and the whole proof for Lemma 2, which characterizes the local convex region of the non-convex population risk function.
In Appendix D, we provide the preliminary lemmas and the whole proof for Lemma 3, which characterizes the difference of  $g_{t}$  and the gradient descent of defined population risk function in (31).
In Appendix E, we provide the proofs for the preliminary lemmas in proving Lemmas 2 and 3.
Before moving to the details, we provide an overview of the techniques in the proofs.
(P1.) The local convex region near  $W^{\star}$ . To characterize the local convex region, we first bound the Hessian matrix of the defined population risk function in (31) at  $W^{\star}$ . Then, we derive the changes in the Hessian matrix when the neuron weights move around the  $W^{\star}$ . Specifically, we prove that when neuron weights  $W$  are not far away  $W^{\star}$ , then the Hessian matrix in this region is always positive-definite, indicating that a local convex region near  $W^{\star}$ . [90] considers the one-hidden-layer neural network, and the lower bound of the Hessian matrix only holds for Gaussian input. Instead, in this paper, we consider multi-layer cases and need to derive a lower bound for the Hessian matrix for all the layers. Instead, the input of the intermediate layer cannot be proved to be Gaussian but belong to sub-Gaussian distribution. Therefore, we built the proof for the lower bound of the Hessian matrix when the input belongs to the sub-Gaussian distribution. Compared with Gaussian input, Sub-Gaussian does not have a closed form of the probability density function. Instead of directly calculating the lower bound, we convert the problem into proving a series of functions are linearly independent over a Hilbert space (see Lemma 7 and the proof in Appendix E). Instead of directly calculating the distance of the population risk function in different points, we characterize a Gaussian variable such that the distance over the sub-Gaussian distribution can be upper bounded by the one over the Gaussian variable (see Lemma 6 and the proof in Appendix E).
(P2.) The difference between the gradient  $g_{t}$  and the population risk function. With the local convex region of the population risk function, we can characterize the convergence of the population risk function. With Lemma 3, we can prove that the distance between the population risk function and  $g_{t}$  is small enough, the behaviors of the iterations via  $g_{t}$  can be described by the ones in the population risk function with some additional error terms. Compared with the proof in [90], We need to address the extension from supervised learning settings to Q learning settings and the extension from the one-hidden-layer neural networks to the multi-layer neural networks. First, similar to challenges in (P.1), we provide a new concentration bound to characterize the distance between the two functions for the intermediate layers (see  $I_{1}$  in the proof of Lemma 3). Second, the distance between the two functions has an additional error term due to the inconsistency of the label defined in (31) and (8) (see  $I_{2}$  in the proof of Lemma 3). Third, we need to develop a new concentration bound to characterize the error term caused by the distribution shift when training samples are collected by  $\varepsilon$ -greedy policy (see  $I_{3}$  in the proof of Lemma 3).
(P3.) The convergence analysis of Algorithm 1. When the initialization is not far away from  $W^{\star}$ , the initialization lies in the local convex region of  $W^{\star}$  for the population risk function. When we have enough samples  $N$  and a large enough  $\varepsilon_{t}$ , we can guarantee that the distance between the  $g_{t}$  and the gradient of the population risk function is small enough such that the iterations following  $g_{t}$  converge to a point nearby  $W^{\star}$  as well. However, if  $\varepsilon_{t}$  is too large, the convergent point nearby  $W^{\star}$  can be even worse than the initial point. To avoid this issue, we have an upper bound for selecting  $\varepsilon_{t}$ , and the upper bound decreases as  $\| W^{(t,0)} - W^{\star}\|$  decreases over  $t$ . Therefore, we build the convergence analysis of Algorithm 1.
A Definitions and Notations
In this section, we implement the details of algorithms described in Algorithm 1, and some important notations are defined to simplify the presentation of the proof.
A.1 Definition of the Empirical Risk Function and Its Corresponding Notations
Recall that the goal of  $Q$ -learning is to find the  $Q^{\star}$ -function to minimize (6). Therefore, we have
$$
Q ^ {\star} (\boldsymbol {s}, a) = r (\boldsymbol {s}, a) + \gamma \cdot \mathbb {E} _ {\boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a} \max  _ {a ^ {\prime} \in \mathcal {A}} Q ^ {\star} \left(\boldsymbol {s} ^ {\prime}, a ^ {\prime}\right) \quad f o r \quad (\boldsymbol {s}, a) \sim \mu^ {\star}. \tag {29}
$$
Since  $W^{\star}$  is the global minimal to (6), we have
$$
Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s}, a\right) = r (\boldsymbol {s}, a) + \gamma \cdot \mathbb {E} _ {\boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a} \max  _ {a ^ {\prime} \in \mathcal {A}} Q \left(\boldsymbol {W} ^ {\star}; \boldsymbol {s} ^ {\prime}, a ^ {\prime}\right). \tag {30}
$$
Therefore, the population risk function is defined as
$$
\begin{array}{l} f (\pmb {W}) = \mathbb {E} _ {(\pmb {s}, a) \sim \mu^ {\star}} \left[ Q (\pmb {W}; \pmb {s}, a) - r (\pmb {s}, a) - \gamma \cdot \mathbb {E} _ {\pmb {s} ^ {\prime} | \pmb {s}, a} \max _ {a ^ {\prime} \in \mathcal {A}} Q (\pmb {W} ^ {\star}; \pmb {s} ^ {\prime}, a ^ {\prime}) \right] ^ {2} \\ = \mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu^ {\star}} \left[ Q (\boldsymbol {W}; \boldsymbol {s}, a) - Q (\boldsymbol {W} ^ {\star}; \boldsymbol {s}, a) \right] ^ {2}, \\ \end{array}
$$
where  $\mu^{*}$  is the distribution of the sampled data following the optimal policy  $\pi^{\star}$ .
The gradient of the (31) is
$$
\begin{array}{l} \nabla_ {\boldsymbol {W}} f (\boldsymbol {W}) = \mathbb {E} _ {\boldsymbol {x} \sim \mu^ {\star}} \left(Q (\boldsymbol {W}; \boldsymbol {x}) - r (\boldsymbol {x}) - \gamma \cdot \mathbb {E} _ {\boldsymbol {s} ^ {\prime} \sim p _ {\boldsymbol {s}, \boldsymbol {s} ^ {\prime}} ^ {a}} \max  _ {a ^ {\prime} \in \mathcal {A}} Q ^ {\star} (\boldsymbol {s} ^ {\prime}, a ^ {\prime})\right) \cdot \nabla_ {\boldsymbol {W}} Q (\boldsymbol {W}; \boldsymbol {x}) \\ = \mathbb {E} _ {\boldsymbol {x} \sim \mu^ {*}, \boldsymbol {s} ^ {\prime} \sim p _ {s, s ^ {\prime}} ^ {a}} \left(Q (\boldsymbol {W}; \boldsymbol {x}) - r (\boldsymbol {x}) - \gamma \cdot \max  _ {a ^ {\prime} \in \mathcal {A}} Q \left(\boldsymbol {W} ^ {*}; \boldsymbol {s} ^ {\prime}, a ^ {\prime}\right)\right) \cdot \nabla_ {\boldsymbol {W}} Q (\boldsymbol {W}; \boldsymbol {x}). \tag {32} \\ \end{array}
$$
As  $\pmb{W}^{\star}$  is one of the ground truths to  $f(\pmb{W})$ , i.e.,  $f(\pmb{W}^{\star})$  achieves the minimum value as  $f(\pmb{W}^{\star}) = 0 \leq f(\pmb{W})$  for any other  $\pmb{W}$ . Given  $f$  is a smooth function, we have the gradient of  $f$  with respect to any  $\pmb{W}_{\ell}$  at the ground truth  $\pmb{W}^{\star}$  equals to zero, namely,
$$
\nabla_ {\ell} f \left(\boldsymbol {W} ^ {\star}\right) := \nabla_ {\boldsymbol {W} _ {\ell}} f \left(\boldsymbol {W} ^ {\star}\right) = \mathbf {0}, \quad \forall \ell \in [ L ]. \tag {33}
$$
In addition, without special descriptions,  $\pmb{\alpha} = [\pmb{\alpha}_1^\top, \pmb{\alpha}_2^\top, \dots, \pmb{\alpha}_K^\top]^\top$  stands for any unit vector that in  $\mathbb{R}^{K_{\ell}K_{\ell-1}}$  with  $\pmb{\alpha}_j \in \mathbb{R}_{\ell-1}^K (K_0 = d)$ . Therefore, we have
$$
\begin{array}{l} \| \nabla_ {\ell} h \| _ {2} = \max  _ {\boldsymbol {\alpha}} \| \boldsymbol {\alpha} ^ {\top} \nabla_ {\ell} h \| _ {2} = \max  _ {\boldsymbol {\alpha}} \left| \sum_ {j = 1} ^ {K} \boldsymbol {\alpha} _ {j} ^ {\top} \frac {\partial h}{\partial \boldsymbol {w} _ {\ell , j}} \right|, \\ \left\| \nabla_ {\ell} ^ {2} h \right\| _ {2} = \max  _ {\boldsymbol {\alpha}} \left\| \boldsymbol {\alpha} ^ {\top} \nabla_ {\ell} ^ {2} h \boldsymbol {\alpha} \right\| _ {2} = \max  _ {\boldsymbol {\alpha}} \left(\sum_ {j = 1} ^ {K} \boldsymbol {\alpha} _ {j} ^ {\top} \frac {\partial h}{\partial \boldsymbol {w} _ {\ell , j}}\right) ^ {2}. \tag {34} \\ \end{array}
$$
A.2 Notations in Algorithm 1
Recall that the gradient in the  $t$ -th loop is
$$
\begin{array}{l} g _ {t} (\boldsymbol {W}) = \frac {1}{\left| \mathcal {D} _ {t} ^ {(m)} \right|} \sum_ {n \in \mathcal {D} _ {t} ^ {(m)}} \left(Q (\boldsymbol {W}; \boldsymbol {x} _ {n}) - y _ {n} ^ {(t)}\right) \cdot \nabla_ {\boldsymbol {W}} Q (\boldsymbol {W}; \boldsymbol {x} _ {n}) \\ = \frac {1}{N} \sum_ {n = 1} ^ {N} \left(Q (\boldsymbol {W}; \boldsymbol {x} _ {n}) - r (\boldsymbol {x} _ {n}) - \gamma \cdot \max  _ {a ^ {\prime} \in \mathcal {A}} Q \left(\boldsymbol {W} ^ {(t - 1)}; \boldsymbol {s} _ {n} ^ {\prime}, a ^ {\prime}\right)\right) \cdot \nabla_ {\boldsymbol {W}} Q (\boldsymbol {W}; \boldsymbol {x} _ {n}). \tag {35} \\ \end{array}
$$
Then, we define  $g_{t}^{(m)}(\pmb{W}_{\ell}; \pmb{W})$  as the components of  $g_{t}^{(m)}(\pmb{W})$  with respect to  $\pmb{W}_{\ell}$ . Recall that in (4) we have
$$
\boldsymbol {W} = \left[ \operatorname {v e c} \left(\boldsymbol {W} _ {1}\right) ^ {\top}, \quad \operatorname {v e c} \left(\boldsymbol {W} _ {2}\right) ^ {\top}, \quad \dots , \quad \operatorname {v e c} \left(\boldsymbol {W} _ {L}\right) ^ {\top} \right] ^ {\top}. \tag {36}
$$
Then, with the definition of  $g_t^{(m)}(\pmb{W}_\ell; \pmb{W})$ , we have
$$
g _ {t} ^ {(m)} (\boldsymbol {W}) = \left[ g _ {t} ^ {(m)} \left(\boldsymbol {W} _ {1}; \boldsymbol {W}\right) ^ {\top}, \quad g _ {t} ^ {(m)} \left(\boldsymbol {W} _ {2}; \boldsymbol {W}\right) ^ {\top}, \quad \dots , \quad g _ {t} ^ {(m)} \left(\boldsymbol {W} _ {L}; \boldsymbol {W}\right) ^ {\top} \right] ^ {\top}. \tag {37}
$$
To simplify the analysis, the update of  $\mathbf{W}^{(t,m)}$  is analyzed in the form of
$$
\boldsymbol {W} _ {\ell} ^ {(t, m + 1)} = \boldsymbol {W} _ {\ell} ^ {(t, m)} - \eta \cdot g _ {t} ^ {(m)} \left(\boldsymbol {W} _ {\ell}; \boldsymbol {W} ^ {(t, m)}\right) + \beta \left(\boldsymbol {W} _ {\ell} ^ {(t, m)} - \boldsymbol {W} _ {\ell} ^ {(t, m - 1)}\right), \quad \forall \ell \in [ L ]. \tag {38}
$$
One can see that (38) returns the same  $\mathbf{W}^{(t,m + 1)}$  as the gradient step at line 9 in Algorithm 1.
Table 3: Notations for the proofs
gt(W)	The gradient function at point W in the t-th outer loop, defined in (7).
gt(Wl;W)	The gradient function of gt(W) with respect to the components of Wl.
d	Dimension of the feature mappings of the state-action pair (s,a) ∈ S × A.
K	Number of neurons in the hidden layer.
L	Number of hidden layers.
W*	The desired Weights for approximating the optimal Q function.
W(t,m)	Model returned by Algorithm 1 at t-th outer loop and m-th inner loop.
f	The population risk function defined in (31).
∇Wf(W*)	The full gradient of a function f at point W*.
∇lf(W*)	The gradient of a function f with respect to the components of Wl at point W*.
∇2f(W*)	The Hessian matrix of a function f with respect to the components of Wl at point W*.
n	The dimension of W.
nl	The dimension of vectorized Wl.
h(ℓ)(W)	The input to the l-th layer, defined in (39).
Kl	The dimension of h(ℓ).
Jl(W)	A function in Rn → RK, defined in (42).
εt	The value of ε in the behavior policy at t-th outer loop.
Ct	The distribution shift between the optimal policy and behavior policy at iteration t.
N	The size of the experience replay buffer.
Rmax	The upper bound of the reward.
A.3 Notations for the Deep Neural Networks.
Let  $n$  denote the dimension of  $W$  defined in (4). We denote  $n_l$  as the dimension of the vectorized neuron weights in the  $\ell$ -th layer, namely,  $n_\ell = \dim(\operatorname{vec}(W_\ell))$ .
Then, let  $h^{(\ell)}(\mathbf{W})$  denote the input in the  $\ell$ -th layer (or the output in the  $(\ell - 1)$ -th layer) with respect to the neuron weights as  $\mathbf{W}$ , and  $h^{(1)} = (\mathbf{s}, a)$ , where
$$
\boldsymbol {h} ^ {(\ell)} (\boldsymbol {W}) = \phi \left(\boldsymbol {W} _ {\ell - 1} ^ {\top} \boldsymbol {h} ^ {(\ell - 1)}\right) = \dots = \phi \left(\boldsymbol {W} _ {\ell} ^ {\top} \phi \left(\boldsymbol {W} _ {\ell - 1} \dots \phi \left(\boldsymbol {W} _ {1} ^ {\top} \boldsymbol {x}\right)\right)\right). \tag {39}
$$
$\pmb{h}^{(\ell)}(\pmb{W})$  may be shortened as  $\pmb{h}^{(\ell)}$  when the neuron weights are clear from the contexts. Then, we denote the dimension of  $\pmb{h}^{(\ell)}$  as  $K_{\ell}$ , where
$$
K _ {\ell} = \left\{ \begin{array}{l l} K, & i f \quad \ell > 1 \\ d, & i f \quad \ell = 1. \end{array} \right. \tag {40}
$$
Then,  $Q(\boldsymbol{W}; \boldsymbol{s}, a)$  can be written as
$$
Q (\boldsymbol {W}; \boldsymbol {s}, a) = \frac {\mathbf {1} ^ {\top}}{K} \phi \left(\boldsymbol {w} _ {L, k} ^ {\top} \boldsymbol {h} ^ {(L)}\right) = \frac {\mathbf {1} ^ {\top}}{K} \phi \left(\boldsymbol {W} _ {L} ^ {\top} \phi \left(\boldsymbol {W} _ {L - 1} ^ {\top} \boldsymbol {h} ^ {(L - 1)}\right)\right), \tag {41}
$$
where  $\pmb{w}_{\ell,k}$  denotes the  $k$ -th neuron weights in the  $\ell$ -th layer. Then, we define a group of functions  $\mathcal{J}_{\ell}(\pmb{W}) \in \mathbb{R}^n \longrightarrow \mathbb{R}^K$  such that
$$
\mathcal {J} _ {\ell} (\boldsymbol {W}) = \left\{ \begin{array}{l} \left[ \mathbf {1} ^ {\top} \phi^ {\prime} \left(\boldsymbol {W} _ {L} ^ {\top} \boldsymbol {h} ^ {(L)}\right) \boldsymbol {W} _ {L} ^ {\top} \cdot \phi^ {\prime} \left(\boldsymbol {W} _ {L - 1} ^ {\top} \boldsymbol {h} ^ {(L - 1)}\right) \boldsymbol {W} _ {L - 1} ^ {\top} \dots \phi^ {\prime} \left(\boldsymbol {W} _ {\ell + 1} ^ {\top} \boldsymbol {h} ^ {(\ell + 1)}\right) \boldsymbol {W} _ {\ell + 1} ^ {\top} \right] ^ {\top} \\ \mathbf {1} \quad \text {i f} \quad \ell = 1. \end{array} \right. \tag {42}
$$
Then, the gradient of  $Q$  can be represented as
$$
\frac {\partial Q}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}) = \frac {1}{K} \mathcal {J} _ {\ell , k} (\boldsymbol {W}) \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , k} ^ {\top} \boldsymbol {h} ^ {(\ell)} (\boldsymbol {W})\right) \boldsymbol {h} ^ {(\ell)} (\boldsymbol {W}), \tag {43}
$$
where  $\mathcal{J}_{\ell,k}$  stands for the  $k$ -th component of  $\mathcal{J}_{\ell}$ .
A.4 Notations for Order-wise Analysis
Without loss of generality, we consider the case that  $d \gg K$ . If  $K \gg d$ , we can always switch the order of  $K$  and  $d$  in the proof. Let  $\sigma_{i}(L)$  denote the  $i$ -th largest singular value of  $W_{L}^{\star}$ . In this paper, we consider the case that  $W_{L}^{\star}$  is will-conditioned and bounded, i.e.,  $\sigma_{1}(L)$  and  $\sigma_{1}(L) / \sigma_{K}(L)$  can be viewed as the constant and will be ignored in the analysis. In addition, some constant numbers will be ignored in most steps. In particular, we use  $h_1(z) \gtrsim (\text{or} \lesssim, \simeq) h_2(z)$  to denote there exists some positive constant  $C$  such that  $h_1(z) \geq (\text{or} \leq, =) C \cdot h_2(z)$  when  $z \in \mathbb{R}$  is sufficiently large.
B Proof of Lemma 1 and Theorem 1
The main idea in proving Theorem 1 is to characterize the gradient descent term by the Mean Value Theorem (MVT) in Lemma 4 as shown in (47) and (48). The MVT is not directly applied in  $g_{t}$  because it is not smooth. However, the population risk functions defined in (31), which are the expectations over random variables, are smooth. Lemma 2 characterizes the bounds of the Hessian matrix defined in (49). Lemma 3 characterizes the bounds of gradient differences between the population risk function defined in (31) and  $g_{t}$  in (7) as shown in (60). Furthermore, according to Lemma 3, we know that the distance  $\| \nabla_{\ell}f(\boldsymbol{W}) - \nabla_{\ell}f(\boldsymbol{W}^{*})\|_{2}$  is upper bounded in the order of  $\| \boldsymbol{W} - \boldsymbol{W}^{*}\|_{2}$  as shown in (60). Then, we can establish the connection between  $\| \boldsymbol{W}^{(t,m + 1)} - \boldsymbol{W}^{*}\|_{2}$  and  $\| \boldsymbol{W}^{(t,m)} - \boldsymbol{W}^{*}\|_{2}$  as shown in (59). Then, by mathematical induction over  $m$ , one can characterize the iteration of  $\{\| \boldsymbol{W}^{(t,0)} - \boldsymbol{W}^{*}\|_{2}\}_{t = 1}^{T}$  as shown in (65), which completes the proof of Lemma 1. Finally, selecting  $\varepsilon_{t}$  based on (68) for all  $t\in [T]$ , we derive the error bound of  $\| \boldsymbol{W}^{(T,0)} - \boldsymbol{W}^{\star}\|_{2}$  by mathematical induction over  $t$ , which completes the proof of Theorem 1.
Lemma 2. Given any  $\mathbf{W} \in \mathbb{R}^n$ , let  $\mathbf{W}$  satisfy
$$
\left\| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \right\| _ {2} \lesssim \frac {\rho \cdot c _ {I} \cdot \sigma_ {K}}{K} \tag {44}
$$
for some constant  $c_{I}\in (0,1)$ . Then, for the  $f$  defined in (31), we have
$$
\frac {\left(1 - c _ {I}\right) \rho}{K ^ {2}} \preceq \nabla_ {\ell} ^ {2} f (\boldsymbol {W}) \preceq \frac {7}{K}. \tag {45}
$$
Lemma 3. Let  $f$  be the function defined in (31). Let  $g_{t}$  be the function defined in (7). Then, we have
$$
\begin{array}{l} \| \nabla_ {\ell} f (\boldsymbol {W}) - g _ {t} (\boldsymbol {W} _ {\ell}; \boldsymbol {W}) \| _ {2} \lesssim \frac {2 - \varepsilon_ {t}}{K} \sqrt {\frac {K _ {\ell} \cdot \log q}{N}} \cdot \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \\ + \frac {(1 - \varepsilon_ {t} / 2) \cdot \gamma}{K} \cdot \| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \| _ {2} \tag {46} \\ + C _ {d} \cdot \left(C _ {t} + (1 - C _ {t}) \varepsilon\right) \cdot \frac {R _ {\max}}{1 - \gamma}. \\ \end{array}
$$
with probability at least  $1 - q^{-K_{\ell}}$
Lemma 4 (Mean Value Theorem). Let  $\mathbf{U} \subset \mathbb{R}^{d_1}$  be open and  $\mathbf{f}: \mathbf{U} \longrightarrow \mathbb{R}^{d_2}$  be continuously differentiable, and  $\mathbf{x} \in \mathbf{U}$ ,  $\mathbf{h} \in \mathbb{R}^{d_1}$  vectors such that the line segment  $\mathbf{x} + t\mathbf{h}$ ,  $0 \leq t \leq 1$  remains in  $\mathbf{U}$ . Then we have:
$$
\boldsymbol {f} (\boldsymbol {x} + \boldsymbol {h}) - \boldsymbol {f} (\boldsymbol {x}) = \left(\int_ {0} ^ {1} \nabla \boldsymbol {f} (\boldsymbol {x} + t \boldsymbol {h}) d t\right) \cdot \boldsymbol {h},
$$
where  $\nabla f$  denotes the Jacobian matrix of  $f$ .
Proof of Theorem 1. Let  $W_{\ell}$  denote the neuron weights in the  $\ell$ -th layer. From Algorithm 1 and (38), in the  $s$ -th iteration and  $t$ -th episode, we have
$$
\begin{array}{l} \boldsymbol {W} _ {\ell} ^ {(t, m + 1)} = \boldsymbol {W} _ {\ell} ^ {(t, m)} - \eta g _ {t} ^ {(m)} (\boldsymbol {W} _ {\ell}; \boldsymbol {W} ^ {(t, m)}) + \beta (\boldsymbol {W} _ {\ell} ^ {(t, m)} - \boldsymbol {W} _ {\ell} ^ {(t, m - 1)}) \\ = \boldsymbol {W} _ {\ell} ^ {(t, m)} - \eta \nabla_ {\ell} f (\boldsymbol {W} ^ {(t, m)}) + \beta \left(\boldsymbol {W} _ {\ell} ^ {(t, m)} - \boldsymbol {W} _ {\ell} ^ {(t, m - 1)}\right) \tag {47} \\ + \eta \cdot \left(\nabla_ {\ell} f (\boldsymbol {W} ^ {(t, m)}) - g _ {t} ^ {(m)} (\boldsymbol {W} _ {\ell}; \boldsymbol {W} ^ {(t, m)})\right). \\ \end{array}
$$
From (31), we can see that  $\pmb{W}^{\star}$  is the global optimal to  $f$  because  $f(\pmb{W}^{\star})$  achieves the minimum value as 0. Therefore, we have  $\nabla_{\ell}f_t(\pmb{W}^{\star}) = \mathbf{0}$ . Since  $\nabla_{\ell}f$  is a smooth function  $\pmb{W}^{\star}$ , from the Mean Value Theorem in Lemma 4, we have