## 1 Introduction  
Deep learning has been highly successful across a variety of applications, including speech recognition, visual object recognition, and object detection [1, 2, 3]. In general application, deep learning consists of training and inference phases. In the training phase, a predefined network is trained on a given dataset (known as the training set) to learn the underlying distribution characteristics. In the inference phase, the well-trained network is then used for unforeseen data (known as the test set) to implement specific tasks, such as regression and classification. One fundamental purpose of deep learning is to achieve as high accuracy as possible in the inference phase after only learning from the training set. In essence, training a deep learning network is equivalent to minimizing an unconstrained non-convex but smooth function. \[\min_{w\in\mathbb{R}^{n}}f(w):=\mathbb{E}_{\xi\sim\mathcal{D}}F(w,\xi) \tag{1}\]  
where \(\mathcal{D}\) is the population distribution, \(F(w,\xi)\) is the loss function of sample \(\xi\), and \(f(w)\) is the expectation of \(F(w,\xi)\) with respect to \(\xi\). One effective method to solve problem (1)is mini-batch stochastic gradient descent (SGD) [4]. That is, in each iteration \(t\), the model parameter \(w_{t}\) is updated to \(w_{t+1}\), following  
\[w_{t+1}=w_{t}-\gamma_{t}\frac{1}{|\mathcal{B}_{t}|}\sum_{\xi\in\mathcal{B}_{t}} \
abla F(w_{t},\xi_{t}), \tag{2}\]  
where \(\gamma_{t}\) is the learning rate and \(\mathcal{B}_{t}\) is a random mini-batch of size \(b\). Here, based on three advantages discussed below, we consider using problem Equations (1) and (2) to analyze deep learning. _First_, generality. Because most deep learning networks correspond to non-convex optimization, the derived results for problem (1) can be applied to general deep learning tasks in practice. _Second_, effectiveness. Because the data scale used in deep learning is usually huge, Equation (2) can achieve a better utility-efficiency tradeoff than SGD or batch GD. _Third_, simplicity. When using Equation (2), searching for a solution to problem (1) is reduced to setting a proper learning rate \(\gamma_{t}\). Therefore, the research question is how to set a proper \(\gamma_{t}\) in Equation (2) to ensure the convergence of problem (1). Without of loss generality, we make the following assumptions. * Non-convex but smooth \(f(w)\). That is, \(f(w)\) is non-convex but satisfies \(f(w_{t+1})-f(w_{t})\leq\langle\
abla f(w_{t}),w_{t+1}-w_{t}\rangle+\frac{1}{2} \|w_{t+1}-w_{t}\|^{2}\). * Unbiased estimate and bounded variance of \(g(w)\stackrel{{\triangle}}{{=}}\
abla F(w,\xi)\). That is, \(\mathbb{E}_{\xi\sim\mathcal{D}}[g(w)]=\
abla f(w)\), and \(\mathbb{E}_{\xi\sim\mathcal{D}}[\|g(w)-\
abla f(w)\|^{2}]\leq\sigma^{2}\). It is well-known that searching for the global minima of Equation (1) is NP-hard [5]. Therefore, one usually aims to search for the first-order stationary point, the gradient of which satisfies \(\|\
abla f(w)\|<\epsilon\) where \(\epsilon\) is a given error bound. For simplicity, denote the average of gradients on mini-batch \(\mathcal{B}_{t}\) as \(\mathcal{S}_{t}=\frac{1}{|\mathcal{B}_{t}|}\sum_{\xi\in\mathcal{B}_{t}}\
abla F (w_{t},\xi_{t})\). It is well-known that SGD can converge for proper settings of \(\gamma_{t}\)[6; 7; 8; 9]. Actually, based on the above assumptions and through direct calculation, we have  
\[\sum_{t=1,\cdots,T}\biggl{(}\gamma_{t}-\frac{L\gamma_{t}^{2}}{2}\biggr{)} \mathbb{E}\|\
abla f(w_{t})\|^{2}\leq f(w_{1})-f^{*}+\frac{L\sigma^{2}\sum_{t =1}^{T}\gamma_{t}^{2}}{2b},\]  
where \(f^{*}=\min_{w\in\mathbb{R}^{n}}f(w)\) and \(L,\sigma^{2}\) are defined in the assumptions. Therefore, for any given constant \(\gamma=\min\{1/L,O(1/\sqrt{L+T})\}\), we can deduce that \(\sum_{t=1}^{T}\mathbb{E}\|\
abla f(w_{t})\|\) is bounded, which implicates that \(\|\
abla f(w_{t})\|\to 0\) as \(t\to\infty\). That is, SGD is convergent and one can output the stationary solution with high probability [6].