# Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode
## 2 Gradient Descent Learning of Stochastic Neural Networks
### Gradient Descent Learning  
Once a specific model of stochastic neural networks and its corresponding loss function are determined, the weight parameters are optimized by gradient descent method. The well-known error-backpropagation algorithm is the standard type of gradient descent learning method. There have been numerous variations of the standard gradient descent method, including second-order methods, momentum method, and adaptive learning rate methods [23; 24]. Since the natural gradient learning method is also based on the gradient descent method, we describe the basic formula of gradient descent learning and its online version that is called stochastic gradient descent method.  
When a set of training data \(D=\left\{\left(\mathbf{x}_{i},\mathbf{y}_{i}\right)\right\}_{i=1..N}\) is given, a neural network is trained in order to find an input-output mapping \(\mathbf{f}(\mathbf{x},\mathbf{\theta})\) that is specified with weight parameter vector \(\mathbf{\theta}\). The error of neural network for the whole data set \(D\) can then be defined by using a loss function such as  
\[E(D,\mathbf{\theta})=\sum\
olimits_{i=1}^{N}l\left(\mathbf{x}_{i},\mathbf{y}_{i},\mathbf{ \theta}\right), \tag{7}\]and the goal of learning is to get the optimal \(\mathbf{\theta}\) minimizing \(E(D,\mathbf{\theta})\). To achieve the goal, the weight parameter is updated starting from the current position \(\mathbf{\theta}_{t}\), according to the opposite direction of the gradient of \(E(D,\ \mathbf{\theta})\), which can be written as  
\[\mathbf{\theta}_{t+1}=\mathbf{\theta}_{t}-\eta\frac{\partial E(D,\mathbf{\theta}_{t})}{ \partial\mathbf{\theta}}. \tag{8}\]  
This update rule is called batch learning mode, meaning that an update is done for the whole batch set \(D\). Theoretically, the batch mode learning gives the steepest descent direction of \(E(D,\mathbf{\theta}_{t})\) at the current position of \(\mathbf{\theta}_{t}\), but it has two practical drawbacks. First, it is too stable to be easily trapped in undesirable local minima. In addition, when the number of data \(N\) is large, it needs large amounts of computation for just a single update, making the learning process slow.  
To overcome this practical inefficiency, online stochastic gradient decent learning is proposed [23], in which parameters are updated for each data sample by using gradient of loss function \(l(\mathbf{x},\mathbf{y},\mathbf{\theta})\) defined with a single data pair \((\mathbf{x},\mathbf{y})\), such as  
\[\mathbf{\theta}_{t+1}=\mathbf{\theta}_{t}-\eta\frac{\partial l(\mathbf{x},\mathbf{y},\mathbf{ \theta}_{t})}{\partial\mathbf{\theta}}. \tag{9}\]  
Compared to the batch mode learning, the update direction of online learning mode is theoretically incorrect because it does not calculate the gradient of total error \(E(D,\mathbf{\theta})\) and has some stochastic randomness due to the online sample \((\mathbf{x},\mathbf{y})\) at each update. Nevertheless, it can be a good practical alternative, because its convergence properties have been well-studied and the stochastic uncertainty can make the possibility of jumping over shallow undesirable local minima [23].