# Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode
## 5 Computational Experiments on Benchmark Data
### Experiment for Additive Gaussian Model  
The Mackey-Glass time series prediction is a highly nonlinear regression task, which is widely used as a benchmark problem. The chaotic time series data is generated from the difference equation of the form,  
\[\mathbf{x}(\tau+1)=(1-b)\mathbf{x}(\tau)+a\frac{x(\tau-\Delta\tau)}{1+\mathbf{x}(\tau- \Delta\tau)^{10}}, \tag{24}\]  
where we set \(a=0.2,\ b=0.1\), and \(\Delta\tau=30\) in our experiment. Figure 1 shows some part of a generated time series. The input of the neural networks are set as four previous examples in the series, \(\mathbf{x}(\tau)\), \(\mathbf{x}(\tau+1)\), \(\mathbf{x}(\tau+2)\), \(\mathbf{x}(\tau+3)\), and the target output is given as the next future value, \(\mathbf{x}(\tau+4)\). From the generated time series, we chose 10,000 training samples (\(\mathbf{x}(1001)\),..., x (11,000)) and 1000 test samples (\(\mathbf{x}(11,101)\),..., \(x\) (12,100)).  
The network structure for training is the multilayer Perceptrons with 4 inputs, 10 hidden, and 1 output, which is the same as the model used in [21]. Since the output has continuous real values, we adopted the additive Gaussian noise model. Other user-defined parameters for five learning methods are shown in Table 1. Note that we used a larger mini-batch size for SFO in order to ensure its comparable execution time.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & & & \multicolumn{3}{c}{**Learning Methods**} \\ \cline{2-6}  & \begin{tabular}{c} **SG** \\ **(Mini-Batch)** \\ \end{tabular} & \
\begin{tabular}{c} **ANG** \\ **(On-Line)** \\ \end{tabular} &
\begin{tabular}{c} **Proposed ANG** \\ **(Mini-Batch)** \\ \end{tabular} & **ADAM** & **SFO** \\ \hline \(|D_{\lambda}|\,K\) & \(10,1000\) & \(1,\,10,000\) & \(10,1000\) & \(10,1000\) & \(100,100\) \\ \hline \multirow{3}{*}{hyper-parameters} & \multirow{3}{*}{\(\eta=0.01\)} & \(\eta=0.1\) & \(\alpha=0.001\) & \multirow{3}{*}{\(-\)} \\  & & \(\lambda=0.0164\) & & & \\  & & \(\epsilon_{t}=1/t\) & & & \\ \hline execution time per epoch (s) & \(0.0191\) & \(0.4336\) & \(0.0716\) & \(0.0340\) & \(0.2590\) \\ \hline squared error (training data) & \(9.77\times 10^{-6}\) & \(2.13\times 10^{-6}\) & \(9.27\times 10^{-7}\) & \(2.76\times 10^{-6}\) & \(1.03\times 10^{-5}\) \\ \hline squared error (test data) & \(1.01\times 10^{-5}\) & \(2.13\times 10^{-6}\) & \(9.08\times 10^{-7}\) & \(2.76\times 10^{-6}\) & \(1.03\times 10^{-5}\) \\ \hline total number of epochs & \(10^{4}\) & \(10^{4}\) & \(10^{4}\) & \(10^{4}\) & \(10^{4}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Hyper parameters and training performances on Mackey-Glass time series data.  
Figure 1: Mackey-Glass time series data (test samples).  
In Figure 2, the changes of training errors of three learning methods according to learning epochs are shown. From the figure, we can see that two adaptive natural gradient methods show better convergence properties than the other three methods. In addition, the proposed mini-batch method gets smaller training error than the conventional online adaptive natural gradient method. Adam optimizer achieved slightly higher training error than the online ANG, but required about 10 times more iterative updates. SFO showed very fast convergence in the early learning stage, but seemed to be trapped at a local minima, giving undesirable solution.  
In Table 1, the computational efficiency of three methods are also compared by using the average execution time per single epoch, which shows large differences among the methods. In order to compare computational efficiency and convergence performance at the same time, we plotted the change of errors depending on the execution time (see Figure 3). Note that the horizontal axis of Figure 3 is the execution time in log scale. Form the figure, the superiority of the proposed mini-batch ANG method is confirmed more clearly.  
Figure 3: Change of prediction errors for Mackey-Glass series depending on the execution time.  
Figure 2: Change of prediction errors for Mackey-Glass series depending on the number of epochs.