# Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode
## 5 Computational Experiments on Benchmark Data
### Experiment for Logistic Regression Model  
In order to verify performance of the proposed method for the logistic model, we conducted experiments on the well-known spiral classification data. The task is to classify two dimensional input data scattered in spiral shapes into two classes. Though the network structure was small, it had a highly nonlinear decision boundary as shown in Figure 4, and thus is often used as benchmark data for classifiers. For training, we generated 100,000 samples for each class, (i.e., totally 200,000 samples) as plotted in Figure 4a, and an additional 1000 samples per class for testing (Figure 4b). Whereas the training data can be clearly discriminated by a spiral-shaped decision boundary, there was some overlap in test data, which may cause misclassification.  
The network structure is set to 2 inputs, 20 hidden, 2 binary outputs. Other user-defined parameters are given in Table 2. The shape of five learning curves for cross-entropy error depending on the number of epochs is shown in Figure 5. As shown in the figure, the proposed method shows fast convergence compared to the other methods. It should be also remarked that there are large differences in the average execution times for single epoch. Since the online natural gradient need to calculate the natural gradient for each data sample, it takes about 30 times longer execution time than the proposed mini-batch adaptive natural gradient methods. Considering this difference, we also compared the learning curves depending on the execution time, as shown in Figure 6. From the figure, we can confirm the efficiency of the proposed method again. We can also see that Adam optimizer had similar convergence speed to the proposed one in this specific computational environment with a single CPU. Figure 7 shows the decision boundaries that were obtained by three learning methods.  
\begin{table}
\begin{tabular}{c c c c c c} \hline  & \multicolumn{5}{c}{**Learning Methods**} \\ \cline{2-6}  & \begin{tabular}{c} **SG** \\ **(Mini-Batch)** \\ \end{tabular} & \begin{tabular}{c} **ANG** \\ **(On-Line)** \\ \end{tabular} & \begin{tabular}{c} **Proposed ANG** \\ **(Mini-Batch)** \\ \end{tabular} & \
\begin{tabular}{c} **ADAM** \\ \end{tabular} &
\begin{tabular}{c} **SFO** \\ **(500, 4000)** \\ \end{tabular} \\ \hline \(|D_{\text{L}}|,K\) & 50, 4000 & 1, 200,000 & 50, 4000 & 50, 4000 & 500, 4000 \\ \hline \multirow{3}{*}{Hyper-parameters} & \multirow{3}{*}{\(\eta=0.0001\)} & \(\eta=0.001\) & \(\eta=0.0001\) & \(\eta=0.0001\) & \(\beta_{1}=0.9\) & \multirow{3}{*}{\(-\)} \\  & & \(\alpha=0.0066\) & \(\lambda=0.0066\) & \(\beta_{2}=0.999\) & & \\ \cline{1-1}  & & \(\alpha=1/t\) & & \(\epsilon=10^{-8}\) & & \\ \hline execution time per epoch (s) & 0.1625 & 17.1659 & 0.6504 & 0.1867 & 0.9021 \\ \hline cross entropy loss (training) & 0.0060 & 0.0048 & 0.0018 & 0.00016 & 0.05776 \\ \hline cross entropy loss (test) & 0.2743 & 0.558 & 0.3562 & 0.6369 & 0.2391 \\ \hline classification rates (\%, test) & 91.60 & 90.20 & 91.80 & 91.40 & 90.20 \\ \hline total number of epochs & \(10^{3}\) & \(3\times 10^{2}\) & \(3\times 10^{2}\) & \(10^{3}\) & \(10^{3}\) \\ \hline \end{tabular}
\end{table}
Table 2: Hyper parameters and training performances on spiral data classification.  
Figure 4: Spiral data sets: (**a**) Training data samples and (**b**) test data samples.