# Neural Architecture Search for a Highly Efficient Network with Random Skip Connections
## 4 Experimental Results
### Classification Using MNIST and Permuted MNIST  
In this section, we evaluate the methods considered on two publicly available datasets: the MNIST and the permuted MNIST (pMNIST) [17]. The MNIST comprises 55,000 images for training, 5000 for verification, and 10,000 for testing, where each of them is flattened from a size of \(28\times 28\) into 784-dimensional. The pMNIST is generated by permuting the image pixels in the MNIST and intends to create longer-term dependencies [17]. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **RNN** & **sLSTM** & **JANET** & **Skip GRU** & **Ours** \\ \hline \multirow{4}{*}{MSE} & \(4.37\times 10^{-4}\) & \(6.43\times 10^{-5}\) & \(6.03\times 10^{-5}\) & \(1.11\times 10^{-4}\) & \(3.44\times 10^{-5}\) \\ \cline{2-6}  & \(8.55\times 10^{-2}\) & \(9.03\times 10^{-5}\) & \(3.91\times 10^{-5}\) & \(4.76\times 10^{-3}\) & \(2.83\times 10^{-5}\) \\ \cline{2-6}  & \(3.40\times 10^{-3}\) & \(1.23\times 10^{-4}\) & \(3.12\times 10^{-5}\) & \(4.44\times 10^{-4}\) & \(2.77\times 10^{-5}\) \\ \cline{2-6}  & \(4.79\times 10^{-4}\) & \(1.90\times 10^{-4}\) & \(6.64\times 10^{-5}\) & \(3.14\times 10^{-4}\) & \(4.93\times 10^{-5}\) \\ \cline{2-6}  & \(2.62\times 10^{-3}\) & \(2.41\times 10^{-5}\) & \(4.97\times 10^{-5}\) & \(5.20\times 10^{-4}\) & \(4.08\times 10^{-5}\) \\ \hline Mean & \(1.85\times 10^{-2}\) & \(9.83\times 10^{-5}\) & \(4.93\times 10^{-5}\) & \(1.23\times 10^{-3}\) & \(3.61\times 10^{-5}\) \\ \hline Std & \(3.75\times 10^{-2}\) & \(6.27\times 10^{-5}\) & \(1.45\times 10^{-5}\) & \(1.98\times 10^{-3}\) & \(9.09\times 10^{-6}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The experimental results of the adding task. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model Task** & **Copy (len = 120)** & **Frequency (len = 120)** \\ \hline RNN & 0.1475 & 86.4\% \\ \hline sLSTM & 0.1263 & 81.2\% \\ \hline JANET & 0.0886 & 89.8\% \\ \hline Skip GRU & 0.1147 & 90.4\% \\ \hline Ours & 0.0829 & 91.2\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Experimental results of the copy and frequency discrimination tasks. The means and standard deviations of the testing accuracy are presented in Table 3, where the values are displayed in percentage form. We produced the results of our model and the results of Skip GRU on the datasets over five independent runs, and following the experimental setup in JANET [4], we utilized two hidden layers with 128 units for the MNIST and a single layer with the same units for the pMNIST. Then, as for the other results, we cited the presented results directly and the total number of layers or runs have been declared in their works; for instance, RNN, sLSTM, and JANET all utilized two layers on the MNIST and performed ten runs independently. From Table 3, we can see that our model achieved the best accuracy and had the most stable performance due to its minimum standard deviations. The test accuracies on two datasets over the epochs of training are depicted in Figure 2a,b. We executed the programs of all the comparative methods and obtained the average results over five runs. Regarding the results using the MNIST set, which are presented in Figure 2a, all the methods behaved well, except for the RNN, and the Skip GRU converged the quickest in the beginning stage, whereas our method eventually achieved the best accuracy. Figure 2b depicts the accuracy curves for the methods used on pMNIST, which was more challenging than the MNIST; all the methods had a slower ascent to reaching their highest accuracies. Our method obtained the best accuracy compared with others, although its ascent curve was not the steepest. We also present the curves of different runs in Figure 2c, from which we can see that the curves differed from each other in the ascent stage but reached similar final maximum values. The sampled skip connections in our network partly contributed to this phenomenon as proper connections could accelerate the network's convergence. Figure 2: The change in test accuracy with training epochs.