# Assessing Efficiency in Artificial Neural Networks
## 4 Results
### Batch Normalization and Dropout as Neural Efficiency Modifiers  
#### 4.2.1 Batch Normalization  
Batch normalization generally increased the accuracy and \(\eta_{N}\) for both LeNet-300-100 and LeNet-5 networks, resulting in a rise in alQ for most networks (Table 2). For LeNet-300-100 networks, 178 (59.26%) of the networks with batch normalization had a mean \(\eta_{N}\) higher than the mean \(\eta_{N}\) of corresponding networks trained without batch normalization. When only small networks are considered (\(N_{1}\leq 16\), \(N_{2}\leq 16\)), 36 (74.13%) of the networks with batch normalization had a mean \(\eta_{N}\) higher than the same network trained without batch normalization. In addition to higher network efficiency, neural networks trained with batch normalization also had higher accuracy on test data, where 233 (77.78%) of the networks with batch normalization achieved higher accuracies than the same networks without batch normalization. alQ increased in 222 (74.07%) of the networks where all layers had 16 or fewer neurons.  
For LeNet-5 networks, 2226 (74.2%) of all networks with batch normalization had a mean \(\eta_{N}\) higher than their corresponding networks without batch normalization. However, the batch normalization generally resulted in lower accuracy, with only 978 (32.60%) of the networks achieving a higher accuracy than their corresponding networks without batch normalization. When considering only small networks where \(N_{l}\leq 16\) for all layers, 178 (92.6%) of the networks with batch normalization had a higher accuracy. Furthermore,  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{4}{c}{LeNet-300-100 (\(N_{l}\))} \\ \hline Layer 1 & Layer 2 & Test Accuracy (\%) \({}^{\ddagger}\) & alQ (\(\beta=2\)) \({}^{\ddagger}\) & Parameters (fold decrease) \\ \hline
128 & 1024 & **97.58 \(\pm\) 0.04** & 32.70 \(\pm\) 0.01 & 242,826 (1\(\times\)) \\
256 & 1024 & 97.54 \(\pm\) 0.06 & 29.12 \(\pm\) 0.01 & 474,378 (0.5\(\times\)) \\
64 & 1024 & 97.53 \(\pm\) 0.08 & 36.67 \(\pm\) 0.02 & 127,050 (1.9\(\times\)) \\
11 & 4 & 92.91 \(\pm\) 0.19 & **86.41 \(\pm\) 0.71** & 8733 (27.8\(\times\)) \\
11 & 5 & 93.59 \(\pm\) 0.29 & 85.93 \(\pm\) 1.04 & 8755 (27.7\(\times\)) \\
7 & 4 & 90.76 \(\pm\) 0.25 & 85.90 \(\pm\) 1.00 & 5577 (43.5\(\times\)) \\ \hline \hline \multicolumn{4}{c}{LeNet-5 (\(N_{l}\))} \\ \hline Layer 1 & Layer 2 & Layer 3 & Test Accuracy (\%) \({}^{\ddagger}\) & alQ (\(\beta=2\)) \({}^{\ddagger}\) & Parameters (fold decrease) \\ \hline
1024 & 1024 & 1024 & **99.16 \(\pm\) 0.02** & 24.28 \(\pm\) 0.004 & 43,030,538 (1\(\times\)) \\
1024 & 128 & 512 & 99.15 \(\pm\) 0.03 & 33.02 \(\pm\) 0.008 & 4,357,770 (9.9\(\times\)) \\
1024 & 512 & 1024 & 99.14 \(\pm\) 0.03 & 26.22 \(\pm\) 0.005 & 21,534,218 (2.0\(\times\)) \\
3 & 9 & 4 & 96.84 \(\pm\) 0.18 & **88.28 \(\pm\) 0.46** & 1392 (30,912.9\(\times\)) \\
3 & 5 & 5 & 96.72 \(\pm\) 0.16 & 88.18 \(\pm\) 1.52 & 923 (46,620.3\(\times\)) \\
3 & 4 & 4 & 95.37 \(\pm\) 0.19 & 88.02 \(\pm\) 0.64 & 692 (62,182.9\(\times\)) \\ \hline \hline \multicolumn{4}{l}{\({}^{\ddagger}\) Accuracy and alQ values are mean \(\pm\)95% CI (n = 11). alQ values are \(\times\)100. Data shown are for metrics calculated on the test dataset.} \\ \end{tabular}
\end{table}
Table 1: The top three network architectures by accuracy and alQ.  
2133 (71.10%) of all networks with batch normalization had a higher \(\eta_{N}\) relative to their corresponding networks without batch normalization, and \(\eta_{N}\) decreased to 123 (64.35%) for networks where all layers had less than 16 neurons. Overall, 2226 (74.20%) of all networks had a higher aIQ when trained with batch normalization.  
These results confirm the hypothesis that batch normalization generally acts to improve \(\eta_{N}\) in addition to improving classification accuracy (Table 2), making neural networks \"more intelligent\", as assessed by aIQ.