# Assessing Efficiency in Artificial Neural Networks
## 4 Results
### Batch Normalization and Dropout as Neural Efficiency Modifiers
#### 4.2.2 Dropout  
Dropout had different effects on the LeNet-300-100 and LeNet-5 architectures. Dropout generally decreased \(\eta_{N}\) in LeNet-300-100 networks and increased \(\eta_{N}\) in the LeNet-5 networks, but decreased aIQ in nearly all networks due to a drop in accuracy for nearly all networks (Table 2). For LeNet-300-100 networks, none of the networks trained with dropout had an accuracy higher than their corresponding networks trained without dropout, and only 18 (5.59%) of the networks had a higher efficiency. Accordingly, no networks with dropout had a higher aIQ relative to their corresponding networks without dropout. None of the networks with dropout had an accuracy higher than their corresponding networks without dropout, and 1818 (60.60%) of the dropout networks had a higher \(\eta_{N}\). When considering neural networks with 16 or fewer neurons in each layer, 191 (99.33%) of the networks with dropout had a higher \(\eta_{N}\) than their corresponding networks without dropout. However, only 468 (15.60%) of the dropout networks had a higher aIQ than their corresponding networks without dropout, meaning the increase in efficiency was not sufficient to offset the decrease in accuracy in the majority of networks. The general conclusion from these results is that dropout generally decreases accuracy, leading to a drop in aIQ. The reason why accuracy dropped in all networks may be due to a variety of factors, including the dropout rate being too high (\(p=50\%\)) or dropout being placed in every layer of the network. The explanation for why dropout appeared \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{3}{c}{LeNet-300-100 (\(N_{l}\))} \\ \hline Layer 1 & Layer 2 & Modifier & Test Accuracy (\%) \({}^{\ddagger}\) & aIQ (\(\beta=2\)) \({}^{\ddagger}\) & \(\eta_{N}\) (\%) \\ \hline
11 & 4 & **None** & 92.91 \(\pm\) 0.19 & **86.41 \(\pm\) 0.71** & **74.77 \(\pm\) 1.72** \\  & & BatchNorm & **93.50 \(\pm\) 0.36** & 82.10 \(\pm\) 1.44 & 63.33 \(\pm\) 3.07 \\  & & Dropout & 78.41 \(\pm\) 3.82 & 74.36 \(\pm\) 4.06 & 66.89 \(\pm\) 4.47 \\
10 & 6 & None & 93.52 \(\pm\) 0.14 & 83.16 \(\pm\) 2.52 & 66.20 \(\pm\) 5.75 \\  & & **BatchNormNorm** & **93.65 \(\pm\) 0.44** & **87.76 \(\pm\) 0.44** & **77.07 \(\pm\) 1.16** \\  & & Dropout & 83.63 \(\pm\) 0.63 & 75.62 \(\pm\) 0.93 & 61.83 \(\pm\) 1.49 \\
7 & 5 & None & 91.53 \(\pm\) 0.31 & 83.88 \(\pm\) 1.75 & 70.65 \(\pm\) 4.21 \\  & & BatchNorm & **91.91 \(\pm\) 0.25** & **86.42 \(\pm\) 1.20** & **76.43 \(\pm\) 2.76** \\  & & **Dropout** & 80.63 \(\pm\) 2.96 & 78.31 \(\pm\) 2.44 & 73.88 \(\pm\) 1.47 \\ \hline \multicolumn{3}{c}{LeNet-5 (\(N_{l}\))} \\ \hline Layer 1 & Layer 2 & Layer 3 & Modifier & Accuracy (\%) \({}^{\ddagger}\) & aIQ (\(\beta=2\)) \({}^{\ddagger}\) & \(\eta_{N}\) (\%) \\ \hline
3 & 9 & 4 & **None** & 96.84 \(\pm\) 0.18 & **88.28 \(\pm\) 0.46** & 73.38 \(\pm\) 1.22 \\  & & & BatchNorm & **96.95 \(\pm\) 0.58** & 82.44 \(\pm\) 3.08 & 59.73 \(\pm\) 6.47 \\  & & & Dropout & 79.72 \(\pm\) 4.96 & 77.82 \(\pm\) 2.91 & **74.23 \(\pm\) 0.99** \\
2 & 8 & 8 & None & 97.83 \(\pm\) 0.10 & 85.58 \(\pm\) 2.28 & 65.81 \(\pm\) 5.03 \\  & & & **BatchNormNorm** & **98.05 \(\pm\) 0.20** & **88.12 \(\pm\) 1.68** & 71.21 \(\pm\) 4.30 \\  & & & Dropout & 88.06 \(\pm\) 2.00 & 82.75 \(\pm\) 0.38 & **73.16 \(\pm\) 4.21** \\
3 & 4 & 7 & None & 97.17 \(\pm\) 0.22 & 83.71 \(\pm\) 2.65 & 62.55 \(\pm\) 5.74 \\  & & & BatchNorm & **97.27 \(\pm\) 0.34** & 84.91 \(\pm\) 2.16 & 64.76 \(\pm\) 4.45 \\  & & & **Dropout** & 90.15 \(\pm\) 1.66 & **85.45 \(\pm\) 1.47** & **76.82 \(\pm\) 3.32** \\ \hline \hline \end{tabular} \(\dagger\) Accuracy, aIQ, and efficiency values are mean \(\pm\)95% CI (n = 11 for None, n = 3 otherwise). aIQ values are \(\times\)100. Data shown are for metrics calculated on the test dataset.  
\end{table} 
Table 2: The top network architectures by aIQ when batch normalization (BatchNormNorm), dropout (Dropout), or neither (None) are added to every layer.