# Assessing Efficiency in Artificial Neural Networks
## 5 Discussion  
The major contribution of this work is the establishment of an assessment technique called neuron layer state space and the capacity to use state space as a means of evaluating neuron layer utilization using the metrics of neural efficiency and aIQ. Quantizing neurons into on/off positions that encode information collectively as a neural layer state provides a different perspective on how data are processed in the network. One prevailing thought is that neurons are discrete units that encode individual features, but the work here suggests that the information an individual neuron encodes may have value in the context of the other neurons with which it fires. One advantage of conceptualizing the flow of information between layers using state space is the number of tools that become available for network analysis. In this work, a rudimentary metric was created to understand layer utilization, but many other methods of analyzing state space could be used, such as the relative entropy of a single layer or the mutual information between two layers. Further investigation of state space may help to further compress the size of the network without a significant decrease in accuracy, and may even permit the training of the number of neurons in a layer. While neural architecture search has become a topic of interest in order to search for an ideal network computational cell, few if any of these methods include parameters to learn layer sizes. One benefit of understanding neural networks in terms of state space is that guidelines on how many neurons to place in a layer can be established knowing only superficial information about the training data. The current thought is that a higher number of neurons leads to higher accuracy, but this does not appear to provide an improved generalization for the small convolutional models tested here (see Table 4). Using the concept of state space, the number of states in a dense layer cannot exceed the number of input images. If there are X training examples, then ceil(log2X) neurons are sufficient to memorize every training image. As an example, the ImageNet dataset has 14 million images (leading to \(\sim\)2\({}^{24}\) states), which is considerably smaller than the available statespace of AlexNet (4096 densely connected neurons), which is \(\sim\)6 \(\times\) 10\({}^{1225}\) times larger than the number of available training examples in ImageNet [27]. Due to random initialization, it is doubtful \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{3}{c}{LeNet-300-100 (\(N_{l}\))} & \multicolumn{3}{c}{MNIST Accuracy (\%)} & EMNIST Accuracy (\%) \\ \hline Layer 1 & Layer 2 & 0\% Rand & 75\% Rand & 0\% Rand & 75\% Rand \\ \hline
1024 & 1024 & 98.03 & 36.99 & 76.21 & 24.93 \\
10 & 6 & 93.65 & 84.68 & 58.43 & 55.46 \\ \hline \multicolumn{3}{c}{LeNet-5 (\(N_{l}\))} & & & \\ \hline Layer 1 & Layer 2 & Layer 3 & 0\% Rand & 75\% Rand & 0\% Rand & 75\% Rand \\ \hline
1024 & 1024 & 1024 & 99.11 & 44.24 & 90.56 & 34.16 \\
2 & 8 & 8 & 98.05 & 92.51 & 89.36 & 74.67 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Generalization tests on the networks with the highest aIQ or accuracy. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multicolumn{3}{c}{LeNet-300-100 (\(N_{l}\))} & \multicolumn{3}{c}{Accuracy (\%)} \\ \hline Layer 1 & Layer 2 & 0\% Rand & 25\% Rand & 50\% Rand & 75\% Rand & 100\% Rand \\ \hline
1024 & 1024 & 98.03 & 85.91 & 63.43 & 36.99 & 9.48 \\
10 & 6 & 93.65 & 91.56 & 89.59 & 84.68 & 10.89 \\ \hline \multicolumn{3}{c}{LeNet-5 (\(N_{l}\))} & & & & & \\ \hline Layer 1 & Layer 2 & Layer 3 & 0\% Rand & 25\% Rand & 50\% Rand & 75\% Rand & 100\% Rand \\ \hline
1024 & 1024 & 1024 & 99.11 & 93.99 & 76.07 & 44.24 & 10.34 \\
2 & 8 & 8 & 98.05 & 96.73 & 95.69 & 92.51 & 7.49 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Memorization tests on the networks with the highest aIQ or accuracy. that a dense layer would memorize every training image if it contains exactly the number of neurons required to memorize all inputs.