# Assessing Efficiency in Artificial Neural Networks
## 5 Discussion
However, assuming the network is generating general rules for classification, the entire bandwidth of the channel should never need to be used. An analogous guideline could be applied to convolutional layers, where all combinations of pixel intensities of an 8-bit grayscale image in a 3 \(\times\) 3 grid could be perfectly represented by 72 neurons with 3 \(\times\) 3 convolutional kernels (256\({}^{9}=2^{72}\)), so the first convolutional layer of a network should never contain more than 72 neurons when analyzing 8-bit grayscale images. Thus, simple upper limits on the number of neurons in different layers could be inferred from the implications of state space, where the upper limits may be considerably smaller than the number of neurons that are currently observed in modern networks. Using the guidelines laid out above, it is reasonable to say that most networks that are created contain many more neurons than needed, explaining why most neural networks are prone to memorization and attack vectors. The authors of [PERSON] et al. [28] demonstrated the existence of single directions in neural networks, and although they found that models trained on randomized labels did not generalize, they did find that single directions were significantly more important in models with randomized labels for training data accuracy [28]. In the present work, we constrained the state space by reducing the number of neurons in a layer. If memorization is the construction of unique states within state space, then constraining the number of available states in a layer should reduce or even prevent memorization by limiting the number of potential single directions. For attack vectors, adversarial networks can be trained to add imperceptable amounts of noise to an image to cause the neural network to misclassify the image [29]. It is plausible that these attack vectors take advantage of noise in an over-parameterized network, a problem that a smaller neural network may not face. Thus, the use of state space and neural efficiency may help to make networks more resistant to such attacks. Related to the size complexity of current neural networks is the concept of Occam's razor in machine learning. The general concept behind Occam's razor is that the simplest model should be preferred, including in artificial intelligence [30]. Geometric analysis of the parameter space of neural networks reveals low-dimensional manifolds, suggesting that, despite the vast number of parameters in most modern neural networks, they have relatively simple parameter spaces [31]. Thus, the work presented here evaluating efficiency could be interpreted as a metric that assesses model simplicity. There are a few deficits in the current approach that should be resolved in future work. The first is the memory utilization and computational complexity of capturing state space. Only small networks were analyzed in this work due to the potential size of state space, which may be unmanageable in a large neural network assessing a large dataset such as ImageNet. The second problem is the issue of class imbalances. If there are class imbalances in either the training or test data, it would be expected that the efficiency metrics would be skewed. Class imbalances in the training data might cause more neurons in the network to be dedicated to classification of the most common class, while imbalances in the test data might overrepresent states that occur. Another issue is the underlying assumption that maximum efficiency is achieved when all states occur at the same frequency. It might be expected that some states occur far more frequently than others so that an ideal distribution of states might look more like an exponential distribution. This was superficially confirmed by looking at the distribution of states in the LeNet-300-100 networks, by analyzing networks with \(N_{1}=8\) around the local minimum when \(N_{2}=16\) (see Figure 1). Therefore, some other metric of calculating efficiency that accounts for an ideal distribution of states might be a better measure of efficiency.