# Assessing Efficiency in Artificial Neural Networks
###### Abstract  
The purpose of this work was to develop an assessment technique and subsequent metrics that help in developing an understanding of the balance between network size and task performance in simple model networks. Here, exhaustive tests on simple model neural networks and datasets are used to validate both the assessment approach and the metrics derived from it. The concept of neural layer state space is introduced as a simple mechanism for understanding layer utilization, where a state is the on/off activation state of all neurons in a layer for an input. Neural efficiency is computed from state space to measure neural layer utilization, and a second metric called the artificial intelligence quotient (aIQ) was created to balance neural network performance and neural efficiency. To study aIQ and neural efficiency, two simple neural networks were trained on MNIST: a fully connected network (LeNet-300-100) and a convolutional neural network (LeNet-5). The LeNet-5 network with the highest aIQ was 2.32% less accurate but contained 30,912 times fewer parameters than the network with the highest accuracy. Both batch normalization and dropout layers were found to increase neural efficiency. Finally, networks with a high aIQ are shown to be resistant to memorization and overtraining as well as capable of learning proper digit classification with an accuracy of 92.51%, even when 75% of the class labels are randomized. These results demonstrate the utility of aIQ and neural efficiency as metrics for determining the performance and size of a small network using exemplar data. neural networks; intelligence; neural efficiency; entropy; metrics +
Footnote †: journal: Medical  
been shown that modern networks are highly capable of memorizing randomized image labels [6]. To combat overtraining and memorization, a variety of techniques and layers are used, including data augmentation [7], dropout layers [8], and even the creation of new neural networks in the case of adversarial neural network training [9]. Thus, the drive to create larger ANNs is often compounded by further increases in ANN size by using architecture components designed to resolve the problems of large networks, which make the deployment of these networks at scale a challenge [10]. The current state of ANN research has an analogous mindset to the human intelligence research prior to the formulation of the neural efficiency hypothesis in the 1990s [11]. The general opinion in neuroscience research was that an individual with high intelligence was more capable of performing tasks with high performance because their brain would be more capable of recruiting large numbers of neurons. This changed with publications that showed fewer neurons fire for a given task in higher performing individuals [PERSON] et al. [12; 13]. One publication showed that individuals with high intelligence had higher scores on Tetris but had lower brain metabolism while playing the game [11]. This finding led to the formulation of the neural efficiency hypothesis, which states that a key factor to intelligence is the capacity of the brain to perform a task by using the smallest amount of neural activity [14]. In the above context of human intelligence and neural efficiency, the current drive to increase neural network task accuracy by increasing the size and complexity of a network may be interpreted as the development of less intelligent neural networks. This is supported anecdotally in the literature by the tendency of most neural networks to memorize images with randomized labels during training [6]. Inspired by the discovery of the neural efficiency hypothesis from human intelligence research, this work develops an assessment technique called neural layer state space, which can be used to understand neural efficiency in simple neural networks. The ultimate goal of the neural state space assessment technique is to develop a method to measure over-parameterization within individual layers of simple neural networks.