# Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks
## 3 Results and Discussion
### Entropy Profile of Shallow Neural Networks  
In this section we discuss the possibility of using the entropy as a measurement of randomness to represent the information flow in the shallow neural networks. This study is necessary since the use of activation functions and different model architectures affects the information flow in neural networks, which ultimately could affect the performance of the model. We have already shown that the use of different linearity factors affects the  
Figure 4: **(a)** Validation loss with respect to the validation loss at \(\alpha=0\) as a function of linearity factor (\(\alpha\)) for Dog Breeds, FOOD-11 and cifar10 data sets modeled using VGG16 pre-trained model, (**b**) validation loss with respect to the validation loss at \(\alpha=0\) as a function of linearity factor (\(\alpha\)) for Dog Breeds data using VGG16, VGG19, InceptionV3, ResNet50 and Xception pre-trained models. For each curve, width of the band shows the uncertainty due to different weight initialization in classification layers. model performance in both regression and classification examples that were discussed in this paper. For this study, entropy estimation was performed for each training epoch, using the output of each hidden layer of a shallow neural network with 5 hidden layers. Output of each hidden layer was divided in to 100 equal sized bins and the resulting distribution was used to calculate the entropy (Equation (2)). Figure 5 shows the entropy calculated in a neural network with five hidden layers while changing the linearity factor from \(\alpha=0\) to \(\alpha=0.4\) for two databases: MNIST-digits and MNIST-fashion using 60,000 images from each database with a 33% validation split. The number of nodes in hidden layers 1-5 of the model architecture used for this analysis were 256, 128, 128, 128 and 32, respectively. We used the Negative Log Likelihood Loss (_NLLLoss_) [41] as the loss function and _Adam_ optimizer [42] for the model optimization. As seen in Figure 5, for both MNIST-digits and MNIST-fashion data sets, the entropy of each hidden layer in the network increases as the linearity factor (\(\alpha\)) is increasing. When the value of \(\alpha\) is increased, the negative values are allowed to flow through the network. Therefore, the randomness in the hidden layer outputs becomes enhanced due to the increase in the number of possible outcomes when compared to the \(\alpha=0\) situation. This observation suggest that the entropy can be used as a measure of randomness in hidden layer outputs to visualize and quantify the information flow in neural networks. In addition to this observation, we can also see that the entropy of most layers decrease as the training progresses. This is due to the decrease in randomness of hidden layer outputs as a result of parameter optimization. Since the network is initialized with random parameters, the output of each hidden layer is expected to be highly random leading to a relatively higher entropy at the beginning of the training. However, as the training progresses, the model converges into a state where the composition function represented by the neural network has hidden layer outputs that are more ordered and stabilized leading to a lower entropy. In practice we use the learning curves as a evaluation of the learning process of the whole network together. In conjunction with the learning curve,  
Figure 5: Entropy profile for the 5-layer neural network during the training process. Linearity factor (\(\alpha\)) in L-ReLU was changed from \(\alpha=0.0\) to \(\alpha=0.4\). Hidden layers 1-5 of the neural network consist of nodes combinations: 256, 128, 128, 128 and 32, respectively. This test was done using two different data sets, 60,000 MNIST-digits images (**upper panel**) and MNIST-fashion images (**lower panel**) with 33% validation split. Black curves show the average (avg.) entropy in the network.