# Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks
###### Abstract  
The nonlinearity of activation functions used in deep learning models is crucial for the success of predictive models. Several simple nonlinear functions, including Rectified Linear Unit (ReLU) and Leaky-ReLU (L-ReLU) are commonly used in neural networks to impose the nonlinearity. In practice, these functions remarkably enhance the model accuracy. However, there is limited insight into the effects of nonlinearity in neural networks on their performance. Here, we investigate the performance of neural network models as a function of nonlinearity using ReLU and L-ReLU activation functions in the context of different model architectures and data domains. We use entropy as a measurement of the randomness, to quantify the effects of nonlinearity in different architecture shapes on the performance of neural networks. We show that the ReLU nonliearity is a better choice for activation function mostly when the network has sufficient number of parameters. However, we found that the image classification models with transfer learning seem to perform well with L-ReLU in fully connected layers. We show that the entropy of hidden layer outputs in neural networks can fairly represent the fluctuations in information loss as a function of nonlinearity. Furthermore, we investigate the entropy profile of shallow neural networks as a way of representing their hidden layer dynamics. neural networks; activation function; entropy; nonlinearity +
Footnote †: journal:  
0000-0001-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-1400-14000-14000-1400-1400-14000-14000-14000-1400-14000-14000-14000-14000-14000  
where \(\alpha\) is the linearity factor (\(0<\alpha<1.0\)). Note that the L-ReLU with \(\alpha=0\) is the ReLU function and it becomes the identity function when \(\alpha=1\). In practice, the value of \(\alpha\) is kept closer to zero. However, there is no supporting theory to predict the ideal value of \(\alpha\). Instead, it is usually determined through trial-and-error observations. Similarly, selection between ReLU and L-ReLU for a certain network is determined by prior experimental knowledge. Usually, investigation of the information loss due to the use of activation functions is not given a priority since the accuracy is used as the primary method of evaluating the performance of many deep learning models. However, when it comes to shallow networks, effects of activation functions may play an important role in understanding the learning process and for the interpretation of results. Therefore, a study of experimental evidence to enhance our understanding of the behavior of activation functions is necessary in order to minimize the possible speculations. The amount of nonlinearity of a deep network results from the composition of the layers of nonlinear functions. Therefore, the impact of different activation functions on the final performance is likely linked to the model architecture (width and the depth). In addition, as pointed out by [PERSON] [9], model performance also depends on the data type (for example, continuous, categorical, calligraphic, or photographic). Based on the considerations above, we take \(\alpha\) as a measure of nonlinearity introduced by an activation function and study the model performance for different choices, and in conjunction to different network shapes and architecture, as illustrated in Figure 1, with examples chosen from a wide range of data domains. We use the entropy, a measurement of randomness or disorder, for the estimation of information flow in shallow neural networks. Entropy, \(H(x)\) of a random variable \(x\) is defined as,  
\[H(x)=-\sum_{i=1}^{n}P(x_{i})logP(x_{i}) \tag{2}\]  
where, \(P(x_{i})\) represents the probability of \(i^{\text{th}}\) outcome and \(n\) represents the total number of distinct outcomes. Entropy can be calculated from the output of each hidden layer of a neural network during the training.