# Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks
## 4 Conclusions  
We have extensively analysed the effects of the number of parameters, shape of the model architecture and nonlinearity on the performance of deep learning models. We found that the L-ReLU function is more effective than ReLU only when the number of parameters is relatively small, according to the tests conducted using MNIST data using shallow neural networks (all observations are summarized in Table 2). We show that, for shallow neural networks, the shape of the model architecture is less important compared to the number of trainable parameters, for the enhancement of the model performance, regardless of the nonlinearity in the network. Then, we looked into different data domains (continuous, categorical with or without transfer learning) as inputs to the networks and their effects on the model performance under different nonlinearities in the network. Results show that, for classification and regression examples that do not use transfer learning, strong nonlinearity (values of \(\alpha\) closer to zero) increases the performance of the model. However, in the case of using transfer learning to extract bottleneck features, we found that the use of L-ReLU in the final fully connected layers shows better accuracy than the use of a strong nonlinear function like ReLU. Finally we introduced the entropy as a way of quantifying the information flow in shallow neural networks. We observe that the average entropy of shallow neural networks increases as the nonlinearity of the network is decreased. Therefore, we suggest the entropy as a way of quantifying the information loss due to the use of nonlinear activation functions specially for shallow neural networks. Furthermore, we show that the entropy evolution of the hidden layer outputs can fairly represent the transformation of composition function space from a disordered initial state to an ordered final state during the training. However, we do not observe strong evidence linking the model performance to the variations in the entropy profile of neural networks. Conceptualization, Y.W. and D.V. ; methodology, N.K. ; validation, N.K. and N.R.R. ; initial numerical study, Z.K. ; formal analysis, N.K. ; writing-original draft preparation, N.K. and Y.W. ; writing-review and editing, Y.W., D.V. and N.R.R. ; visualization, N.K. ; supervision, L.H., D.V. and Y.W. ; funding acquisition, Y.W., D.V. and L.H. All authors have read and agreed to the published version of the manuscript. This work is funded by the National Science Foundation (NSF), grant no: CNS-1831980, CNS-1832034, PHY-1831977, and HRD-1800406. Initial computational work in this study was conducted using computational resources at high performance computing centers at Texas Southern University and Prairie View A&M University. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system, which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC). Not applicable. [leftmargin=*]
**Index Availability Statement:** Publicly available data sets were analyzed in this study. This data can be found here:  
(a). MNIST-digits ([[http://yann.lecun.com/exdb/mnist](http://yann.lecun.com/exdb/mnist)]([http://yann.lecun.com/exdb/mnist](http://yann.lecun.com/exdb/mnist)), accessed on 20 December 2020),  
(b). MNIST-fashion ([[https://github.com/zalandoresearch/fashion-mnist](https://github.com/zalandoresearch/fashion-mnist)]([https://github.com/zalandoresearch/fashion-mnist](https://github.com/zalandoresearch/fashion-mnist)), accessed on 20 December 2020),  
(c). EMNIST-letters ([[https://www.kaggle.com/crawford/emnist](https://www.kaggle.com/crawford/emnist)]([https://www.kaggle.com/crawford/emnist](https://www.kaggle.com/crawford/emnist)), accessed on 20 December 2020),  
(d).