# Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks
## 2 Background and Objectives  
Researchers have been thoroughly investigating the selection process of activation functions for a given modeling task ever since the deep learning became practical with  
Figure 1: Illustrations in (**A–E**) are the five model architecture shapes used in the analysis. For each architecture shape, number of nodes per hidden layer was varied according to Table 1, in order to change the model complexity. modern technology. As a result of empirical and theoretical advancements in the field, there are multiple studies that propose the use of new activation functions [4; 8] while others discuss the effects and reliability of them in deep networks [10; 11; 12]. However, ReLU still remains the most popular and effective activation function in many deep learning models [11]. Rectifier activation is first discussed by [PERSON] et al. in 2000 with regard to its biological motivations [13] and later in 2010 it is introduced as ReLU by [PERSON] et al. [4]. In 2011, the successful use of the rectified activation in deep learning for large data sets was demonstrated by [PERSON] et al. [10]. As a method that applies a variable non-linearity into ReLU, the activation L-ReLU was discussed in parallel to the introduction of ReLU in deep learning [8]. [PERSON] et al., compares the effects of activation functions with the state-of-the-art research results and outline the trends and functionalities such as feature detection capabilities associated with the activation functions [2]. However, there is lack of insight into the effects of activation functions as an important hyper-parameter on the performance of deep learning models. Optimization of hyper-parameters of neural networks are often analyzed by Auto Machine Learning Tools [14]. [PERSON], [PERSON], et al. [15] introduced an ANOVA based approach to understand the effects of hyper-parameters in deep learning. [PERSON], [PERSON], et al. [16] and [PERSON], [PERSON], et al. [17] have presented a study on the importance of hyper-parameters across different data sets and algorithms. They mainly focus on the commonly used algorithms such as, support vector machines, random forests and Adaboost. An interesting work done by [PERSON], [PERSON], et al. [18] describes a method of hyper parameter optimization in neural networks using a network construction algorithm. [PERSON], et al. [19] explore the significance of automated approaches to find best hyper-parameters for machine learning workflows. However, in neural network models, the number of layers (depth) and the number of nodes per layer (width) or in other words, the shape of the architecture has been studied as the most important hyper-parameters [20]. In this study, we show that the correct selection of most commonly used activation functions, ReLU and L-ReLU is as important as the selection of the shape of the network architecture for optimal results. The ReLU activation only allows to pass positive input and cuts-off the negative inputs. This is shown to be problematic in the training of neural networks and often leads to dying neurons [21]. In practice, possible empirical remedies to overcome this problem is either to lower the learning rate or if possible, to reduce the depth of the network. However, L-ReLU on the other hand becomes a handy alternative to ReLU in such situations which has been discussed extensively by [PERSON] et al. [8]. There are several recent studies done on the information flow and information loss optimizations in neural networks [22; 23] in which authors discuss about the information loss due to the limitations in data sampling, and data encoding. It is equally important to address the variations in information flow in neural networks due to the nonlinearity in the network since the application of nonlinearity is an essential tool in deep learning. Furthermore, discussions on the non-linearity in L-ReLU are not widely available in literature compared to the studies based on ReLU.