# Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks
## 3 Results and Discussion
### Effects of Nonlinearity in Different Network Architectures  
For this test, we use five model architecture shapes (Figure 1), each with three hidden layers. ReLU or L-ReLU (with variable nonlinearity) were used as the activation function. For the convenience, we will refer to the shapes of the network shown in Figure 1A-E as Architecture A-E. Architecture A consists of a constant number of nodes per hidden layer while in the other four architectures (Architecture B-E) the number of nodes varies from layer to layer. In Architecture B and C, the ratio between the number of nodes in two consecutive layers is constant while in Architecture D and E, the ratio between the nodes in first two layers is the reciprocal of the ratio between the number of nodes in last two layers. The complexity of each model architecture was varied by changing the number of nodes per hidden layer according to the combinations shown in Table 1. We trained the models for total of 20 epochs and present the maximum validation accuracy obtained during the training process. We used stochastic gradient descent [31] with a learning rate of 0.1 as the optimizer, and the categorical cross entropy [32] as the loss function. We explored the effect of the linearity factor, \(\alpha\) on the model performance based on the five architectures illustrated in Figure 1 with different number of trainable parameters. Figure 2 shows the maximum validation accuracy for Architectures A-D as a function of \(\alpha\) for MNIST-digits images and MNIST-fashion images with the uncertainties estimated by the standard error. Note that, in the error estimation, we were mainly concerned \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Width** & **Arch. A** & **Arch. B** & **Arch. C** & **Arch. D** & **Arch. E** \\ \hline width-1 & (8, 8, 8) & (8, 16, 32) & (16, 8, 4) & (16, 8, 16) & (8, 16, 8) \\ width-2 & (32, 32, 32) & (32, 64, 128) & (64, 32, 16) & (64, 32, 64) & (32, 64, 32) \\ width-3 & (256, 256, 256) & (128, 256, 512) & (512, 256, 128) & (512, 256, 512) & (256, 512, 256) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Different combinations of number of nodes used in three hidden layers of the five network Architecture shapes A–E (Arch. A–E) presented in in Figure 1. Width of each model architecture shape was varied by changing the number of nodes per layer. For each model architecture, the number of nodes used is shown as, (\(n_{1}\), \(n_{2}\), \(n_{3}\)) and they represent the number of nodes in first, second and third hidden layers, respectively. about the uncertainty arises due to the parameter initialization (initial conditions) of the models. Therefore, the uncertainties were calculated by training the models 20 times with randomized weight initialization in each run and, by taking the standard deviation of results obtained in all runs. As shown in Figure 2, for the Architecture shapes A-D, the model accuracy increases as the number of parameters is increasing for all values of \(\alpha\). One interesting observation in Figure 2 is that, for both MNIST-digits and MNIST-fashion data sets, L-ReLU was able to increase the validation accuracy only when the number of parameters were relatively small (data points in black), regardless of the shape of the model architecture and the data set used for the training. Based on these observations, in Table 2, we show the optimal value for the liniarity factor, \(\alpha\) for each model architecture (Arch. A-E) discussed in this study (see Table 1 for the number of nodes used in each model architecture). According to Table 2, it is clear that the optimized value for the linearity factor decreases (approaches zero) as the width of the network increases. That means, for networks with sufficiently large number of trainable parameters, ReLU is a better choice for the activation function. A possible reason for L-ReLU being a better alternative to ReLU for small networks could be that ReLU has suppress some essential information that were preserved by L-ReLU (in a regularized fashion).