# Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks
## 3 Results and Discussion
### Effects of Nonlinearity for Different Data Domains  
In Figure 3, we show the trends in validation loss as a function of linearity factor, \(\alpha\) in L-ReLU using two types of data sets: continuous data (left panel) and categorical image data (right panel) by comparing results form six databases in total. In this test, the shape of the model architecture and the number of nodes in each hidden layer of the fully connected network were set to a fixed sequence as explained in Appendix A.4, Tables A2 and A3. Three categorical data bases (MNIST-digits data, MNIST-fashion data and EMNIST-letters data) each with 60,000 sample size were used to study and compare the results. To interpret the effects of nonlinearity for continuous data with different dimensions, results are presented for 3 continuous databases with 8, 16, and 24 features, each with 60,000 sample size. Continuous data were simulated according to the following model using a Gaussian noise with standard deviation of one. \[f(x)=\exp(\sum_{n=1}^{N}a_{n}x_{n})+\text{Gaussian Noise} \tag{3}\]  
According to Figure 3, left panel, for the continuous data, the validation loss increases significantly as a function of \(\alpha\) and, we only show the results up to \(\alpha\) = 0.6 because, the training loss for the continuous data become unstable and rapidly increases when \(\alpha\) is greater than 0.6. Validation loss for regression data with 16 and 24 features reaches an extremely large value at \(\alpha=1\) (in the order of \(10^{4}\)). This demonstrates the need for ReLU when training data consist of features nonlinearly related to the target variable (see Equation (3)). Furthermore, in Figure 3 left panel, the model with \(\alpha=0.6\) shows much weaker performance on the data set with 24 features (in green) compared to results obtained using the data set with eight features (in red). This observation shows the inability of L-ReLU in capturing the nonlinear function space when the data becomes complex with more input features. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Training** & \multicolumn{2}{c}{**Width**} & \multicolumn{4}{c}{**Optimal Valve of the Linearity (\(\alpha\)) in the Network**} \\ \cline{3-6}
**Data Set** & **per Layer** & **Arch. A** & **Arch. B** & **Arch. C** & **Arch. D** & **Arch. E** \\ \hline MNIST- & width-1 & 0.3 & 0.3 & 0.2 & 0.1 & 0.5 \\ digits & width-2 & 0.0 & 0.1 & 0.0 & 0.0 & 0.0 \\  & width-3 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ \hline MNIST- & width-1 & 0.4 & 0.3 & 0.1 & 0.1 & 0.5 \\ fashion & width-2 & 0.1 & 0.1 & 0.1 & 0.0 & 0.1 \\  & width-3 & 0.1 & 0.0 & 0.1 & 0.0 & 0.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Optimal value of the linearity factor (\(\alpha\)) at which the maximum validation accuracy was achieved for neural network models tested using MNIST-digits and MNIST-fashion images. Width of each model architecture shape was varied by changing the number of nodes per layer. See Table 1 for the number of nodes per layer corresponding to the network widths used in each model. According to Figure 3, right panel, the dependence of validation loss on \(\alpha\) for categorical image data is much weaker compared to the continuous data and it does not blow-up even at \(\alpha=1\). This may be due to the nonlinearity in the softmax activation function applied to the last layer of the model. That means, even at \(\alpha=1\), the softmax activation in the last layer preserves nonlinearity which prevents the validation loss from blowing up. The ReLU function adds strong nonlinearity to the network in comparison to the L-ReLU and enhances the independent learning in each hidden layer. This could be one possible reason for why the validation loss at \(\alpha=0\) happens to be the best value for both continuous and categorical data sets.