# Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks
## 3 Results and Discussion
### Effects of Nonlinearity for Different Data Domains
Another interesting observation in Figure 4, right panel is that the dependence of loss on \(\alpha\) for EMNIST-letters data is much stronger (green curve) compared to the MNIST-digits and MNIST-fashion data and the model with \(\alpha=1.0\) shows the worst performance of all. Compared to number of classes in the MNIST-digits and MNIST-fashion data (each has 10 classes), the EMNIST-letters data has more classes (26) and the results suggest that the ReLU becomes most efficient when the number of classes in the database is relatively large. In Figure 4, we show the validation loss (with respect to the validation loss at \(\alpha=0\)) as a function of \(\alpha\) when transfer learning is used to extract important features in image classifications. Transfer learning is very efficient when there is limited number of images available to train a model. Small databases make the model-training quite difficult, specially in image classifications. In such situations, pre-trained models can be used to extract the important features (bottleneck features) from the images. A block with classification layers must be built on top of the pre-trained model and it is trained to classify the bottleneck features extracted from the pre-trained model. This classification block often consists of fully connected layers and uses nonlinear activation functions. However, the use of strong nonlinear functions in the classification layer could destroy the useful information extracted using transfer learning. To investigate on this matter, we have studied the effects of nonlinearity in the classification layers on the model performance when the transfer learning is used. First, we used three data sets: FOOD-11 [33], cifar10 [34] and Dog Breeds [35] and extracted the bottleneck features using the pre-trained model VGG16 [36] (pre-trained using ImageNet data [37]) and results are shown in Figure 4, left panel. It is clear that the  
Figure 3: **(left panel)** Validation loss as a function of linearity factor (\(\alpha\)) for continuous data with different dimensions: 8, 16 and 24 features (ft.). (**right panel**) Training and validation loss as a function of linearity factor (\(\alpha\)) for categorical data (MNIST-digits, MNIST-fashion and EMNIST-letters data sets). Note that, to test the effects of data domains, a neural network with a fixed architecture shape was used in the modeling of all data sets shown in this figure. (see Appendix A.4, Tables A2 and A3) for model details. L-ReLU performs better for all three data sets compared to ReLU. However, we observed that, for both cifar10 and Dog Breeds data, classification layers with L-ReLU were not able to fit to data when \(\alpha=1\) and the increasing trend in validation loss is visible for Dog Breeds data set after \(\alpha=0.6\). In Figure 4, right panel, we show the validation loss (with respect to the validation loss at \(\alpha=0\)) for Dog Breeds data using five pre-trained models [36; 38; 39; 40]. For all five pre-trained models, we observe that the model performance improves as the \(\alpha\) is increased. However, both VGG-16 and VGG-19 models show week performance after \(\alpha=0.6\). Due to the transfer learning, we can assume that, most of the important information is already extracted into the bottleneck layer. In addition, the nonlinearity is already applied to pre-trained models through nonlinear operations such as ReLU activation function and max-pooling operations. Therefore, the use of a strong nonlinear activation function (ReLU) to classify the bottleneck layer does not seem to be improving the accuracy. Instead, there is a loss of important information caused by the application of ReLU in the classification layers. Above results suggest that the amount of information passed to the next layer by the activation function has a significant effect on the model performance. We find that, it is worth quantifying the level of information flow in the network and analyzing how it is affected by the nonlinearity of activation functions.