## 4 Results
### Architecture of Modified AlexNet-33 (MAlexNet-33)
#### 4.2.1 Exploring Variations of the Rectified Linear Unit and the Number of Layers  
For this experiment, the response of the system to reducing the number of layers is investigated. Further, different variations of the ReLU activation function are also examined. Table 5 displays the different combinations tested for this experiment with regards to decreasing the number of layers and changing the activation function, presented as an average between 11 classes. Hence, throughout the results, it is apparent that the micro averaging results between the four measures are the same and there are close similarities between some of the measures. This is due to the total number of false negatives and false positives being the same. More distinct differences between the classes can be seen in the per-level comparison, such as that of Table 3.  
From Table 5, AlexNet-20 was achieved by removing one grouped convolutional, two ReLU, one fully connected, and one 50% dropout layer from the original network. It is observed that removing convolutional and fully connected layers from the network reduces its performance as well.  
However, it is also apparent that using other activation functions improves the performance. For instance, using a Leaky ReLU with a 0.01 parameter in place of the ReLU activation function increased the weighted F1-score to 85.58%, having less than 1% difference from the original network's performance. Such improvement is reportedly due to the Leaky ReLU's added parameter to solve the dying ReLU problem. Due to having less layers in the system, a reduction of about 30% from the original size was also achieved. MAlexNet-20 with a Leaky ReLU activation function has a network size of about 150 MB, compared against AlexNet's 220 MB network size.  
Subsequent to this, the concept of a successive activation function was also looked at. For this, two activation function layers were placed successively throughout the network. However, as per Table 6, it is implied that using two successive activation functions does not necessarily improve the overall system performance. However, it is also apparent that using more than one activation function does not affect the overall size of the network.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Network** & **Type** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline \multirow{4}{*}{AlexNet} & Micro & 86.36\% & 86.36\% & 86.36\% & 86.36\% \\  & Weighted & 86.36\% & 86.72\% & 86.36\% & 86.24\% \\  & Macro & 81.03\% & 82.99\% & 81.03\% & 81.69\% \\  & Micro & 85.19\% & 85.19\% & 85.19\% & 85.19\% \\ AlexNet-20 & Weighted & 85.19\% & 86.01\% & 85.19\% & 85.02\% \\  & Macro & 79.68\% & 82.42\% & 79.68\% & 80.44\% \\  & Micro & 84.30\% & 84.30\% & 84.30\% & 84.30\% \\  & Weighted & 84.30\% & 84.80\% & 84.30\% & 84.18\% \\  & Macro & 78.08\% & 81.20\% & 78.08\% & 79.22\% \\ AlexNet-20 & Micro & 85.70\% & 85.70\% & 85.70\% & 85.70\% \\ with Leaky & Weighted & 85.70\% & 86.37\% & 85.70\% & 85.58\% \\ ReLU (0.01) & Macro & 79.45\% & 83.62\% & 79.45\% & 80.99\% \\ AlexNet-20 & Micro & 84.10\% & 84.10\% & 84.10\% & 84.10\% \\ with Clipped & Weighted & 84.10\% & 84.25\% & 84.10\% & 84.04\% \\ ReLU (6) & Macro & 78.38\% & 78.55\% & 78.38\% & 78.26\% \\ AlexNet-17 & Micro & 81.89\% & 81.89\% & 81.89\% & 81.89\% \\ with Leaky & Weighted & 81.89\% & 82.67\% & 81.89\% & 81.74\% \\ ReLU (0.01) & Macro & 75.04\% & 76.39\% & 75.04\% & 75.13\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance Measures of Different Networks using Variations of the F1-score.