## 5 Experimental Results
### Experimental Results  
Reported in Tables 4 and 5 is the performance (accuracy) of the different activation functions on the CNN topologies VGG16 and ResNet50, each trained with a batch size (BS) of 30 and a learning rate (LR) of 0.0001 for 20 epochs (the last fully connected layer has an LR 20 times larger than the rest of the layers (i.e., 0.002)), except the stochastic architectures that are trained for 30 epochs (because of slower convergence). The reason for selecting these settings was to reduce computation time. Images were augmented with random reflections on both axes and two independent random rescales of both axes by two factors uniformly sampled in [1,2] (using MATLAB data augmentation procedures). The objective was to rescale both the vertical and horizontal proportions of the new image. For each stochastic approach, a set of 15 networks was built and combined by sum rule. We trained the models using MATLAB 2019b; however, the pre-trained architectures of newer versions perform better. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c c c c c} \hline \hline
The performance (accuracy) of the following ensembles is reported in Tables 4 and 5:  
* ENS: sum rule of {MeLU (\(k=8\)), Leaky ReLU, ELU, MeLU (\(k=4\)), PReLU, SReLU, APLU, ReLU} (if \(maxInput=1\)) or {MeLU (\(k=8\)), MeLU (\(k=4\)), SReLU, APLU, ReLU} (if \(maxInput=255\));
* eENS: sum rule of the methods that belong to ENS considering both \(maxInput=1\) and \(maxInput=255\);
* ENS_G: as in ENS but Small GaLU and GaLU are added, and in both cases \(maxInput=1\) or \(maxInput=255\);
* eENS_G: sum rule of the methods that belong to ENS_G but considering \(maxInput=1\) and \(maxInput=255\);
* ALL: sum rule among all the methods reported in Table 4 with \(maxInput=1\) or \(maxInput=255\). Notice that when the methods with \(maxInput=255\) are combined, standard ReLU is also added to the fusion. Due to computation time, some activation functions are not combined with VGG16 and so are not considered;
* eALL: sum rule among all the methods, both with \(maxInput=1\) and \(maxInput=255\). Due to computation time, some activation functions are not combined with VGG16 and thus are not considered in an ensemble;
* 15 ReLU: ensemble obtained by the fusion of 15 ReLU models. Each network is different because of the stochasticity of the training process;
* Selection: ensemble selected using SFFS (see Section 3.1);
* Stoc_1: MeLU(\(k=8\)), Leaky ReLU, ELU, MeLU(\(k=4\)), PReLU, SReLU, APLU, GaLU, sGaLU. A \(maxInput=255\) has been used in the stochastic approach (see Section 3.2);
* Stoc_2: the same nine functions of Stoc_1 and an additional set of seven activation functions: ReLU, Soft Learnable, PReLU, learnableMish, SRS, Swish Learnable, and Swish. A \(maxInput=255\) has been used;
* Stoc_3: same as Stoc_2 but excluding all the activation functions proposed in [46; 47; 61] (i.e., MeLU, GaLU, and sGaLU);
* Stoc_4: the ensemble detailed in Section 4. The most relevant results reported in Table 4 on ResNet50 can be summarized as follows:
* ensemble methods outperform stand-alone networks. This result confirms previous research showing that changing activation functions is a viable method for creating ensembles of networks. Note how well 15 ReLU outperforms (\(p\)-value of 0.01) the stand-alone ReLU;
* among the stand-alone ResNet50 networks, ReLU is not the best activation function. The two activations that reach the highest performance on ResNet50 are MeLU (\(k=8\)) with \(maxInput=255\) and Splash with \(maxInput=255\). According to the Wilcoxon signed rank test, MeLU (\(k=8\)) with \(maxInput=255\) outperforms ReLU with a \(p\)-value of 0.1. There is no statistical difference between MeLU (\(k=8\)) and Splash (with \(maxInput=255\) for both);
* according to the Wilcoxon signed rank test, Stoc_4 and Stoc_2 are similar in performance, and both outperform the other stochastic approach with a \(p\)-value of 0.1;
* Stoc_4 outperforms eALL, 15 ReLU, and Selection with a \(p\)-value of 0.1.