# Deep Ensembles Based on Stochastic Activations for Semantic Segmentation
## 2 Methods
### Stochastic Activation Selection  
Given a neural network architecture and a pool of different activation functions, Stochastic Activation Selection consists in creating different versions of the same architecture that differ in the choice of the activation layers. This method was first introduced in [37]. The process to create a new network is based on the replacement of each activation layer (ReLU) by a new activation function which can be fixed a priori or randomly selected. This leads to a new network, which in the stochastic version, has different activation layers through the layers. Since this is a random procedure, it yields a different network every time. Hence, we iterate the process multiple times to create many different networks that we use to create an ensemble of neural networks. We train each network independently on the same set of data and then we merge their results using the average rule, which consists in averaging the softmax output of the networks in the ensemble. The pseudo-code of the procedures for stochastic activation selection is reported in Figure 1.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Network** & **Depth** & **Size (MB)** & **Parameters (Millions)** & **Input Size** \\ \hline mobilenetv2 & 53 & 13 & 3.5 & 224 \(\times\) 224 \\ resnet18 & 18 & 44 & 11.7 & 224 \(\times\) 224 \\ resnet50 & 50 & 96 & 25.6 & 224 \(\times\) 224 \\ xception & 71 & 85 & 22.9 & 299 \(\times\) 299 \\ IncR & 164 & 209 & 55.9 & 299 \(\times\) 299 \\ efficientnetb0 & 82 & 20 & 5.3 & 224 \(\times\) 224 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of CNN models.  
In this paper, we use Deeplabv3+ [27] as neural architecture. The pool of activation functions is made by the following activation functions. ReLU [38] is the most common activation in modern deep learning. Leaky ReLU [39] is a modification of ReLU that has a small positive slope for negative values to avoid zero gradient regions. ELU [40] also has a positive slope for negative values that saturates as the input goes to minus infinity. PReLU [41] is a leaky ReLU whose slopes on negative values are different in every channel and are learnable. S-Shaped ReLU [42] (SReLU), as its name suggests, has the shape of the letter S, since it is equal to the identity function in a compact interval that contains zero and has a learnable slope outside the interval. This introduces further representation power to the activation. Adaptive Piecewise Linear Unit [43] (APLU) goes deeper in this direction since its learnable parameters are all the non-differentiability points and the slopes of a generic piecewise linear function with a fixed number of non-differentiability points. Mexican ReLU [44] (MeLU) (with \(k\in\{4,8\}\)) is similar to APLU, since it is a generic piecewise linear function, although non-differentiability points are fixed and the different slopes are obtained by adding triangular function (Mexican hat functions) to leaky ReLU. This allows a more stable training. Gaussian Linear Unit (GaLU) [37] (with \(k\in\{4,8\}\)) follows the idea of MeLU but adding Gaussian-like function, PDELU is a smooth function equal to ReLU for positive values and whose slope on positive values is calculated so that its output has zero mean and unit variance if the input follows a Gaussian distribution with zero mean and unit variance [45]. Swish (fixed and learnable) is an activation introduced in [46] using a random search and reinforcement learning to find the \"optimal\" activation function, Soft Root Sign [47] is similar to PDELU in the sense that is the function that maps a standard Gaussian into a random variable with unit mean-variance. Mish (fixed and learnable) [48] is defined as  
\[f(x)=x\cdot\tan\mathrm{h}(\log(1+e^{ax}))\]  
and their authors report very high performances for this function that rapidly goes to minus infinity for small values, as it does not happen often for activation functions. Lastly, Soft Learnable [37] is yet another function that maps Gaussian distributions into zero mean and unit variance distributions.