# Plant-Leaf Recognition Based on Sample Standardization and Transfer Learning
## 2 Materials and Methods
### Improved ResNet50 Model and Transfer Learning
#### 2.3.1 Comparison of Several Excellent Activation Functions  
The experiments test the performance of several commonly used activation functions on the improved model; they are _ReLU_, _LeakyReLU_, _ELU_, and _Swish_. The following is a brief introduction to these activation functions. And Figure 7 shows an image of them.  
_ReLU_ is the most widely used activation function, which enhances the sparsity of the network by forcing negative inputs to zero, but it also causes the problem of neuronal death [23]. It is defined as the following expression:  
\[ReLU(x)=\max(0,x). \tag{10}\]  
_LeakyReLU_ solves the problem of neuron death [24], and its expression is as follows:  
\[LeakyReLU(x)=\max(ax,x). \tag{11}\]  
where \(\alpha\) is the initialization constant, usually \(\alpha=0.01\).  
The _ELU_ activation function [24] can accelerate the learning of deep neural networks and effectively alleviate the problem of gradient vanishing. It is defined as follows:  
\[ELU(x)=\begin{cases}x,&x>0\\ \alpha(e^{x}-1),&x\leq 0\end{cases}. \tag{12}\]  
where \(\alpha\) is also used as an initialization constant, usually \(\alpha=1\).  
_Swish_ is a variable non-monotonic activation function [24]. It is defined as the following expression:  
\[Swish(x)=\frac{x}{1+e^{-\beta x}}. \tag{13}\]  
where \(\beta\) is a constant or trainable parameter. When \(\beta=1\), _Swish_ is equivalent to Sigmoid-Weighted Linear Unit (SiL). When \(\beta=0\), _Swish_ becomes a linear function, \(f(x)=x/2\). As \(\beta\rightarrow\infty\), _Swish_ approximates the _ReLU_ activation function. In this experiment, \(\beta=1\).  
The abovementioned four activation functions with excellent performance were used to test the improvement of ResNet50 model, and the evaluation results are shown in Table 1. The enhanced model utilizing the _ELU_ activation function demonstrates a superior classification performance, potentially attributed to the heightened learning capability inherent in the _ELU_ activation function. _ELU_ has negative values compared to _ReLU_, thus allowing it to push the mean unit activation closer to zero. The shifts of the mean to zero can make the normal gradient closer to the unit natural gradient and thus accelerate  
Figure 7: The graph of the activation functions.  
the learning [25]. While _LeakyReLU_ also has a negative value, it does not guarantee a robust deactivation state in the presence of noise. The _Swish_ activation function has good generalization capabilities. However, there is evidence [26] that it outperforms _ReLU_ only in deeper networks, and for those that are not deep enough, their results are very similar. The experimental results in this paper also confirm this phenomenon, and it can be seen that they are exactly the same in terms of validation accuracy. Moreover, from the perspective of model training time, the model that uses the _ReLU_ activation function before improvement takes the shortest time, while the model improved with the _LeakyReLU_, _Swish_, and _ELU_ activation functions takes more time successively. Although the time consumption of the improved model using _ELU_ is significantly increased compared with that before the improvement, this trade-off is evidently justified by the observed performance enhancement. Therefore, it can be proved that _ELU_ is more suitable for the improved model proposed in this paper.