# Improved U-Net: Fully Convolutional Network Model for Skin-Lesion Segmentation
## 2 Materials and Methods
### Activation Functions  
#### 2.5.1 ReLU  
Activation functions are key to artificial neural networks discovering and making sense of complex and nonlinear relationships between input and output, because they bring nonlinearity to the network. Their objective is to convert an input signal layer of the neural network into an output signal. The ReLU is the most common activation function in deep networks. It expedites the convergence of the training procedures and provides better results compared with standard sigmoid-like activation functions [36; 37; 38]. The function and derivative of this nonlinearity are represented as follows:  
\[f(x)=\left\{\begin{array}{l}x\ if\ x\geq 0\\ 0\ if\ x<0\end{array}\right.\qquad\quad f^{\prime}(x)=\left\{\begin{array}{l} 1\ if\ x>0\\ 0\ if\ x<0\end{array}\right. \tag{3}\] \[\text{(a) Function}\qquad\qquad\qquad\qquad\text{(b) Derivative}\]  
Geometrically, the ReLU function and its derivative are illustrated as follows:  
The ReLU is linear for all non-negative values and zero otherwise. The derivative, an important factor for updating gradients, is zero when it takes negative values and one otherwise. Apart from the numerous benefits of the ReLU rectifier, recent studies show that there are some shortcomings [39; 40]. Several gradients can be unstable during training and can \"die\" by causing a weight to update in such a way that it will never activate on any data point again. Thus, the ReLU can result in \"dead neurons\". For activations in the region (x < 0) of the ReLU, the gradient will be zero, and the weights 0 will not be fine-tuned during descent. Thus, the neurons that go into that state will stop responding to variations in the input. Simply because the gradient is zero, nothing will change. This is called the \"dying ReLU\" problem [30; 32].  
Systems using these units sometimes manifest the dying ReLU issue during training, in which the state (x < 0) is very likely for most training examples for nodes within the network. It is comprehensible that the error backpropagation algorithm utilizes a derivative function, \(\hat{\mathsf{r}}^{\prime}(x)\), such that the ReLU nonlinearity is zero for x < 0 (see Figure 7). Thus, patterns that cause x < 0 do not change the unit's parameters. This condition can cause ambiguity in practice, because units that are not active will probably not be trained. The dying ReLU problem is likely to occur when:  
* The learning rate is too high.
* There is a large negative bias.  
Figure 6: Example of bilinear interpolation.  
Consider the following equation used to update the weight values during backpropagation:  
\[w_{new}=w_{old}-\left(\frac{tsL}{tsLt}\right)*\alpha+b. \tag{4}\]  
Here, \(\alpha\) is the learning rate, b is the bias, and \(\frac{tsL}{tsLt}\) is the derivative of the loss function with respect to the weight value (\(w\)). Thus, if the learning rate is too high or the bias is negative, it may bring negative new weight values after the backpropagation step. After the weight becomes negative, the ReLU activation function of that neuron will never be activated and will lead to the death of the neuron.