# Balanced Loss Function for Accurate Surface Defect Segmentation
## 2 Method
### Balanced Loss Function
#### 2.3.2 Truncated Cross-Entropy Loss  
Most point-wise segmentation loss functions can be interpreted as the penalization on the point error \(\epsilon\) of probabilistic prediction \(y\in(0,1)\) (an entry of \(Y\)) given the label \(\hat{y}\in\{0,1\}\) (an entry of \(\hat{Y}\)):  
\[\epsilon=\begin{cases}y,&\hat{y}=0\\ 1-y,&\hat{y}=1\end{cases} \tag{3}\]  
The cross-entropy loss uses a logarithmic function to scale up the error:  
\[CE(\epsilon)=-ln(1-\epsilon) \tag{4}\]  
To scale down the loss value of easy and hard examples, we propose a truncated cross-entropy loss. First, an upper bound \(\epsilon_{sup}\in(0,1]\) and a lower bound \(\epsilon_{inf}\in[0,1)\) (satisfying \(\epsilon_{sup}>\epsilon_{inf}\)) are set to identify easy and hard examples. The truncated cross-entropy loss is defined as:\[TruncCE(\epsilon)=\begin{cases}0,\epsilon<\epsilon_{inf}\\ -ln(1-\epsilon)-t_{inf},\epsilon_{inf}\leq\epsilon\leq\epsilon_{sub}\\ \frac{\epsilon-\epsilon_{sub}}{1-\epsilon_{sub}}+t_{sub}-t_{inf},\epsilon_{ sub}<\epsilon\end{cases} \tag{5}\]  
where  
\[t_{inf} =-ln(1-\epsilon_{inf})\] \[t_{sub} =-ln(1-\epsilon_{sub}) \tag{6}\]  
The truncated cross-entropy loss is a consistent piece-wise loss function with two non-differentiable points at \(\epsilon=\epsilon_{inf}\) and \(\epsilon=\epsilon_{sub}\). To make it compatible with gradient descent optimization, we set the derivative of the two singular points consistent with one side, and the derivative of the truncated cross-entropy function is formalized as:  
\[\frac{d\;TruncCE(\epsilon)}{d\;\epsilon}=\begin{cases}0,&\epsilon<\epsilon_{ inf}\\ \frac{1}{1-\epsilon},&\epsilon_{inf}\leq\epsilon\leq\epsilon_{sub}\\ \frac{1}{1-\epsilon_{sub}},&\epsilon_{sub}<\epsilon\end{cases} \tag{7}\]  
The derivative of the truncated cross-entropy loss is inconsistent, and the derivative is zero for easy examples (\(\epsilon<\epsilon_{inf}\)). Such properties can be found in many other classical loss functions. For example, mean absolute error (MAE) regression loss has an inconsistent point at \(\epsilon=0\); Huber loss has two singular points the separate easy and hard examples; Hinge loss also sets zero loss value and zero derivative for easy examples. In modern auto-gradient-based neural network implementations, the inconsistency and non-smoothness of the loss function on a few points can be tackled by user-specified gradient calculation.  
Figure 4 visualizes the loss value and the gradient over \(\epsilon\) of cross-entropy loss, focal loss and truncated cross-entropy loss. It throws light on how easy/hard examples are treated differently. For cross-entropy loss, the loss value and the gradient of easy examples (small \(\epsilon\)) are still significantly large compared to difficult examples. Especially, when \(\epsilon\) is close to 0, cross-entropy loss still assigns gradient value 1 to the point, forcing the model to fit very easy examples. Considering that the quantity of easy examples is enormous, the loss will be dominated by them. Truncated loss function sets the loss value and gradient of easy examples to zero to suppress such imbalance. Focal loss scales down the loss and gradient of easy examples by a continuous means.  
For hard examples, most loss functions produce a very large loss and gradient. For a better view, it is not plotted that cross-entropy and focal loss soar to \(10.0+\) when \(\epsilon\) approaches 1.0. The loss and gradient can probably be dominated by hard examples in certain phases of training. Although focal loss (\(\beta=4\)) seems to have relatively low gradients on hard examples, the gradients of other examples have already been suppressed. Truncated cross-entropy loss produces a constant gradient value for hard examples, allowing a balanced distribution of training focus on hard and not-very-hard examples.