# Leveraging Expert Knowledge for Label Noise Mitigation in Machine Learning
## 4 Evaluation and Discussion  
### Accuracy  
In this section, the accuracy metric is used to evaluate the performance of the aforementioned six algorithms. As shown in Equation (1), the accuracy is defined as the ratio of correct prediction samples over the whole tested data.  
\[Accuracy=\frac{number\,correct\,predictionsamples}{number\,samples} \tag{1}\]  
To ensure that the partitions in the train-test set do not bias the results, \(K\)-fold cross-validation (with K = 3) is used. Tables 2 and 3 show the accuracy of six algorithms when the non-expert training data contain label noise at some different level and tested with the test set.  
As shown in Table 2, on Manga109 dataset, the accuracy of six algorithms decreases as the level of noise increases. When there are no noisy samples in the training set, the accuracy of all methods is nearly the same. The performances of Classification Filtering (CF) methods worsen more than the others when the noise level increases. When the noise level increases, the accuracy of Rule-Weight (SW) and Rule-Remove decreases less than the other algorithms and even better than the benchmark algorithm LC-True. The influence of the expert rules was clearly shown since the accuracy of our proposed methods denominated the Classification Filtering one.  
Table 4 shows the rate of label noise sample detected by the rules on Manga109 dataset. The rules can detect more than 68% of the data samples whose labels have been incorrectly assigned. This asserts that the Logic inference machine works well in detecting label noise. There was no significant difference in the accuracy between Rule-Weight and Rule-Remove. It proves that the detected label noise data samples have low weight and, thereby, low contribution to the performance of the final model.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**MethodNoise Level** & **0\%** & **20\%** & **30\%** & **40\%** & **50\%** \\ \hline Normal & 0.678 & 0.603 & 0.553 & 0.505 & 0.324 \\ CF & 0.679 & 0.597 & 0.422 & 0.411 & 0.303 \\ LC-True & 0.675 & 0.600 & 0.571 & 0.529 & 0.398 \\ LC-Est & 0.681 & 0.603 & 0.579 & 0.516 & 0.333 \\ Rule-Remove & 0.679 & 0.618 & 0.580 & 0.568 & 0.445 \\ RW (\(\alpha\) = 0.02) & 0.679 & 0.619 & 0.568 & 0.551 & 0.419 \\ RW (\(\alpha\) = 0.1) & 0.679 & 0.627 & 0.577 & 0.565 & 0.439 \\ RW (\(\alpha\) = 0.4) & 0.679 & 0.616 & 0.580 & 0.568 & 0.459 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Accuracy of 6 algorithms at different noise levels on Manga109.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**MethodNoise Level** & **0\%** & **20\%** & **30\%** & **40\%** & **50\%** \\ \hline Normal & 0.841 & 0.699 & 0.645 & 0.575 & 0.446 \\ CF & 0.824 & 0.678 & 0.625 & 0.565 & 0.435 \\ LC-True & 0.848 & 0.736 & 0.673 & 0.575 & 0.507 \\ LC-Est & 0.84 & 0.725 & 0.66 & 0.57 & 0.466 \\ Rule-Remove & 0.842 & 0.742 & 0.664 & 0.591 & 0.491 \\ RW (\(\alpha\) = 0.02) & 0.835 & 0.739 & 0.666 & 0.585 & 0.466 \\ RW (\(\alpha\) = 0.1) & 0.835 & 0.744 & 0.68 & 0.592 & 0.505 \\ RW (\(\alpha\) = 0.4) & 0.835 & 0.744 & 0.688 & 0.597 & 0.516 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy of six algorithms at different noise levels on CIFAR-10.  
Table 3 shows the accuracy of the algorithms for CIFAR-10 dataset. The accuracy of six algorithms decreases as the level of noise is higher and the accuracy of all methods is nearly the same when there are no noisy samples in the training set. The accuracy of Rule-Weight (SW) and Rule-Remove are the highest, and the Classification Filtering (CF) can still be the worst method as the noise level increases. These results are consistent with the ones for Manga109.  
Table 5 shows the rate of label noise sample detected by the rules on CIFAR-10 dataset. The rules just can detect 23% to 29% of the data samples whose labels have been incorrectly assigned. That explain there was no significant improvement in the performance of our proposed methods to the normal one like in Manga109 dataset.