## 3 Experimental Verification
### Noise Generation  
In the experiments, we used all three noise models described in the introduction: uniform noise (UNI), class-dependent noise (CDN), and instance-dependent noise (IDN). The application of UNI noise to the tested databases was straightforward--a defined fraction of labels were randomly changed to another class. However, the application of CDN and IDN noise was more complex and is described in this section. To obtain the matrices for CDN noise application, we used the classification results of DNNs trained on full sets with accurate labels. The resulting confusion matrices for all datasets are presented in Figure 2. In contrast to some previous works, instead of using the confusion matrix corresponding to the final training epoch, we opted to use an average matrix containing the average probability of incorrect classifications across all 200 training epochs and all eight DNN architectures. A confusion matrix that represents only the final training epoch does not reflect the time required for the DNN to develop a high score. Some classes require many training epochs, while other more unique or distinctive classes will be \begin{table}
\begin{tabular}{c c c} \hline \hline
**Abbrev.** & **Full Name** & **Reference** \\ \hline DLA & Deep Layer Aggregation & [18] \\ DPN & Dual Path Networks & [19] \\ PreActResNet18 & Deep Residual Network & [20] \\ SimpleDLA & Deep Layer Aggregation & [18] \\ ResNeXt29 & Residual Network & [21] \\ MobileNetV2 & Inverted Residuals and & [22] \\  & Linear Bottlenecks Network & \\ RegNetY400 & Regular Network & [23] \\ VGG16 & Very Deep Network & [24] \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Experimental DNN architectures and their performance when applied to the CIFAR10 dataset. classified within few epochs. Therefore, we argue that misclassification behavior across the entire training process better reflects the probability of inaccurate labeling than the discrete results corresponding to only the final training epoch or a single DNN architecture. The evolution of misclassification probability for the three most common error types is shown in Figure 3. These errors displayed the largest probability of class switching. The spread of probabilities across each DNN architecture is shown by the faded regions. It can be observed that the probability stabilizes at approximately the 100 th epoch and that there is little variation in probability between the different architectures. This suggests similar types of misclassification errors result from each DNN model. Note that the probability scale varies for each dataset. For CIFAR10 and CIFAR100, the average probability of misclassification is within the range 10-20%, while for MNIST it is approximately 1%. This is because the images within the MNIST dataset are considerably easier to classify than those within the CIFAR10 and CIFAR100 datasets. Moreover, the GTSRB dataset produces a final error rate of zero--no errors are present. This is a further argument for the use of average probability in the place of final result. Despite the obvious errors present throughout the training process, the final result would indicate no possibility of misclassification for the GTSRB dataset. Instead, the average probability gives a more intuitive interpretation of misclassification and considers, for example, that a \"Speed limit 20 km/h\" image can be mistaken for a \"Speed limit 30 km/h\" image. Figure 3: The evolution of misclassification probability for the three most common error types in each investigated dataset. Figure 2: Confusion matrices corresponding to CDN noise generation. The colors correspond to the average confusion probability as observed over 200 epochs of training for all eight DNN architectures. We included the IDN noise model in our experiments as it has been identified as the most reliable such model [11]. When using this model, the probability of misclassification is calculated individually for each image in the database.