Question: Why is MobileNet performing the best across the 3 models in Figure 2?
Evidence: To assess the relationship between the hardness degree of samples and the misclassification rate, we compute the hardness degree of CIFAR10 and CIFAR100 test samples and then partition them into ten groups based on their hardness degree. It is important to note that the number of samples in each group is different. Afterward, we calculate what percentage of samples in each group is classified incorrectly. Figure 6 demonstrates that the misclassification rate is increased by increasing the hardness degree of samples. In other words, there is a strong positive correlation between the hardness degree of samples and the misclassification rate. The figure indicates the percentage of samples in each hardness degree range by a green curve. For example, the hardness degree of 40.65% of CIFAR10 test samples (4065 samples) is in the range [0,9] for MobileNet classifier, from which 99.88% is classified correctly, or the hardness degree of 7.4% of CIFAR10 test samples (740 samples) is in the range [90,99] for MobileNet classifier, from which 55.27% is classified correctly. More than 99% and 95% of samples being learned in the first 30 epochs (hardness degree < 30) are correctly classified in CIFAR10 and CIFAR100 test samples, respectively. On the other side, less than 55% and 36% of samples being learned in the last 10 epochs (hardness degree ≥ 90) are correctly classified in CIFAR10 and CIFAR100 test samples, respectively.
Answer: We first want to mention that since Figure 2 is not directly relevant to the main purpose of the paper, we have moved that figure to Appendix B and changed the figure to avoid misunderstanding. As mentioned in the paper, calculating the hardness degree of samples does not require the true label of samples. Nevertheless, since we had access to the true labels of CIFAR10 and CIFAR100 test samples, we depict Figure 2 (Figure 6) to give more intuition about the hardness degree of samples. To depict this figure, we first calculate the hardness degree of CIFAR10 and CIFAR100 test samples and then partition them into ten groups based on their hardness degree. Notably, the number of samples in each group is different. Afterward, we calculate what percentage of samples in each group is classified incorrectly. The figure demonstrates that the misclassification rate is increased by increasing the hardness degree of samples. In other words, there is a strong positive correlation between the hardness degree of samples and the misclassification rate.
Although the curve of MobileNet is upper than two other classifiers in Figure 2, this does not mean that the MobileNet is the best classifier across the three models. The number of samples in each range of hardness degree is different between models. Given Figure 1, CIFAR10 and CIFAR100 test samples are relatively harder for MobileNet classifiers than two other classifiers. Since models misclassify harder samples with higher probability, and since the number of harder samples is more for the MobileNet classifier, this classifier has the worst performance across models. Figure 6 indicates the percentage of samples in each hardness degree range by a green curve. We have added an example to Appendix B to clarify Figure 6.