# An Explainable Scheme for Memorization of Noisy Instances by Downstream Evaluation
## 4 Experimental Setup and Results
### Ablation Test of Fraction Removing from Training Data  
To further evaluate the impact of noisy images with high memorization scores in the training data on the test dataset, we conducted the following ablation test: Models were trained by progressively excluding images with high memorization scores (i.e., high importance) from the training data, and their effects on the test dataset were observed. The experiments utilized the STL-10, CIFAR-10, and CIFAR-100 datasets, which were mixed with noise sampled from normal and uniform distributions. Training data were filtered by removing the top 10% to 50% fractions of images based on their memorization scores in descending order. This approach was compared to randomly excluding 10% to 50% of the data, without considering their importance. The results of the experiment are shown in Figures 8-13. Figure 5: A dog image in CIFAR-10 with noises sampled from uniform distribution. Figure 6: A fish image in CIFAR-100 with noises sampled from normal distribution. Figure 7: A bug image in CIFAR-100 with noises sampled from uniform distribution. Figure 11: Ablation test for CIFAR-10 with noises sampled from uniform distribution. Figure 8: Ablation test for STL-10 with noises sampled from normal distribution. Figure 12: Ablation test for CIFAR-100 with noises sampled from normal distribution. Figure 10: Ablation test for CIFAR-10 with noises sampled from normal distribution. Figure 9: Ablation test for STL-10 with noises sampled from uniform distribution. From the overall observations, several key insights were identified:  
1. Across all datasets and noise sampling distributions, a consistent phenomenon is: Models trained by excluding training data in proportion to high memorization scores show a significantly greater drop in accuracy (as shown by the red lines) as the exclusion ratio increased, compared to models trained by randomly excluding data (as shown by the blue lines). This indicates that the absence of high-memorization-score images has a critical and detrimental impact on model learning, leading to a substantial decline in test accuracy. 2. A common phenomenon observed across all datasets is that, due to the higher diversity of noise sampled from a uniform distribution compared to a normal distribution, noise from the uniform distribution has a more harmful effect on the model under the same noise intensity. 3. An interesting phenomenon across all datasets is that, as the noise intensity increases, the red line tends to approach the blue line more closely when less than 10% of the training data is excluded. This trend is particularly noticeable when the noise intensity reaches 1.0. This occurs because data with exceptionally high memorization scores are mostly concentrated in the top 10% (as shown by the green lines). As the noise intensity increases, the top 10% of data with high memorization scores no longer entirely consist of the high memorization-score data from the original, noise-free dataset. Many of these data instances are likely originally low memorization-score data that, after being subjected to high-intensity noise, are misidentified by the model as high memorization-score data, thus being considered as good training data. Under such interference, the impact of the top 10% of important data on the model becomes less significant, resulting in performance that is nearly identical to randomly excluding 10% of the data. 4. Due to the feature complexity of the STL-10 dataset, it would likely be relatively simpler than that of the other two datasets, even after excluding a high fraction of high memorization-score training data, as the decrease in test accuracy is not as pronounced. In contrast, the CIFAR-100 dataset, with its higher number of classes and more complex features, experiences the greatest drop in test accuracy as the proportion of high memorization-score data excluded increases. The performance of the CIFAR-10 dataset falls between the two.