# Methodology for Evaluating the Generalization of ResNet
## 5 Experimental Analysis
### Experimental Results  
#### 5.2.1 Comparison Experiments  
In this section, we used the [PERSON]'s correlation coefficient of the constructed model set to verify the validity of the model generalizability assessment approach. This rigorous analysis ensures the effectiveness and accuracy of the assessment methodology. The IoU values of the feature distribution and object position for all models in the proposed model set, as well as the generalization gaps of each model, were computed in this study. For calculating the class activation maps, the Grad-CAM method implemented in Captum [26] was used. In this study, three comparison methods were selected, namely Spectral Norm [8], Nuclear Norm [21], and Effective Invariance (EI) [22]. Since the original Spectral Norm method cannot be directly applied to CNN models, the implementation approach described in reference [3] was used. The details of these methods can be found in Appendix B. Table 3 delineates the correlation coefficients between the generalization assessment methods and generalization gap across various ResNet architectures. Each row corresponds to a distinct ResNet architecture, with the final row summarizing the results across all the models. The columns indicate the model subsets with training accuracies exceeding specific thresholds. Ideally, all methods should correlate positively with a model's generalization capacity, which is inversely proportional to its generalization gap. Consequently, a negative correlation between the methods and generalization gaps is anticipated. The effectiveness of an assessment method is indicated by a smaller absolute correlation coefficient value. The most promising results within each category are emphasized in bold. The analysis from Table 3 reveals a strong correlation (absolute value > 0.8) between IoU and the IoU-B method and the generalization gap, e.g., the model's generalization capacity. Due to the potential noise introduced by using the minimum bounding box in the IoU-B calculations, the correlation between IoU-B and the generalization gap was weaker than that of IoU. Spectral Norm had almost no correlation with the generalization ability of the model, which means that this method is not suitable for generalizability assessments of deep neural network models. Nuclear Norm was the least computationally intensive and most efficient, but its correlation with the generalization ability of the model was weak. The correlation between EI and the generalization ability of the model was second only to the IoU-based method, but its computation was computationally intensive as it needs to process the input and its multiple transformed versions. In summary, the evaluation method in this paper using IoU to calculate the ratio of task-relevant features had the highest accuracy and is computationally efficient. As training accuracy thresholds rise, the correlation between the evaluation methods and model generalization is strengthened. This trend arises because lowering the threshold broadens the spectrum of model accuracies without altering the generalization ability range, thus complicating model differentiation. For instance, models with training accuracies of 96% and 86%, and respective test accuracies of 93% and 83%, yielded identical generalization gaps of 3%. However, achieving the same IoU value for these models is implausible due to the inherent differences in their class activation maps. Among models with varying architectures, the IoU's correlation with generalization ability was consistent. Notably, the ResNet18 model demonstrated significantly higher correlation coefficients, potentially attributable to its simpler architecture and enhanced class activation map generation. The ResNet101 model also showed slightly higher correlation coefficients than the ResNet34/50 models, which may be influenced by the limited sample size and associated randomness in the training process.