# Methodology for Evaluating the Generalization of ResNet
## 3 Method
### IoU-Based Generalizability Evaluation Methodology  
Building upon the analyses presented in the preceding section, we identified a robust correlation between the spatial distribution of features within a model and its capacity for generalization. Consistent with our earlier observations, an increased alignment between the model's learned features and the spatial location of the category object within an image correlates positively with enhanced generalization performance, as indicated by a reduced generalization gap. In this research, we adopted the IoU metric to quantitatively evaluate the generalization capability of CNNs for image classification tasks, focusing on the ratio of task-relevant features. As illustrated in Figure 3, our methodology commences with the extraction of the positional distribution of the category-specific object, delineated by its bounding box. Subsequently, we generate class activation maps that illustrate the spatial emphasis of features within the model's focus. The model's learned feature distribution is then delineated on these class activation maps, employing a threshold to segment and isolate the features that are most pertinent to the classification task. The penultimate stage uses IoU to calculate the ratio of task-relevant features, which quantifies the spatial overlap between the model's learned feature distribution and the actual positions of the category objects. This IoU value serves as a proxy for the proportion of accurately localized, task-relevant features, thereby offering a metric for the model's proficiency in generalizing from the training dataset to novel scenarios. The above computational process can be formally defined as  
\[\text{IoU}^{\text{Img}}=\frac{\sum\mathbf{A}_{\text{L}}\cap\mathbf{A}_{\text{C}}}{ \sum\mathbf{A}_{\text{L}}\cup\mathbf{A}_{\text{C}}} \tag{2}\]  
Figure 2: Examples of distribution of features learned by models and category object locations (i.e., the green bounding box) with different generalization abilities. where \(A_{L}\) and \(A_{G}\), respectively, represent the feature distribution learned by the network and the position distribution of the category object. In the formula, \(A_{L}\) is the portion of the feature distribution that has a weight greater than the threshold value, and \(A_{G}\) is the rectangular bounding box of the corresponding category object in the ImageNet [17], and are, respectively, defined as follows:  
\[A_{L}(i,j) =I(\mathbf{M}^{c}(i,j),\rho) \tag{3}\] \[I(x,\rho) = \begin{cases}1\text{ if }x\geq\rho\\ 0\text{ if }x<\rho\end{cases}  \tag{4}\]  
\[A_{G}(i,j)=\left\{ \begin{array}{ll}1&(i,j)\text{{\it Inside the bounding box}}\\ 0&(i,j)\text{Outside the target box}\end{array} \right. \tag{5}\]  
where \(\mathbf{M}^{c}\in\mathbb{R}^{h\times w}\) represents the class activation map of the model relative to category c. Category c is the model's classification result of the input image. The value of each element in \(\mathbf{M}^{c}\) is normalized to \([0,1]\). In the experiments in this study, the threshold was 0.1. The value range of IoUImg was \([0,1]\). The larger the value, the higher the overlap between the feature distribution learned by the model and the location of the corresponding category object, which means a better the generalization ability of the model. \(A_{L}\) in Equation (3) is an irregular region similar to a circle, and we can similarly use its minimum outer rectangular box to calculate the IoU. In this paper, the calculation using the minimum outer rectangle is termed IoU-B. In our experiments, we also compared the performance of these two calculations. The IoU value calculated from a single image lacks statistical significance. In actual tasks, the IoUModel value at the model level is used as an indicator to judge the generalization ability of the model.