# Methodology for Evaluating the Generalization of ResNet
## 4 Model Set Construction  
### Construction Method  
To verify the effectiveness and accuracy of evaluating model generalization methods, a comprehensive model set is needed. The two existing model sets for the generalizability study were constructed on two simplified datasets, CIFAR and SVHN [1,3], which contain 756 and more than 10,000 CNN models, respectively. On the one hand, the image resolution in the CIFAR and SVHN datasets is only 32 \(\times\) 32 pixels, with low data dimensions. On the other hand, the models in these sets mostly consist of around 10 layers with a small scale, which has a gap with the practical application scenarios. To bridge this gap and achieve a more precise and exhaustive validation of generalization assessment methodologies, this study constructed a model set with more than 2000 CNN models trained on the ImageNet subset by systematically varying commonly used hyperparameters. The models were developed utilizing the ResNet framework, specifically employing ResNet18, ResNet34, ResNet50, and ResNet101 architectures, which are prevalent in practical applications. The model set constructed in this study is comparable to the above two model sets in terms of quantity. Moreover, the size of the models is larger, and the images used for training are of higher resolution (with an average resolution of 496 \(\times\) 387 pixels), aligning more closely with practical scenarios. Taking into account the constraints of training duration, the study selected a random subset of 20 categories from the ImageNet dataset for experimental analysis. The data encompass a diverse array of categories, including animals, everyday objects, musical instruments, food items, and transportation-related items. This diversity is further accentuated by the extensive variation in the sizes and shapes of the objects represented within the dataset. A detailed breakdown of the data categories and their respective quantities used for model training is delineated in Table 1. For the purpose of determining test accuracy, a sample of 50 images per category was utilized, culminating in a comprehensive evaluation across 1000 images. In the course of model training, we meticulously fine-tuned a suite of hyperparameters to generate a diverse array of CNN models. These hyperparameters, which encompass batch size, learning rate, optimization algorithm, regularization coefficients (weight decay), model architecture, data augmentation, and the utilization of pre-trained weights from the ResNet models on the ImageNet dataset, are widely recognized for their influence on the generalization capacity of machine learning models. Through an exhaustive exploration of various hyperparameter combinations, we successfully cultivated a spectrum of models with distinct generalization capabilities. The hyperparameters for tuning can be formally defined as \(p_{i}\), taking values from the set \(\Theta_{i}\), for i = 1, \(\ldots\), \(n\), and \(n\) denoting the total number of hyperparameter types. In our study, 7 hyperparameters were selected, so \(n\) = 7. The selected hyperparameters were as follows: \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Category** & **Number** & **Category** & **Number** \\ \hline Hair slide & 579 & Sleeping bag & 486 \\ German short-haired pointer & 493 & Koala & 560 \\ Eel & 500 & Tree frog & 683 \\ Brassiere & 487 & Tibetan mastiff & 859 \\ Ostrich & 512 & Gordon setter & 925 \\ Radiator & 450 & Standard schauzer & 450 \\ Flute & 533 & Green mamba & 472 \\ Tiger & 542 & Grey whale & 421 \\ Cockroach & 562 & Bagel & 545 \\ Speedboat & 504 & Water jug & 610 \\ \hline
**Total** & & **11,173** & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Data categories and quantity information used for model training. 1. **Batch size**: It determines the amount of data the model sees at each training step, which can influence the stability and diversity of the learning updates, thereby impacting the model's exposure to the overall data distribution.