# A Histopathological Image Classification Method Based on Model Fusion in the Weight Space
## 3 Model Fusion in the Weight Space for Histopathological Image Classification  
### Basic Model Training  
As indicated by previous studies [16; 20; 25; 27], model fusion in the weight space works well in models from snapshots of the trajectory of an SGD procedure, or from fine-tuned pre-trained models with different hyperparameters. We applied both methods to generate basic models (ingredient models for soup) in this study. Specifically, we used three famous deep convolutional neural network architectures, i.e., ResNet [34], VGG [35], and DenseNet [36], for basic model training. All models were pre-trained on the ImageNet-1000 dataset [8] and, hence, our model works in the fine-tuning mode. The pre-trained models were fine-tuned with two training settings, i.e., the cyclical learning rate and different training hyperparameters. We saved the model weights in both cases as the potential ingredients for model fusion. A. Ingredient Model Training with Cyclical Learning Rate  
The main idea of the cyclical learning rate (CLR) [21; 23; 24], which is different from the traditional learning rate-adjusting strategy, which decreases monotonically, is to vary the learning rate cyclically within a proper range as the number of training epochs increases. Let \(\alpha_{1}\) and \(\alpha_{2}\) represent the lower and upper bounds of the learning rate; the CLR strategy changes the learning rate according to the iteration number \(k\), as shown in Equations (3) and (4):  
\[\alpha(k)=(1-t_{k})\alpha_{1}+t_{k}\alpha_{2} \tag{3}\]  
\[t_{k}=\frac{\text{mod }(k-1,c)+1}{c} \tag{4}\]  
In Equation (4), \(c\) represents the length of a cycle. Here, we refer to processing a mini-batch of data at a time as an epoch. At the end of each cycle, i.e., when the learning rate reaches the minimum, the model weights are saved as checkpoints. Note that \(\alpha_{1}\), \(\alpha_{2}\), and \(c\) are hyperparameters for model training, and have a great impact on the effectiveness of model weight averaging. The maximum learning rate \(\alpha_{2}\) must be able to jump out of a local minima basin, which determines the exploration ability of the model. In this study, we set \(\alpha_{2}\) to a slightly larger value than the initial learning rate used in the traditional learning rate decay training strategy. The minimum learning rate \(\alpha_{1}\) and the length of a cycle \(c\) are set according to [24]. B. Ingredient Model Training with Different Training Hyperparameters  
In addition to CLR, we also fine-tuned the pre-trained model with different (but constant) learning rates with data augmentation. For constant learning rates, we set the learning rate to 0.0001, 0.0015, and 0.0002, respectively. We used RandAugment [37] for data augmentation, which aims to reduce the parameter space and maintain the diversity of data (images) by using a non-parametric process instead of the learning strategy and probability. There are only two parameters involved when using RandAugment, i.e., the number of transformations \(N\) and the degree of augmentation distortion \(M\). RandAugment makes use of a linear scale to represent the intensity of each conversion. Simply put, each transformation is in the integer range of 0 to 10, where 10 represents the maximum range of a given transformation. The larger the values of \(N\) and \(M\), the stronger the degree of regularization. We set \(N=9\) and \(M=2\) in all of the experiments in this study. C. The Effectiveness of Soup Ingredients  
The effectiveness of model fusion in the weight space depends on whether the model weights are located in the same loss basin. To show that the ingredients obtained through our two training strategies meet the requirement of fusion in the weight space, we will first provide a formal definition of the loss basin, and then use a linear interpolation method for validation. We use the definition of loss basin proposed in [27], which can be formally written as follows:  
**Definition 1**.