# Stochastic Weight Averaging Revisited
## 3 Main Results
### Does SWA Always Identify Wider Optima Than SGD? The results reported in the SWA paper [5] showed that SGD generally converged to a boundary of a wide basin region and SWA assisted in finding an optimum exactly located in that wide basin region. All experiments conducted there used image datasets, such as CIFAR-\(\{10,100\}\)[22] and ImageNet ILSVRC-2012 [23; 24]. We speculated whether SWA would always lead to a better generalization than an SGD. We conducted experiments on graph- and text-based datasets.The results showed that the answer was no, which further indicated that for these cases, SWA could not locate wider optima than an SGD. Specifically, on the graph dataset MUTAG, we used SWA to train a graph isomorphism network (GIN) for graph classification. The baseline optimizer selected was Adam, which is an advanced SGD method that performs better for graph-data-based tasks. We found that if we applied Adam with 300 epochs, then we obtained a test accuracy value of 89%; however, when we replaced Adam with SWA for the last 30 epochs, we obtained a smaller test accuracy value of 84%. We considered the graph-node-classification task using graph-neural-network (GNN) models, such as a graph convolutional network (GCN) [25], GraphSAGE [26], and a graph attention network (GAT) [27], on public open-source datasets Cora, Citeseer, and Pubmed. The parameter setting for the experiment is shown in Table A1 in Appendix A.4. The test accuracy comparison result is presented in Table 2.  
We also considered a graph-classification task using models MinCutPool [28] and SAGPool [29], on public open-source datasets NCII ([[https://paperswithcode.com/dataset/nci1](https://paperswithcode.com/dataset/nci1)]([https://paperswithcode.com/dataset/nci1](https://paperswithcode.com/dataset/nci1)), accessed on 1 November 2021), D&D ([[https://paperswithcode.com/sota/graph-classification-on-dd](https://paperswithcode.com/sota/graph-classification-on-dd)]([https://paperswithcode.com/sota/graph-classification-on-dd](https://paperswithcode.com/sota/graph-classification-on-dd)), accessed on 1 November 2021), and PROTEINS ([[https://paperswithcode.com/sota/graph-classification-on-proteins](https://paperswithcode.com/sota/graph-classification-on-proteins)]([https://paperswithcode.com/sota/graph-classification-on-proteins](https://paperswithcode.com/sota/graph-classification-on-proteins)), accessed on 1 November 2021). The parameter settings are shown in Tables A2-A4 in Appendix A.4. The test accuracy comparison result is shown in Table 3.  
The experimental results for the graph datasets showed that using SWA did not always lead to a better generalization than other advanced SGD optimizers, such as Adam. On a text dataset termed Microsoft Research Paraphrase Corpus (MRPC), we used an SGD with momentum to fine-tune the pre-trained model RoBERTa for testing whether two sentences were semantically equivalent. See details about the experimental setting in Appendix A.1.1. On average, the SGD provided a test accuracy value of 87.98% while SWA only achieved 87.50%. We found that, even for image datasets, an SGD did not always converge to a boundary of a wide basin region, especially when we removed the momentum module from its backbone SGD. In such cases, we found that SWA could converge to a deep loss valley, where the averaged gradients over mini-batch training samples were all close to zero. Then, the products of such gradients and the learning rate were close to zero. In such cases, SWA failed to find a wider optimum with better generalization. See details of the experimental results in Appendix A.3. To understand why removing the momentum module could result in SWA failing to find wider optima, we reviewed the working mechanism of the momentum technique. As we know, a momentum optimizer accelerated an SGD by adding a fraction of the update vector of the past iterations to the current update vector.