# Stochastic Weight Averaging Revisited
## 5 Conclusions  
In this paper, we investigated the contributions of the weight-averaging operation and the cyclical or high-constant learning-rate scheduling to the SWA process. Through experiments on a broad range of NN architectures, we identified a link between SGD and the global loss landscape and developed a novel insight from statistical and geometric perspectives regarding SWA. Specifically, we found that SWA was useful because it provided a mechanism to combine the advantages of the weight-averaging operation and the CHC learning-rate schedule. The CHC learning-rate schedule discovered global-scale geometric structures, and weight-averaging exploited such structures. By leveraging SGD's behavior in its early training phase, we proposed a novel algorithm, periodic SWA, which proved to be capable of finding high-quality local optima more quickly than SGD. Although we covered a broad range of network architectures and different types of datasets in our experiments, our findings still lacked theoretical support and may not be applicable for all DNN tasks. However, our work may promote more theoretical and algorithmic research on demonstrating, discovering, and exploiting non-local geometric structures of DNN's loss landscape in the future. Conceptualization, B.L. ; methodology, B.L. ; software, H.G., J.J. and B.L. ; validation, H.G. and B.L. ; formal analysis, B.L. ; investigation, B.L. ; resources, B.L. ; data curation, H.G. and J.J.; writing--original draft preparation, B.L. ; writing--review and editing, B.L. ; visualization, H.G., J.J. and B.L. ; supervision, B.L. ; project administration, B.L. ; funding acquisition, B.L. All authors have read and agreed to the published version of the manuscript. This research was funded by Research Initiation Project of Zhejiang Lab grant number 2021 KB0 PI01 and Exploratory Research Project of Zhejiang Lab grant number 2022 RC0 AN02. Not applicable  
Appendix A.1.1 Experimental Settings for Results Reported in Section 3.1  
For the graph-classification task, we conducted our experiments on a public open-source dataset MUTAG, which has commonly been used for graph-classification tasks. See details about this dataset at [[https://paperswithcode.com/dataset/mutag](https://paperswithcode.com/dataset/mutag)]([https://paperswithcode.com/dataset/mutag](https://paperswithcode.com/dataset/mutag)) (accessed on 1 November 2021).We used [PERSON] [34] to train a GIN model for 300 epochs. We set the learning rate \(\alpha\) at 0.01 and used the default parameter setting for the exponential decay rates \(\beta_{1}\) and \(\beta_{2}\), namely let \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\), respectively. For SWA, it initialized at the 270 th epoch, using a constant learning rate of 0.02. For experiments on the text dataset MRPC (see details about this dataset at [[https://paperswithcode.com/dataset/mrpc](https://paperswithcode.com/dataset/mrpc)]([https://paperswithcode.com/dataset/mrpc](https://paperswithcode.com/dataset/mrpc)) (accessed on 1 November 2021)) the learning rate of SGD was fixed at \(10^{-4}\) during the first 20 epochs; then, it was linearly decreased to \(10^{-6}\) in the following 20 epochs and then was fixed at \(10^{-6}\) for the last 10 epochs. The momentum and the weight-decaying factors were set at 0.9 and 0.01, respectively. SWA was initialized at the 45 th epoch. For each epoch of SWA, the learning rate was linearly decreased from \(5\times 10^{-5}\) to \(5\times 10^{-6}\). Appendix A.1.2 Experimental Settings for Results Reported in Section 3.2  
For the image-classification experiments presented in Section 3.2, we considered two main cases, termed Case I and Case II, for each DNN architecture under consideration. In Case I, we processed SWA after a converged SGD. In Case II, we processed it after a non-converged SGD. We adopted the same type of learning-rate schedule as used in [5]. An example of learning-rate schedule we used is shown in Figure A1. This learning-rate schedule covered \(L=160\) epochs in total.