# Stochastic Weight Averaging Revisited
## 4 Periodic SWA  
### On the Global Geometric Structure of the DNN Loss Landscape  
As presented previously, we found cases in which SWA was initialized by an SGD that did not converge well, and SWA performed notably better than its backbone SGD. However, if the preceding SGD converged well, then the performance gap between SWA and its backbone SGD was reduced or even indistinguishable. As previously shown, when the preceding SGD converged well, the NN weights employed by SWA centered around a local optimum discovered by its backbone SGD. Since those NN weights were all close to this local optimum, they were close among each other. Therefore, based on these weights, the SWA operation could only employ a highly local geometric structure around the local optimum. On the contrary, when the preceding SGD did not converge well, corresponding to a larger product of the learning rate and the stochastic gradient, the NN weight samples fed into SWA were not close to each other, so they spanned a much wider area. This motivated us to form the following hypothesis:  
Is there any global geometric structure in the DNN loss landscape that could be encountered by an SGD during an early stage of its life cycle? If such a global structure exists, could it be exploited to facilitate the discovery of higher-quality local optima? We proposed a novel algorithm design, termed a periodic SWA (PSWA), that initialized SWA during an early stage of an SGD procedure. The PSWA exploited the aforementioned  
Figure 2: Ablation study on the CHC learning-rate schedule and the weight-averaging operation for DNNs that did not converge well. The legends are defined in the same way as in Figure 1. The sub-figures in the left/middle/right columns correspond to VGG16/PreResNet-164/WideResNet-28-10. The sub-figures in the top/bottom rows correspond to dataset CIFAR-10/CIFAR-100. possible global structures via performing weight-averaging sequentially. We describe the experimental results in the following section, which demonstrated that the PSWA outperformed its backbone SWA remarkably, thus providing evidence for the existence of such global geometric structures. The PSWA consisted of a series of SWA procedures that processed sequentially. The first SWA procedure was initialized by an NN weight provided by the backbone SGD that processed before SWA. For each of the other SWA procedures, its starting weight seed was the output of its previous SWA procedure. As opposed to the original SWA method, which was invoked when its previous SGD converged, the PSWA was initialized when its preceding SGD was in an early stage of its operation. A flowchart of PSWA is shown in Figure 3. Two special examples of PSWA, termed double SWA (DSWA) and triple SWA (TSWA), which consisted of two and three sequentially performed SWA procedures, are shown in Algorithm 2 and Algorithm 3, respectively. In addition, PSWA used a learning-rate schedule that was identical to its backbone SGD, as shown in Figure 11. If the sequentially performed SWA procedures could continually increase performance, as compared to the backbone SGD, then this would indicate that PSWA employed certain global structures of the loss landscape to search for the local optima. Figure 3: Block schema of PSWA. PSWA consisted of a backbone SGD that processed before the SWA operations and a series of SWA procedures that processed sequentially. The first SWA procedure was initialized by the output of the backbone SGD. For each of the other SWA procedures, its starting weight seed was the output of its previous SWA procedure. In contrast to the original SWA method, which was invoked when its preceding SGD converged, the PSWA was initialized when its preceding SGD was in an early stage of its operation. ```
0: weights \(\widehat{w}\), learning-rate schedule, cycle length \(c\), number of iterations \(n\) (assumed to be multiples of 2)
0:\(w_{\text{dswa}}\)
1: Run the SWA procedure (namely Algorithm 1) with input \(\widehat{w}\), \(c\), \(n/2\). Denote the output to be \(w_{\text{swa}}\).