Year: 2021

Abstract:
The success of deep ensembles on improving predictive performance, uncertainty, and out-of-distribution robustness has been extensively demonstrated in the machine learning literature. Albeit the promising results, naively training multiple deep neural networks and combining their predictions at test lead to prohibitive computational costs and memory requirements. Recently proposed efficient ensemble approaches reach the performance of the traditional deep ensembles with significantly lower costs. However, the training resources required by these approaches are still at least the same as training a single dense model. In this work, we draw a unique connection between sparse neural network training and deep ensembles, yielding a novel efficient ensemble learning framework called FreeT ickets. Instead of training multiple dense networks and averaging them, we directly train sparse subnetworks from scratch and extract diverse yet accurate subnetworks during this efficient, sparse-to-sparse training. Our framework, FreeT ickets, is defined as the ensemble of these relatively cheap sparse subnetworks. Despite being an ensemble method, FreeT ickets has even fewer parameters and training FLOPs compared to a single dense model. This seemingly counter-intuitive outcome is due to the ultra training efficiency of dynamic sparse training. FreeT ickets improves over the dense baseline in the following criteria: prediction accuracy, uncertainty estimation, out-of-distribution (OoD) robustness, and training/inference efficiency. Impressively, FreeT ickets outperforms the naive deep ensemble with ResNet50 on ImageNet using around only 1/5 training FLOPs required by the latter.

text: The success of deep ensembles on improving predictive performance, uncertainty, and out-of-distribution robustness has been extensively demonstrated in the machine learning literature. Albeit the promising results, naively training multiple deep neural networks and combining their predictions at test lead to prohibitive computational costs and memory requirements. Recently proposed efficient ensemble approaches reach the performance of the traditional deep ensembles with significantly lower costs. However, the training resources required by these approaches are still at least the same as training a single dense model. In this work, we draw a unique connection between sparse neural network training and deep ensembles, yielding a novel efficient ensemble learning framework called FreeT ickets. Instead of training multiple dense networks and averaging them, we directly train sparse subnetworks from scratch and extract diverse yet accurate subnetworks during this efficient, sparse-to-sparse training. Our framework, FreeT ickets, is defined as the ensemble of these relatively cheap sparse subnetworks. Despite being an ensemble method, FreeT ickets has even fewer parameters and training FLOPs compared to a single dense model. This seemingly counter-intuitive outcome is due to the ultra training efficiency of dynamic sparse training. FreeT ickets improves over the dense baseline in the following criteria: prediction accuracy, uncertainty estimation, out-of-distribution (OoD) robustness, and training/inference efficiency. Impressively, FreeT ickets outperforms the naive deep ensemble with ResNet50 on ImageNet using around only 1/5 training FLOPs required by the latter.

heading: 1 INTRODUCTION
text: Ensembles (Hansen & Salamon, 1990; Levin et al., 1990) of neural networks have received large success in terms of the predictive accuracy (Perrone & Cooper, 1992; Breiman, 1996; Dietterich, 2000; Xie et al., 2013), uncertainty estimation (Fort et al., 2019; Lakshminarayanan et al., 2017; Wen et al., 2020; Havasi et al., 2021), and out-of-distribution robustness (Ovadia et al., 2019a; Gustafsson et al., 2020). Given the fact that there are a wide variety of local minima solutions located in the high dimensional optimization problem of deep neural networks and various randomness (e.g., random initialization, random mini-batch shuffling) that occurs during training, neural networks trained with different random seeds would converge to different low-loss basins with similar error rates (Fort et al., 2019; Ge et al., 2015; Kawaguchi, 2016; Wen et al., 2019; 2020). Deep ensembles achieve large performance boost by simply averaging or voting on the predictions of these networks.
Despite the promising performance improvement, the traditional deep ensemble naively trains multiple deep neural networks independently and ensemble them, whose training and inference cost increases linearly with the number of ensemble members. Recent works on efficient ensembles, e.g., BatchEnsemble (Wen et al., 2020), hyper batch ensembles (Wenzel et al., 2020), and MIMO (Havasi et al., 2021), enable to reach the performance of dense ensemble with only negligible extra computation compared with a single dense model. However, the training resources required by these approaches are still at least the same as training a single dense model. Since the size of advanced deep neural networks is inevitably exploding (Touvron et al., 2020; Dosovitskiy et al., 2021; Brown et al., 2020; Touvron et al., 2021), the associated enormous training costs are potentially beyond the reach
of most researchers and startups, leading to financial and environmental concerns (Garcı́a-Martı́n et al., 2019; Schwartz et al., 2019; Strubell et al., 2019).
On the other hand, to accelerate training, researchers explored the possibility of directly training sparse neural networks from scratch (sparse training) (Mocanu et al., 2016; Liu et al., 2020a; Evci et al., 2019), while trying to maintain comparable performance. However, training a sparse network from scratch typically results in worse performance than their dense counterparts (Kalchbrenner et al., 2018; Evci et al., 2019). The current only way for matching performance while keeping the network sparse throughout training is Dynamic Sparse Training (DST) (Mocanu et al., 2018; Bellec et al., 2018; Evci et al., 2020a; Liu et al., 2021b). DST seeks to train sparse neural networks from scratch with a strict training budget. It starts from a randomly-initialized sparse network and optimizes the model weights together with the sparse connectivity during training. The current only way for DST to match the performance of its dense counterpart on the popular benchmark, e.g., ResNet-50 on ImageNet, is to extend the training time (Evci et al., 2020a), which might require thousands of training epochs for extremely sparse models (Liu et al., 2021b).
In this paper, we attempt to address the above-mentioned two challenges jointly by drawing a unique connection between sparse training and deep ensembles. Specifically, we ask the following question:
Instead of allocating all resources to find a strong winning ticket, can we find many weak tickets with very low costs (free tickets), where the combination of these free tickets can significantly outperform the single dense network, even the dense ensemble?
Note that it is not trivial to obtain free tickets. To guarantee superior ensemble performance, three key desiderata the free tickets are expected to satisfy (1) high diversity: according to the ensemble theory (LeCun et al., 2015; Hansen & Salamon, 1990; Ovadia et al., 2019b), higher diversity among ensemble members leads to higher predictive performance; (2) high accessibility: free tickets should be cheap to obtain so that the overall training cost does not compromise too much; and (3) high expressibility: the performance of each free ticket should be comparable with the dense model.
Leveraging the insight from Liu et al. (2020b) that a full network contains a plenitude of performative subnetworks that are very different in the topological space, we introduce the concept ofFreeT ickets, an efficient ensemble framework that utilizes sparse training techniques to generate subnetworks for the ensemble. Furthermore, we instantiate FreeT ickets by proposing two efficient ensemble methods – Dynamic Sparse Training Ensemble (DST Ensemble) and Efficient Dynamic Sparse Training Ensemble (EDST Ensemble). Both methods yield diverse subnetworks that consummately satisfy the above-mentioned criteria. We summary our contributions below:
• Our first method, DST Ensemble, independently trains multiple subnetworks from scratch with dynamic sparsity. By averaging the predictions of these subnetworks, DST Ensemble improves the predictive accuracy, OoD robustness, uncertainty estimation, and efficiency over the traditional dense ensemble.
• Our second, light-weight method ( EDST Ensemble) yields many free tickets in one single run, which is more efficient to train and inference than a single dense model, while approaching the performance of the traditional dense ensemble.
• We analyze the diversity of the individual subnetworks generated by our methods and confirm the effectiveness of our methods on inducing model diversity.
• Our results suggest that besides the training/inference efficiency, sparse neural networks enjoy also other favorable properties which are absent in dense networks (robustness, out-ofdistribution generalization, etc), opening the path for new research directions.

heading: 2 RELATED WORKS
text: Efficient Ensembles. One major drawback of ensembles is the expensive computational and memory costs for both training and testing. To address this problem, various approaches have been proposed. TreeNet (Lee et al., 2015) shares weights in earlier layers and splits the following model into several branches, improving accuracy over the dense ensemble. Monte Carlo Dropout (Gal & Ghahramani, 2016) can be used to approximate model uncertainty in deep learning without sacrificing either computational complexity or test accuracy. BatchEnsemble (Wen et al., 2020) proposed to improve parameter efficiency by decomposing the ensemble members into the product of a shared matrix and a rank-one matrix personalized for each member. MIMO (Havasi et al., 2021) uses a multi-input multi-output configuration to concurrently discover subnetworks that co-habit the dense network without explicit separation. Snapshot (Huang et al., 2017) and FGE (Garipov et al., 2018) discover diverse models by using cyclical learning rate schedules. Furlanello et al. (2018) applied knowledge distillation (Hinton et al., 2015) to train several generations of dense students. The ensemble of the dense students outperforms the teacher model significantly. Other related works include but are not limited to hyper-batch ensembles (Wenzel et al., 2020) and Late Phase (Oswald et al., 2021). However, the training resources required by these methods are still at least the same as training a single dense model. In contrast to the existing efficient ensemble methods, our methods (EDST Ensemble) can match the performance of naive ensemble with only a fraction of the resources required by training a single dense network.
Dynamic Sparse Training. Dynamic Sparse Training (DST) is a class of methods that enables training sparse neural networks from scratch by optimizing the sparse connectivity together with the weight values during training. DST stems from Sparse Evolutionary Training (SET) (Mocanu et al., 2018), a sparse training algorithm that outperforms training a static sparse model from scratch (Mocanu et al., 2016; Evci et al., 2019). Weight reallocation was further proposed in Mostafa & Wang (2019); Dettmers & Zettlemoyer (2019); Liu et al. (2021a) to reallocate new weights across layers for better layer-wise sparsity. Further, Dettmers & Zettlemoyer (2019); Evci et al. (2020a) leverage the gradient information in the backward pass to guide the optimization of sparse connectivity and demonstrate substantial performance improvement. Some recent works (Jayakumar et al., 2020; Raihan & Aamodt, 2020; Liu et al., 2021b) demonstrate that a large range of exploration in the parameter space is important for dynamic sparse training. Price & Tanner (2021) improved the performance of DST by adding additional non-trainable parameters. DST has also demonstrated its strength in feature detection (Atashgahi et al., 2020), lifelong learning (Mallya & Lazebnik, 2018; Sokar et al., 2021), federated learning (Zhu & Jin, 2019), and adversarial training (Özdenizci & Legenstein, 2021). Sparse MoE (Shazeer et al., 2017; Fedus et al., 2021) sparsely activates one of the few expert networks to increase the model capacity – but with a constant computational cost. It usually requires specialized modules, such as gating and selector networks to perform the sparsification.

heading: 3 FREETICKETS

heading: 3.1 PRELIMINARIES
text: Dynamic Sparse Training. Let’s consider an i.i.d. classification setting with data {(xi, yi)}Ni=1, where x usually denotes input samples and y is the corresponding labels. For a network f(x;θ) parameterized by θl ∈ R in each layer of l ∈ {1, . . . ,L}, we train f(x;θ) to solve the following optimization problem: θ̂ = arg minθ ∑N i=1 L(f(xi;θ), yi).
Dynamic sparse training (DST) starts with a randomly-initialized sparse neural network f(x;θs), parameterized by a fraction of parameters θs. The sparsity level of the sparse model is pre-defined as S = 1− ‖θs‖0‖θ‖0 , where ‖ · ‖0 is the `0-norm. The goal of DST is to yield a sparse network with the target sparsity S after training, while maintaining the overall computational and memory overheads close to training a static sparse model (fixed sparse connectivity).
During training, DST continuously minimizes the loss ∑N
i=1 L(f(xi;θs), yi), while periodically exploring the parameter space for better sparse connectivity with non-differentiable heuristics. A
simple yet effective exploration heuristics is prune-and-grow, that is, pruning p percentage of the unimportant weights from θs, followed by regrowing the same number of new weights. By repeating this prune-and-grow cycle, DST continuously searches for better sparse connectivities while stick to a fixed parameter budget. See Appendix A for a general pseudocode and full details of DST, where a brief literature review of DST is provided.
FreeTickets Ensemble. We propose the concept of FreeT ickets here. FreeT ickets refers to efficient ensemble methods that utilize DST to generate subnetworks for ensemble. A free ticket is a converged subnetwork with its specific sparse connectivity. These free tickets {θ1s , θ2s , . . . θMs }, observed and collected either within one training run (EDST Ensemble) or multiple training runs (DST Ensemble), are further used to construct the FreeT ickets Ensemble. Assuming that the output probability of the kth neuron of the jth free ticket is given by p(ajk). Then the output of the corresponding kth neuron in the ensemble can been obtained by taking the average across all the M subnetworks, i.e., 1M ∑M j=1 p(a j k).
Compared with the existing efficient ensemble techniques (Huang et al., 2017; Wen et al., 2020), FreeT ickets induces diversity from the perspective that there exists many performative subnetworks with very different sparse connectivities within the full network (Liu et al., 2020b). The efficiency of FreeT ickets comes from the fact that each subnetwork is sparse from the beginning, so that the memory and floating-point operations (FLOPs) required by FreeT ickets can be even fewer than training a single dense network. To instantiate the concept of FreeT ickets, we introduce two DST-based ensemble methods, DST Ensemble and EDST Ensemble, as described below.

heading: 3.2 DST ENSEMBLE
text: Dynamic Sparse Training Ensemble (DST Ensemble) is presented in Algorithm 2, Appendix B. DST Ensemble takes advantage of the training efficiency from DST and independently trains M sparse networks with DST from scratch. By averaging the predictions of each sparse neural network, DST Ensemble can improve the predictive accuracy and uncertainty significantly. The diversity of DST ensemble comes from the fact that each ensemble member is trained independently with fully random initializations, random stochastic gradient descent (SGD) noise, etc. Moreover, each DST run converges to different sparse connectivities, promoting even higher diversity over the naive dense Ensemble. The efficiency of DST Ensemble comes from the fact that all the subnetworks are sparse throughout training, so that the training costs are much fewer than training a dense network.
We choose the advanced DST method the Rigged Lottery (RigL) (Evci et al., 2020a) for DST Ensemble. RigL contains three main steps: sparse initialization, model weight optimization, and parameter exploration.
Sparse Initialization. Each subnetwork is randomly initialized with the Erdős-Rényi-Kernel (ERK) (Mocanu et al., 2018; Evci et al., 2020a) distribution at sparsity of S. The sparsity level of layer l is scaled with 1− n l−1+nl+wl+hl
nl−1×nl×wl×hl , where n l refers to the number of neurons/channels of
layer l; wl and hl are the width and the height of the convolutional kernel in layer l. ERK allocates higher sparsities to the layers with more parameters than the layers with fewer parameters.
Model Weight Optimization. After initialization, the activated weights are optimized with the standard optimizer SGD with momentum (Sutskever et al., 2013; Polyak, 1964) and the non-activated weights are forced to zero.
Parameter Exploration. After every ∆T iterations of training, we perform parameter exploration once to optimize the sparse connectivity. More concretely, we first prune p-proportion of weights from θs with the smallest magnitude: θ′s = TopK(|θs|, 1− p) (1) where TopK(v, k) returns the weight tensor retaining the top k-proportion of elements from v. Immediately after pruning, we grow the same number of new weights back with the highest magnitude gradients:
θs = θ ′ s ∪ TopK(|gi/∈θ′s |, p) (2)
where gi/∈θ′s are the gradients of the zero weights. We follow the suggestions in Liu et al. (2021b) and choose p = 0.5 and large update interval (T = 1000 for CIFAR-10/100 and T = 4000 for ImageNet to encourage an almost full exploration of all network parameters.

heading: 3.3 EDST ENSEMBLE
text: While efficient, the number of training runs (complete training phases) required by DST Ensemble increases linearly with the number of subnetworks, leading to extra devices and resource requirements. To further reduce the training resources and the number of training runs, we propose Efficient Dynamic Sparse Training Ensemble (EDST Ensemble) to yield many diverse sparse neural networks in one training run. The overall training procedure of EDST Ensemble is summarized in Algorithm 3.
The challenge of producing many free tickets serially in one training run is how to escape the current local basin during the typical DST training. Here, we force the model to escape the current basin by significantly changing a large fraction of the sparse connectivity, like adding significant perturbations to the model topology. The training procedure of EDST Ensemble is one end-to-end training run consisting of one exploration phase followed by M consecutive/sequential refinement phases. The M refinement phases are performed sequentially one after another within one training run.
Exploration phase. We first train a sparse network with DST using a large learning rate of 0.1 for a time of tex. The goal of this exploration phase is to explore a large range of the parameter space for a potentially good sparse connectivity. Training with a large learning rate allows DST to validly search a larger range of the parameter space, as the newly activated weights receive large updates and become more competitive at the next pruning iteration.
Refinement phase. After the exploration phase, we equally split the rest of training time by M to collect M free tickets.At each refinement phase, the current subnetwork is refined from the converged subnetwork in the previous phase (the first subnetwork is refined from the exploration phase) and then trained with learning rates 0.01 followed by 0.001 for a time of tre. Once the current subnetwork is converged, we use a large exploration rate q = 0.8 and larger learning rates 0.01 to force the converged subnetwork to escape the current basin. We repeat this process several times until the target number of free tickets is reached. The number of subnetworks M that we obtain at the end of training is given by M = ttotal−textre .
The purpose of the large parameter exploration in line 20 of Algorithm 3 is to force the subnetwork to escape the current basin. Therefore, the exploration rate q in line 20 is much larger than the one used in the standard DST parameter exploration (refer to p in line 6 & 15). See Appendix I and Appendix J for the effect of q and different regrowth criteria on EDST Ensemble, respectively.
Different from DST Ensemble, the diversity of EDST Ensemble comes from the different sparse subnetworks (sparse connectivities together with the model weights) the model converges during each refinement phase. The number of training FLOPs required by EDST Ensemble is significantly smaller than training an individual dense network, as DST is efficient and the exploration phase is only performed once for all ensemble learners.

heading: 4 EXPERIMENTAL RESULTS
text: In this section, we demonstrate the improved predictive accuracy, robustness, uncertainty estimation, and efficiency achieved by FreeT ickets. We mainly follow the experimental setting of MIMO (Havasi et al., 2021), shown as follows.
Baselines. We compare our methods against the dense ensemble, and various state-of-the-art efficient ensemble methods, including MIMO (Havasi et al., 2021), Monte Carlo Dropout (Gal & Ghahramani, 2016), BatchEnsemble (Wen et al., 2020), TreeNet (Lee et al., 2015), Snapshop (Huang et al., 2017), and FGE (Garipov et al., 2018). To highlight the fact that the free tickets are non-trivial to obtain, we further implement three sparse network ensemble methods: Static Sparse Ensemble (naively ensemble M static sparse networks), Lottery Ticket Rewinding Ensemble (LTR Ensemble; Frankle et al. (2020)) (ensemble M winning tickets1 trained with the same mask but different random seeds), and pruning and fine-tuning (PF Ensembles; Han et al. (2015)). While Static Sparse Ensemble can be diverse and efficient, it does not satisfy high expressibility property (Evci et al., 2019). LTR Ensemble suffers from low diversity and prohibitive costs. PF Ensemble requires at least the same training FLOPs as the dense ensemble. See Appendix C for the implementation and hyperparameter details.
Architecture and Dataset. We evaluate our methods mainly with Wide ResNet28-10 (Zagoruyko & Komodakis, 2016) on CIFAR-10/100 and ResNet-50 (He et al., 2016) on ImageNet.
1We rewind the tickets to the weights at 5% epoch as used in Frankle et al. (2020); Chen et al. (2020).
Metrics. To measure the predictive accuracy, robustness, and efficiency, we follow the Uncertainty Baseline2 and report the overall training FLOPs required to obtain all the subnetworks including forward passes and backward passes. See Appendix D for more details.
Results. The metrics on the CIFAR-10/100 and ImageNet are reported in Table 1, Table 2, and Table 3, respectively. For a fair comparison, we mainly set M of our methods the same as the one used in MIMO. See Appendix E for the comparison between Snapshot and FGE with our methods.
With multiple training runs, DST Ensemble consistently outperforms other efficient ensemble methods, even the dense ensemble on accuracy, robustness, and uncertainty performance, while using only 1/4 training FLOPs compared with the latter. When the number of training run is limited to 1, EDST Ensemble consistently outperforms the single dense model by a large margin, especially in terms of accuracy, with only 61% training FLOPs. Moreover, we observe that the performance of EDST Ensemble can further be improved by increasing the sparsity level. For instance, with a large sparsity level S = 0.9, EDST Ensemble can collect 7 subnetworks, more than twice as many as S = 0.8. Combining the prediction of these sparser subnetworks boosts the performance of EDST Ensemble towards the dense ensembles with only 57% training FLOPs, beyond the reach of any efficient ensemble methods. More impressively, DST Ensemble achieves the best performance on uncertainty estimation and OoD robustness among various ensemble methods.
2https://github.com/google/uncertainty-baselines
As we expected, Static Sparse Ensemble and LTR Ensemble consistently have inferior ensemble performance compared with DST Ensemble. While PF Ensemble achieves comparable performance with DST Ensemble, its costly procedure requires more than triple the FLOPs of DST Ensemble.
We argue that Lottery Tickets Hypothesis is not suitable for FreeT ickets since (1) it starts from a dense network, with the costly iterative train-prune-retrain process. Hence it is more expensive than training a dense network, in contrary to our pursuit of efficient training (our method sticks to training sparse networks end-to-end); (2) The series of sparse subnetworks yielded by iterative pruning is not diverse enough, since the latter sparse masks are always pruned from and thus nested in earlier masks (Evci et al., 2020b). Evci et al. (2020b) show that the different runs of winning tickets discovered by LTR are always located in the same basin as the pruning solution. Consequently, the ensemble of LTs would suffer from poor diversity, leading to poor ensemble performance due to the lack of diversity. We observe a very similar pattern in Section 5.1 as well. The diversity and performance of LTR Ensemble is lower than DST-based ensembles, highlighting the importance of dynamic sparsity in sparse network ensembles.

heading: 5 FREE TICKETS ANALYSIS

heading: 5.1 DIVERSITY ANALYSIS
text: According to the ensembling theory (LeCun et al., 2015; Hansen & Salamon, 1990; Ovadia et al., 2019b), more diversity among ensemble learners leads to better predictive accuracy and robustness. We follow the methods used in Fort et al. (2019) and analyze the diversity of the subnetworks collected by our methods in the function space.
Concretely, we measure the pairwise diversity of the subnetworks collected by our methods on the test data. The diversity is measured by Dd = E [d (P1(y|x1, · · · , xN),P2(y|x1, · · · , xN))] where d(·, ·) is a metric between predictive distributions and (x, y) are the test set. We use two metrics (1) Prediction Disagreement, and (2) Kullback–Leibler divergence.
Prediction Disagreement. Prediction Disagreement is defined as the fraction of test data the the prediction of models disagree on: ddis(P1,P2) = 1N ∑N i=1(arg maxŷi P1(ŷi) 6= arg maxŷi P2(ŷi)).
Kullback–Leibler Divergence. Kullback–Leibler Divergence (KL-divergence) (Kullback & Leibler, 1951) is defined as: dKL(P1,P2) = EP1 [logP1(y)− logP2(y)], a metric that is widely used to describe the diversity of two ensemble learners (Fort et al., 2019; Havasi et al., 2021).
We report the diversity among various sparse network ensembles in Table 2. We see that DST Ensemble achieves the highest diversity among various sparse ensembles, especially on CIFAR-100 where its diversity even surpasses the deep ensemble. Even though EDST Ensemble produces all subnetworks with one single run, its diversity approaches other multiple-run methods. The diversity comparison with non-sparse ensemble methods is reported in Figure 3-Right. Again, the subnetworks learned by DST Ensemble are more diverse than other methods and the diversity of EDST Ensemble is very close to the diversity of the dense ensemble, higher than the other efficient ensemble methods e.g., TreeNet and BatchEnsemble.
To better understand the source of disagreement, we apply KL divergence across intermediate feature maps learned by different subnetworks in Figure 3-Left. Each line is the averaged KL divergence among M = 3 subnetworks. It is interesting to observe that the KL divergence is quite high at early layers for the DST Ensemble and the Static Ensemble, which implies that the source of their high diversity might locate in the early layers. We further provide a disagreement breakdown for the first ten classes in CIFAR-100 between different subnetworks in Appendix G. The overall breakdown of disagreement is very similar to the diversity measurement results, i.e., DST Ensemble > Static Ensemble > EDST Ensemble > LTR Ensemble. We share heatmaps of prediction disagreement of DST/EDST ensemble in Appendix F.

heading: 5.2 TRAINING TRAJECTORY
text: In this section, we use t-SNE (Van der Maaten & Hinton, 2008) to visualize the diversity of sparse networks discovered by our methods in function space. We save the checkpoints along the training trajectory of each subnetwork and take the softmax output on the test data of each checkpoint to represent the subnetwork’s prediction. t-SNE maps the prediction to the 2D space. From Figure 4 we can observe that subnetworks of DST Ensemble converge to completely different local optima, and the local optima converged by EDST Ensemble are relatively closer to each other. Interestingly, even the last two subnetworks of EDST Ensemble (blue and green lines) start from similar regions, they end up in two different optima.

heading: 5.3 ABLATION STUDY OF PARAMETER EXPLORATION
text: We conduct an ablation study to analyze the effect of parameter exploration on promoting the model diversity. For comparison, we train the same number of subnetworks without any parameter exploration. Shown in Table 5, without parameter exploration, the diversity and the ensemble accuracy consistently decreases, highlighting the effectiveness of parameter exploration on inducing diversity.

heading: 5.4 EFFECT OF SPARSITY
text: We further study the effect of subnetworks sparsity on the ensemble performance and diversity, shown in Figure 6. While the individual networks learned by DST Ensemble and EDST Ensemble achieve similar test accuracy, the higher diversity of DST Ensemble boosts its ensemble performance over the EDST Ensemble significantly, highlighting the importance of diversity for ensemble. What’s more, it is clear to see that the pattern of the ensemble accuracy is highly correlated with the accuracy of the individual subnetwork. This observation confirms our hypothesis in the introduction, that is, high expressibility is also a crucial desideratum for free tickets to guarantee superior ensemble performance. We further support our hypothesis with the Pearson correlation (Pearson, 1895) and the
Spearman correlation (Powers & Xie, 2008) between the accuracy of individual networks and their ensemble accuracy in Appendix H.

heading: 5.5 EFFECT OF ENSEMBLE SIZE
text: In this section, we analyze the effect of the ensemble size (the number of ensemble learners) on our methods. For EDST Ensemble, we fix the total training time and vary the total number of ensemble learners M. The larger ensemble size M leads to shorter training time of each ensemble learner. For DST Ensemble, we simply train a different number of individual sparse models from scratch and report the ensemble performance.
The results are shown in Figure 7. “Individual Subnetworks” refers to the averaged accuracy of single subnetworks learned by DST and EDST. The prediction accuracy of DST Ensemble keeps increasing as the ensemble size increases, demonstrating the benefits of ensemble. As expected, the predictive accuracy of individual EDST subnetworks decreases as the ensemble size M increases likely due to the reduced training time of each subnetwork. Nevertheless, the ensemble performance of EDST is still quite robust to the ensemble size. Specifically, despite the poor individual performance with M = 7, the ensemble prediction is still higher than a sin-
gle dense model. This behavior confirms our hypothesis mentioned before, i.e., multiple free tickets work better than one high-quality winning ticket or even a dense network.

heading: 5.6 EXPERIMENTS WITH OOD AND ADVERSARIAL ROBUSTNESS
text: We further test the performance of FreeT ickets on out-of-distribution (OOD) detection and adversarial robustness, reporting the results in Appendix K and Appendix L, respectively. Our proposed ensemble methods also bring benefits to OOD performance on CIFAR-10/100 and to adversarial robustness on CIFAR-100.

heading: 6 CONCLUSION AND FUTURE WORKS
text: We introduce FreeT ickets, an efficient way to boost the performance of sparse training over the dense network, even the dense ensemble. FreeT ickets is built as a combination of the diverse subnetworks extracted during dynamic sparse training and achieves a co-win in terms of accuracy, robustness, uncertainty estimation, as well as training/inference efficiency. We demonstrate for the first time that DST methods may not match the performance of dense training when standalone, but can surpass the generalization of dense solutions (including the dense ensemble) as an ensemble, while still being more efficient to train than the single dense network.
The aforementioned compelling efficiency has not been fully explored, due to the limited support of the commonly used hardware. Fortunately, some prior works have successfully demonstrated the promising speedup of sparse networks on real mobile processors (Elsen et al., 2020), GPUs (Gale et al., 2020), and CPUs (Liu et al., 2020a). In future work, we are interested to explore sparse ensembles on hardware platforms for real speedups.

heading: 7 REPRODUCIBILITY
text: Our algorithm is implemented based on the work of sparse momentum (Dettmers & Zettlemoyer, 2019) with the corresponding GitHub repository https://github.com/TimDettmers/ sparse_learning. The training configurations and hyperparameters are shared in Appendix C. The metrics used to enable comparisons among different methods are given in Appendix D.

heading: A DYNAMIC SPARSE TRAINING
text: In this Appendix, we describe the full details of dynamic sparse training. See the survey of Mocanu et al. (2021) and (Hoefler et al., 2021) for more details.
A.1 ALGORITHM
Dynamic Sparse Training (DST) (Mocanu et al., 2018) is a class of methods that enable the end-to-end training of sparse neural networks. DST starts from a sparse network and simultaneously optimizes the sparse connectivity and model weights during training. Without loss of generality, we provide the general pseudocode of DST that can cover most of the existing DST methods in Algorithm 1.
While there is an upsurge in increasingly efficient ways for sparse training (Bellec et al., 2018; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019; Evci et al., 2020a; Liu et al., 2021a; Jayakumar et al., 2020; Liu et al., 2021b; Dietrich et al., 2021; Chen et al., 2021), most of DST methods contain three key components: sparse initialization, model weight optimization, and parameter exploration. We explain them below.
Algorithm 1 Dynamic Sparse Training
Require: Network fΘ, dataset {xi, yi}Ni=1, Layer-wise sparsity ratio: S = {s1, . . . , sL} where {s1, . . . , sL} refer to the sparsity of layer from layer 1 to L, Update Interval ∆T, Exploration Rate p
1: # Sparse Initialization 2: f(x;θs)← f(x;θ;S) 3: for each training step t do 4: # Model Weight Optimization 5: f(x;θs)← SGD(f(x;θs)) 6: if (t mod ∆T) = 0 then 7: # Parameter Exploration 8: Pruning p percentage of parameters using Eq. 1 9: Growing p percentage of parameters using Eq. 2
10: Update exploration rate p 11: end if 12: end for
A.1.1 SPARSE INITIALIZATION
Layer-wise Sparsity Ratio. Layer-wise sparsity ratio plays an important role for sparse training. Mocanu et al. (2018) first introduced Erdős-Rényi (ER) (Erdős & Rényi, 1959) from graph theory to the field of neural networks, achieving better performance than the uniform sparsity ratio. Evci et al. (2020a) further extended ER to CNN and brings significant gains to sparse CNN training with the Erdős-Rényi-Kernel (ERK) ratio. Specifically, the sparsity level of layer l is scaled with 1− n l−1+nl+wl+hl
nl−1×nl×wl×hl , where n l refers to the number of neurons/channels of layer l; wl and hl are the width and the height of layer l. Works with weight redistribution (Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019; Liu et al., 2021a) starts with the uniform layer-wise ratio and dynamically change the layer-wise ratio according to heuristic criteria, ending up with non-uniform layer-wise ratios.
Weight Initialization. Weight initialization also affects the performance of sparse training. Unlike the dense network, sparse networks contain a partial of weights to be zero, breaking the appealing property of dense initialization such as dynamical isometry (Lee et al., 2020), gradient flow (Evci et al., 2020b; Tessera et al., 2021), etc.
A.1.2 MODEL WEIGHT OPTIMIZATION
After initializing the sparse model, we need to optimize the model weights to minimize the loss function. In general, dynamic sparse training is compatible with the most widely used optimizers, e.g., minibatch stochastic gradient descent (SGD) (Robbins & Monro, 1951), minibatch stochastic
gradient descent with momentum (Sutskever et al., 2013; Polyak, 1964), and Adam (Kingma & Ba, 2014).
However, extra attention needs to pay to average-based optimizers, such as Averaged Stochastic Gradient Descent (ASGD), which is widely used in the language modeling tasks with RNNs and LSTMs. Liu et al. (2021a) points out that ASGD seriously destroys the sparse connectivity optimization since the average operation brings the newly-grown weights immediately to zero, leading to an over-pruning situation. Moreover, Tessera et al. (2021) shows that optimizers that use an exponentially weighted moving average (EWMA) are sensitive to high gradient flow.
A.1.3 PARAMETER EXPLORATION
Dynamic sparse training outperforms static sparse training mainly due to the parameter exploration (Liu et al., 2021b). One simple yet effective heuristic used here is prune-and-grow. Removing p percentage of the existing weights and regrowing the same number of new weights allows DST gradually optimizing the sparse connectivity in a heuristic way.
Weight Prune: Although there exists various pruning criteria in pruning literature, the most common way for DST is the simple magnitude pruning. We also tried other criteria for pruning at initialization e.g., SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020). All of them fall short of the accuracy achieved by magnitude pruning.
Weight Grow: The most common ways to grow new weights are random-based growth (proposed in SET (Mocanu et al., 2018)) and gradient-based growth (proposed in RigL (Evci et al., 2020a) and SNFS (Dettmers & Zettlemoyer, 2019)). Random-based growth ensures purely sparse forward pass and backward pass, while it very likely needs more steps to find the important connectivity, especially for the very extreme sparsities. Gradient-based growth detects the weights that reduce the current loss fastest, whereas it involves the dense gradient calculation and likely to cause a collapse of the explored parameter space (Liu et al., 2021a;b; Dietrich et al., 2021).
It is worth noting that the choice of when-to-explore is also of importance for DST. The number of training iterations between two parameter exploration is controlled by update interval ∆T. Since the newly-grown weights are initialized to zero, large ∆T is necessary to guarantee that the new weights receive enough weight updates to survive at the next pruning iteration (Liu et al., 2021b).

heading: B PSEUDOCODE OF DST ENSEMBLE AND EDST ENSEMBLE
text: Algorithm 2 DST Ensemble
Require: network fΘ, dataset {xi, yi}Ni=1, ensemble number M, sparsity distribution: S = {s1, . . . , sL} where {s1, . . . , sL} refer to the sparsity of layer from layer 1 to L, update interval ∆T, DST exploration rate p
1: j = 0 2: repeat 3: # Sparse Initialization 4: f(x;θjs)← f(x;θ
j ;S) 5: for each training step t do 6: # Model Weight Optimization 7: f(x;θjs)← SGD(f(x;θ j s)) 8: if (t mod ∆T) = 0 then 9: # Parameter Exploration
10: Prune p percentage of parameters using Eq. 1 11: Grow p percentage of parameters using Eq. 2 12: Update exploration rate p 13: end if 14: end for 15: save the converged sparse subnetwork f(x;θjs) j = j + 1 16: until j ≥ M
Algorithm 3 EDST Ensemble
Require: network fΘ, dataset {xi, yi}Ni=1, ensemble number M, sparsity distribution: S = {s1, . . . , sL} where {s1, . . . , sL} refer to the sparsity of layer from layer 1 to L, update interval ∆T, DST exploration rate p, large exploration rate q to escape local basin
1: f(x;θs)← sparsely initialize f(x;θ) with S 2: # One Exploration Phase 3: for each exploration phase step t do 4: f(x;θs)← SGD(f(x;θs)) 5: if (t mod ∆T) = 0 then 6: DST parameter exploration using Eq. 1 & 2 with p 7: end if 8: end for 9: # M Sequential Refinement Phases
10: j = 0 11: repeat 12: for each refinement phase step t do 13: f(x;θjs)← SGD(f(x;θ j s)) 14: if (t mod ∆T) = 0 then 15: DST parameter exploration using Eq. 1 & 2 with p 16: end if 17: end for 18: Save the sparse subnetwork f(x;θjs) 19: # Escape the current basin by largely exploring parameters with q 20: f(x;θjs)← f(x;θ j s; q) j = j + 1 21: until j ≥ M
C IMPLEMENTATION AND HYPERPARAMETERS
In this Appendix, we provide the hyperparameters used in Section 4.
We mentioned in the main content of paper that these free tickets (subnetowrks) are non-trivial to obtain due to the three key desiderata. To confirm this, we implement and test two sparsity-based efficient ensemble methods, Static Sparse Ensemble and Lottery Ticket Hypothesis (LTR) Ensemble.
Static Sparse Ensemble Static Sparse Ensemble refers to directly training M sparse subnetworks from scratch and ensemble them for test. Even though it naturally satisfies the low training cost and high diversity, its insufficient accuracy leads to worse performance than the proposed DST/EDST Ensemble.
LTR Ensemble Following Evci et al. (2020b), we use gradually magnitude pruning (GMP) (Zhu & Gupta, 2017; Gale et al., 2019) for LTR, which is a well studied and more efficient pruning method than iterative magnitude pruning (IMP). Concretely, we first use gradually magnitude pruning to prune the dense network to the target sparsity. After that, we retrain and rewind the subnetworks to the 5% epoch for M times. The converged subnetworks are further used for ensemble. Note that even GMP is much more efficient than IMP, the overall training FLOPs required by this method is much higher than directly training a single dense model. We expect that using IMP for Ensemble can be more accurate, but also leads to prohibitive training costs, in contrary to our pursuit of efficient ensemble.
PF Ensemble For the pruning baseline, we choose the strong techniques – global magnitude pruning. We first prune three independently pre-trained dense networks (each is trained for 250 epochs) to the target sparsity, and then fine-tune them for another 250 epochs with different random seeds. We believe this setting can provide enough diversity as all the training procedures are independent, with different random seeds.
Hyperparameters of DST Ensemble For a fair comparison, we follow the training configuration of MIMO. For Wide ResNet28-10 on CIFAR, we train the sparse model for 250 epochs with a learning rate of 0.1 decayed by 10. We use a batch size of 128, weight decay of 5e-4. To achieve a good trade-off between ensemble accuracy and sparsity, we set sparsity as S = 0.8. Regarding the hyperparameters of DST, we choose a large update interval ∆T = 1000 between two sparse connectivity updates and a constant DST exploration rate p = 0.5.
For ResNet-50 on ImageNet, to the best of our knowledge, all the state-of-the-art DST approaches can not match the performance of the dense ResNet-50 on ImageNet with a standard training time. To guarantee sufficient predictive accuracy for the individual ensemble learners, we follow the training setups used in Liu et al. (2021b) and train each sparse ResNet-50 for 200 epochs with a batch size of 64. The learning rate is linearly increased to 0.1 with a warm-up in the first 5 epochs and decreased by a factor of 10 at epochs 60, 120, and 180. Even with a longer training time, it takes much fewer FLOPs to train sparse networks using DST compared to the dense network. We choose ∆T = 4000 and a cosine annealing schedule for the exploration rate with an initial value p = 0.5.
Hyperparameters of EDST Ensemble For Wide ResNet28-10 on CIFAR, we train the 80% sparse model for 450 epochs and 90% sparse model for 850 epochs with tex = 150, tre = 100, so that we can get M = 3 subnetworks with sparsity of 0.8 and M = 7 subnetworks with sparsity of 0.9 for ensemble. The numbers of training FLOPs of this two settings are similar due to the different sparsity levels. The DST exploration rate is chosen as p = 0.5 which achieves the best performance as shown in (Evci et al., 2020a; Liu et al., 2021b). To encourage a large diversity between ensemble learners, we use a large exploration rate q = 0.8 to force the subnetowrk escape the current basin. We have also tested a larger exploration rate, i.e., q = 0.9. The results are similar with q = 0.8.
For ResNet-50 on ImageNet, we set tex = 30 and tre = 70 so that we can obtain M = 2 subnetworks with 170 epochs and M = 4 subnetworks with 310 epochs, respectively. We set p = 0.5 and q = 0.8. The rest of hyperparameters are the same as DST Ensemble. Due to the limited resources, we only test the 80% sparsity for ResNet-50. We believe that the ensemble performance can further improve with lower sparsity.

heading: D EXPERIMENTAL METRICS
text: To measure the predictive accuracy and robustness, we follow the Uncertainty Baseline3 and focus on test accuracy (Acc), negative log-likelihood (NLL), and expected calibration error (ECE) on the i.i.d. test set, corrupted test sets (i.e., CIFAR-10-C and ImageNet-C) (Hendrycks & Dietterich, 2019) which contain 19 natural corruptions such as added blur, compression artifacts, frost effects, etc, as well as on the natural adversarial samples (i.e., ImageNet-A) (Hendrycks et al., 2019). We adopt {cAcc, cNLL, cECE} and {aAcc, aNLL, aECE} to denote the corresponding metrics on corrupted test sets and natural adversarial samples, respectively.
Negative log-likelihood (NLL) is a proper scoring rule and a popular metric for evaluating predictive uncertainty (Quinonero-Candela et al., 2005).
Expected Calibration Error (ECE) (Naeini et al., 2015), is a widely adopted metric to approximate the difference in expectation between confidence and accuracy of machine learning models (i.e., miscalibration). Specifically, it partitions predictions into M equally-spaced bins, and then calculates a weighted average of the accuracy/confidence discrepancy in each of these bins. Larger ECE values represent a worse match between confidence and accuracy.
To compare the computational efficiency, we report the training FLOPs required to obtain the targeted number of subnetworks for all methods and normalize them with the FLOPs required by a dense model. Following RigL4 (Evci et al., 2020a), the FLOPs are calculated with the total number of multiplications and additions layer by layer. Briefly speaking, with ERK distribution, the training FLOPs of a sparse Wide ResNet28-10 at sparsity S = 0.8 and S = 0.9 are 33.7% and 16.7% of the dense model, respectively. For sparse ResNet-50 at S = 0.8, the required training FLOPs is 42% of the dense model. We omit the additional computational cost for the extra inputs and the extra outputs of MIMO, as it is negligible compared with the whole training FLOPs. Moreover, we suppose the hardware can fully utilize large batch size so that BatchEnsemble incurs almost no additional computational overhead and memory cost.

heading: E COMPARISON WITH SNAPSHOT AND FGE
text: In this Appendix, we compare our methods with the ensemble methods that use cyclical learning rate schedules to discover diverse dense networks, i.e., Snapshot (Huang et al., 2017) and FGE (Garipov et al., 2018). For a relatively fair comparison, we compare our methods with their ‘1B’ versions (Garipov et al., 2018) which have similar training FLOPs with our methods. However, their test FLOPs would be much larger as their ensemble members are dense networks. As shown in Table 8, while achieving relatively similar performance, Snapshot and FGE require significantly more test FLOPs than the DST-based ensemble methods.
google-research/rigl/tree/master/rigl/imagenet_resnet/colabs

heading: F PREDICTION DISAGREEMENT ON CIFAR-10/100
text: We compare the prediction disagreement among the naive dense ensemble, DST Ensemble with sparsity S = 0.8, EDST Ensemble with sparsity S = 0.8 and S = 0.9. The comparison results on CIFAR-10 and CIFAR-100 are shown in Figure 9 and Figure 10, respectively. We find that the subnetworks discovered by DST Ensemble are even more diverse than the ones discovered by the traditional dense ensemble, confirming our hypothesis that dynamic sparsity can provide extra diversity in addition to random initializations. While the average functional diversity of the subnetworks discovered by EDST Ensemble is a bit lower compared with DST Ensemble, the diversity is still notable. Moreover, as expected, at a higher sparsity S = 0.9, the prediction diversity of EDST Ensemble slowly rises as the ensemble size M increases, and ends up with a similar diversity as the Dense Ensemble.

heading: G DISAGREEMENT BREAKDOWN BETWEEN VARIOUS SPARSE NETWORK
text: ENSEMBLES ON CIFAR-100
In this Appendix, we try to provide detailed illustration about various classes that cause the disagreement between various sparse network ensembles. We choose the first 10 classes in CIFAR-100 and measure the disagreement between the subnetworks learned by the same method on these classes respectively. The results are shared in Table 4. The overall breakdown of disagreement is very similar to the diversity measurement results, i.e., DST Ensemble > Static Ensemble > EDST Ensemble > LTR Ensemble. The prediction on classes like Fish, Baby, and Bear have higher diversity than Apple and Beetle.

heading: H CORRELATION MEASUREMENT BETWEEN THE ACCURACY OF SUBNETWORKS AND THEIR ENSEMBLE ACCURACY
text: In the main paper, we hypothesize that high expressibility is a crucial desideratum for FreeT ickets to guarantee superior ensemble performance. Here, we apply two widely used correlation measurements, Pearson correlation (Pearson, 1895) and Spearman correlation (Powers & Xie, 2008), to support our hypothesis. We use these two methods to measure the correlation between the mean accuracy of individual networks and the accuracy of their ensemble (higher values refer to higher correlation). The results are shared in Table 5. We can find that the individual accuracy and the ensemble accuracy are very highly correlated on CIFAR-100. Their correlation is relatively low on CIFAR-10 likely due to the almost saturated accuracy of the individual subnetwork (96%). Given such high accuracy, even the usually clear and strong correlation between the sparsity and test accuracy becomes relatively vague on CIFAR-10 in Figure 6-a and Figure 6-b.
I EFFECT OF EXPLORATION RATE q ON EDST ENSEMBLE
Intuitively, a larger exploration rate q leads to larger sparse connectivity diversity whereas an overlarge q would prune the important connections, hurting the ensemble accuracy. We perform a sensitivity study with EDST Ensemble in Table 7, whose results are in line with our conjecture. A large exploration rate q = 0.8 promote higher diversity but degrades the ensemble accuracy a bit compared with q = 0.5. q = 0.5 seems to achieve the best ensemble performance.

heading: J EFFECT OF REGROWTH METHODS ON EDST ENSEMBLE
text: Random regrowth naturally considers a large range of parameters to explore compared with the gradient regrowth as used in the main paper. We added below a small set of experiments to study different growth criteria (random vs gradients) on the EDST Ensemble. As shown in Table 7, it seems that random growth may provide more diversity and better performance for more complex datasets (i.e., CIFAR-100), but more studies are necessary to understand better the phenomenon.

heading: K EXPERIMENTS WITH OOD RESULTS
text: In this section, we report the performance of out-of-distribution (OOD) detection for single sparse networks and ensemble models. Specifically, we use datasets that are not seen during training time for OOD evaluation. Following the classic routines (Augustin et al., 2020; Meinke & Hein, 2019), SVHN (Netzer et al., 2011), CIFAR-100 (Krizhevsky & Hinton, 2009), and CIFAR-10 with random Gaussian noise (Hein et al., 2019) are adopted for models trained on CIFAR-10 (Krizhevsky & Hinton, 2009); SVHN, CIFAR-10, and CIFAR-100 with random Gaussian noise are used for models trained on CIFAR-100. The ROC-AUC performance (Augustin et al., 2020; Meinke & Hein, 2019) are reported over the respective out of distribution datasets, as shown in Table 8 and Table 9. As we expected, our proposed ensemble methods provide gains also to the OOD performance. DST Ensemble and EDST Ensemble can even surpasses the dense models by a clear margin in most cases.

heading: L EXPERIMENTS WITH ADVERSARIAL ROBUSTNESS
text: Table 10 presents the adversarial robustness performance of Wide ResNet-28-10 trained on CIFAR010 and CIFAR-100, where the Fast Gradient Sign Method (FGSM) (Szegedy et al., 2013) with step size 8255 is adopted. Following a similar routine as in Strauss et al. (2017), we craft adversarial examples for each single model and report the {min, mean, max} robust accuracy over generated attacks from different models. The results demonstrate that our proposed ensemble methods improve robustness accuracy as well, especially in terms of average robust accuracy. More surprisingly, the ensemble of static sparse models can even outperform DST-based Ensembles in this setting. Further analysis on this task serves as an interesting future work.

Review summaries:
This work focuses on generating ensembles of sparse networks, called FreeTickets, that requires significantly lesser training FLOPs and parameters compared to single or ensemble dense networks while maintaining high prediction accuracy.  FreeTickets are expected to be diverse, highly expressive and easy to obtain. Apart from the methodology to generate these FreeTickets and ensure they match the desired properties, a key claim in this work is that FreeTickets demonstrate improved prediction accuracy, robustness and uncertainty estimation while being much more efficient, in terms of the computational resources required for training. 

To my knowledge this paper provides the first extensive study on the ensemble of sparsely connected neural networks. Ensembles often improve performance, however they are costly. Sparse neural networks, on the other hand, can reduce the cost of each weak learner significantly. In addition to reducing cost at inference, authors also focus on reducing the cost of training, therefore they use end-to-end sparse training methods (dynamic sparse training). Results show that sparse neural networks can provide comparable (or better) ensemble performance while reducing the training and inference cost significantly. This work also provides some interesting insights into the ensemble of sparse neural networks.

The authors propose to improve the generalization of dynamic sparse neural network training methods, such as RiGL, by learning an ensemble of these models. The authors propose two methods of doing so, the Dynamic Sparse Training (DST) Ensemble, and the Efficient Dynamic Sparse Training (EDST) Ensemble. A DST ensemble is a straightforward ensemble of M DST trained models, which like all ensembles incurs high overhead in training and inference cost (training/using M models). An EDST ensemble on the other hand uses approximately the same training compute as a single model. It does so by splitting the training into two phases (of the total amount of steps as a single standard DST training run). The first phase is standard DST training (called the Exploration Phase), where training uses a large learning rate. The second phase (called the Refinement Phase) trains M models initialized from the weights of the model in the first phase, but with much smaller learning rates decayed over training. Hyperparameters determine the amount of exploration v.s. refinement training steps.  The authors evaluate the trained ensembles for Image Classification with Wide ResNet28-10 on CIFAR-10/100 and ResNet-50 on ImageNet,  and compare these to baselines including: an ensemble of sparse trained models (with a static mask), an ensemble of Lottery Ticket Rewound models, an alternative method with similar motivation (MIMO), a single dense model, BatchEnsemble and TreeNet. The DST and EDST Ensembles show similar or marginally higher generalization than baselines, but specifically in the EDST case, with much fewer (theoretical) training FLOPS. The authors also compare the diversity of the ensemble members in a few of these baselines, showing that DST has much higher diversity than EDST (as might be expected), but also interestingly that DST appears to have higher diversity than a dense ensemble. The authors perform an ablation study on their results, attempting to discern the affects of parameter exploration (i.e. dynamic mask updates in DST methods), affect of sparsity and affect of ensemble size on the results. The authors also claim to show better out-of-distribution performance and robustness, although these results are in the appendix and only referred to in the main text.