Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity

The success of deep ensembles on improving predictive performance , uncertainty , and out-of-distribution robustness has been extensively demonstrated in the machine learning literature . Albeit the promising results , naively training multiple deep neural networks and combining their predictions at test lead to prohibitive computational costs and memory requirements . Recently proposed efficient ensemble approaches reach the performance of the traditional deep ensembles with significantly lower costs . However , the training resources required by these approaches are still at least the same as training a single dense model . In this work , we draw a unique connection between sparse neural network training and deep ensembles , yielding a novel efficient ensemble learning framework called FreeT ickets . Instead of training multiple dense networks and averaging them , we directly train sparse subnetworks from scratch and extract diverse yet accurate subnetworks during this efficient , sparse-to-sparse training . Our framework , FreeT ickets , is defined as the ensemble of these relatively cheap sparse subnetworks . Despite being an ensemble method , FreeT ickets has even fewer parameters and training FLOPs compared to a single dense model . This seemingly counter-intuitive outcome is due to the ultra training efficiency of dynamic sparse training . FreeT ickets improves over the dense baseline in the following criteria : prediction accuracy , uncertainty estimation , out-of-distribution ( OoD ) robustness , and training/inference efficiency . Impressively , FreeT ickets outperforms the naive deep ensemble with ResNet50 on ImageNet using around only 1/5 training FLOPs required by the latter . 1 INTRODUCTION . Ensembles ( Hansen & Salamon , 1990 ; Levin et al. , 1990 ) of neural networks have received large success in terms of the predictive accuracy ( Perrone & Cooper , 1992 ; Breiman , 1996 ; Dietterich , 2000 ; Xie et al. , 2013 ) , uncertainty estimation ( Fort et al. , 2019 ; Lakshminarayanan et al. , 2017 ; Wen et al. , 2020 ; Havasi et al. , 2021 ) , and out-of-distribution robustness ( Ovadia et al. , 2019a ; Gustafsson et al. , 2020 ) . Given the fact that there are a wide variety of local minima solutions located in the high dimensional optimization problem of deep neural networks and various randomness ( e.g. , random initialization , random mini-batch shuffling ) that occurs during training , neural networks trained with different random seeds would converge to different low-loss basins with similar error rates ( Fort et al. , 2019 ; Ge et al. , 2015 ; Kawaguchi , 2016 ; Wen et al. , 2019 ; 2020 ) . Deep ensembles achieve large performance boost by simply averaging or voting on the predictions of these networks . Despite the promising performance improvement , the traditional deep ensemble naively trains multiple deep neural networks independently and ensemble them , whose training and inference cost increases linearly with the number of ensemble members . Recent works on efficient ensembles , e.g. , BatchEnsemble ( Wen et al. , 2020 ) , hyper batch ensembles ( Wenzel et al. , 2020 ) , and MIMO ( Havasi et al. , 2021 ) , enable to reach the performance of dense ensemble with only negligible extra computation compared with a single dense model . However , the training resources required by these approaches are still at least the same as training a single dense model . Since the size of advanced deep neural networks is inevitably exploding ( Touvron et al. , 2020 ; Dosovitskiy et al. , 2021 ; Brown et al. , 2020 ; Touvron et al. , 2021 ) , the associated enormous training costs are potentially beyond the reach of most researchers and startups , leading to financial and environmental concerns ( Garcı́a-Martı́n et al. , 2019 ; Schwartz et al. , 2019 ; Strubell et al. , 2019 ) . On the other hand , to accelerate training , researchers explored the possibility of directly training sparse neural networks from scratch ( sparse training ) ( Mocanu et al. , 2016 ; Liu et al. , 2020a ; Evci et al. , 2019 ) , while trying to maintain comparable performance . However , training a sparse network from scratch typically results in worse performance than their dense counterparts ( Kalchbrenner et al. , 2018 ; Evci et al. , 2019 ) . The current only way for matching performance while keeping the network sparse throughout training is Dynamic Sparse Training ( DST ) ( Mocanu et al. , 2018 ; Bellec et al. , 2018 ; Evci et al. , 2020a ; Liu et al. , 2021b ) . DST seeks to train sparse neural networks from scratch with a strict training budget . It starts from a randomly-initialized sparse network and optimizes the model weights together with the sparse connectivity during training . The current only way for DST to match the performance of its dense counterpart on the popular benchmark , e.g. , ResNet-50 on ImageNet , is to extend the training time ( Evci et al. , 2020a ) , which might require thousands of training epochs for extremely sparse models ( Liu et al. , 2021b ) . In this paper , we attempt to address the above-mentioned two challenges jointly by drawing a unique connection between sparse training and deep ensembles . Specifically , we ask the following question : Instead of allocating all resources to find a strong winning ticket , can we find many weak tickets with very low costs ( free tickets ) , where the combination of these free tickets can significantly outperform the single dense network , even the dense ensemble ? Note that it is not trivial to obtain free tickets . To guarantee superior ensemble performance , three key desiderata the free tickets are expected to satisfy ( 1 ) high diversity : according to the ensemble theory ( LeCun et al. , 2015 ; Hansen & Salamon , 1990 ; Ovadia et al. , 2019b ) , higher diversity among ensemble members leads to higher predictive performance ; ( 2 ) high accessibility : free tickets should be cheap to obtain so that the overall training cost does not compromise too much ; and ( 3 ) high expressibility : the performance of each free ticket should be comparable with the dense model . Leveraging the insight from Liu et al . ( 2020b ) that a full network contains a plenitude of performative subnetworks that are very different in the topological space , we introduce the concept ofFreeT ickets , an efficient ensemble framework that utilizes sparse training techniques to generate subnetworks for the ensemble . Furthermore , we instantiate FreeT ickets by proposing two efficient ensemble methods – Dynamic Sparse Training Ensemble ( DST Ensemble ) and Efficient Dynamic Sparse Training Ensemble ( EDST Ensemble ) . Both methods yield diverse subnetworks that consummately satisfy the above-mentioned criteria . We summary our contributions below : • Our first method , DST Ensemble , independently trains multiple subnetworks from scratch with dynamic sparsity . By averaging the predictions of these subnetworks , DST Ensemble improves the predictive accuracy , OoD robustness , uncertainty estimation , and efficiency over the traditional dense ensemble . • Our second , light-weight method ( EDST Ensemble ) yields many free tickets in one single run , which is more efficient to train and inference than a single dense model , while approaching the performance of the traditional dense ensemble . • We analyze the diversity of the individual subnetworks generated by our methods and confirm the effectiveness of our methods on inducing model diversity . • Our results suggest that besides the training/inference efficiency , sparse neural networks enjoy also other favorable properties which are absent in dense networks ( robustness , out-ofdistribution generalization , etc ) , opening the path for new research directions . 2 RELATED WORKS . Efficient Ensembles . One major drawback of ensembles is the expensive computational and memory costs for both training and testing . To address this problem , various approaches have been proposed . TreeNet ( Lee et al. , 2015 ) shares weights in earlier layers and splits the following model into several branches , improving accuracy over the dense ensemble . Monte Carlo Dropout ( Gal & Ghahramani , 2016 ) can be used to approximate model uncertainty in deep learning without sacrificing either computational complexity or test accuracy . BatchEnsemble ( Wen et al. , 2020 ) proposed to improve parameter efficiency by decomposing the ensemble members into the product of a shared matrix and a rank-one matrix personalized for each member . MIMO ( Havasi et al. , 2021 ) uses a multi-input multi-output configuration to concurrently discover subnetworks that co-habit the dense network without explicit separation . Snapshot ( Huang et al. , 2017 ) and FGE ( Garipov et al. , 2018 ) discover diverse models by using cyclical learning rate schedules . Furlanello et al . ( 2018 ) applied knowledge distillation ( Hinton et al. , 2015 ) to train several generations of dense students . The ensemble of the dense students outperforms the teacher model significantly . Other related works include but are not limited to hyper-batch ensembles ( Wenzel et al. , 2020 ) and Late Phase ( Oswald et al. , 2021 ) . However , the training resources required by these methods are still at least the same as training a single dense model . In contrast to the existing efficient ensemble methods , our methods ( EDST Ensemble ) can match the performance of naive ensemble with only a fraction of the resources required by training a single dense network . Dynamic Sparse Training . Dynamic Sparse Training ( DST ) is a class of methods that enables training sparse neural networks from scratch by optimizing the sparse connectivity together with the weight values during training . DST stems from Sparse Evolutionary Training ( SET ) ( Mocanu et al. , 2018 ) , a sparse training algorithm that outperforms training a static sparse model from scratch ( Mocanu et al. , 2016 ; Evci et al. , 2019 ) . Weight reallocation was further proposed in Mostafa & Wang ( 2019 ) ; Dettmers & Zettlemoyer ( 2019 ) ; Liu et al . ( 2021a ) to reallocate new weights across layers for better layer-wise sparsity . Further , Dettmers & Zettlemoyer ( 2019 ) ; Evci et al . ( 2020a ) leverage the gradient information in the backward pass to guide the optimization of sparse connectivity and demonstrate substantial performance improvement . Some recent works ( Jayakumar et al. , 2020 ; Raihan & Aamodt , 2020 ; Liu et al. , 2021b ) demonstrate that a large range of exploration in the parameter space is important for dynamic sparse training . Price & Tanner ( 2021 ) improved the performance of DST by adding additional non-trainable parameters . DST has also demonstrated its strength in feature detection ( Atashgahi et al. , 2020 ) , lifelong learning ( Mallya & Lazebnik , 2018 ; Sokar et al. , 2021 ) , federated learning ( Zhu & Jin , 2019 ) , and adversarial training ( Özdenizci & Legenstein , 2021 ) . Sparse MoE ( Shazeer et al. , 2017 ; Fedus et al. , 2021 ) sparsely activates one of the few expert networks to increase the model capacity – but with a constant computational cost . It usually requires specialized modules , such as gating and selector networks to perform the sparsification . 3 FREETICKETS . 3.1 PRELIMINARIES . Dynamic Sparse Training . Let ’ s consider an i.i.d . classification setting with data { ( xi , yi ) } Ni=1 , where x usually denotes input samples and y is the corresponding labels . For a network f ( x ; θ ) parameterized by θl ∈ R in each layer of l ∈ { 1 , . . . , L } , we train f ( x ; θ ) to solve the following optimization problem : θ̂ = arg minθ ∑N i=1 L ( f ( xi ; θ ) , yi ) . Dynamic sparse training ( DST ) starts with a randomly-initialized sparse neural network f ( x ; θs ) , parameterized by a fraction of parameters θs . The sparsity level of the sparse model is pre-defined as S = 1− ‖θs‖0‖θ‖0 , where ‖ · ‖0 is the ` 0-norm . The goal of DST is to yield a sparse network with the target sparsity S after training , while maintaining the overall computational and memory overheads close to training a static sparse model ( fixed sparse connectivity ) . During training , DST continuously minimizes the loss ∑N i=1 L ( f ( xi ; θs ) , yi ) , while periodically exploring the parameter space for better sparse connectivity with non-differentiable heuristics . A simple yet effective exploration heuristics is prune-and-grow , that is , pruning p percentage of the unimportant weights from θs , followed by regrowing the same number of new weights . By repeating this prune-and-grow cycle , DST continuously searches for better sparse connectivities while stick to a fixed parameter budget . See Appendix A for a general pseudocode and full details of DST , where a brief literature review of DST is provided . FreeTickets Ensemble . We propose the concept of FreeT ickets here . FreeT ickets refers to efficient ensemble methods that utilize DST to generate subnetworks for ensemble . A free ticket is a converged subnetwork with its specific sparse connectivity . These free tickets { θ1s , θ2s , . . . θMs } , observed and collected either within one training run ( EDST Ensemble ) or multiple training runs ( DST Ensemble ) , are further used to construct the FreeT ickets Ensemble . Assuming that the output probability of the kth neuron of the jth free ticket is given by p ( ajk ) . Then the output of the corresponding kth neuron in the ensemble can been obtained by taking the average across all the M subnetworks , i.e. , 1M ∑M j=1 p ( a j k ) . Compared with the existing efficient ensemble techniques ( Huang et al. , 2017 ; Wen et al. , 2020 ) , FreeT ickets induces diversity from the perspective that there exists many performative subnetworks with very different sparse connectivities within the full network ( Liu et al. , 2020b ) . The efficiency of FreeT ickets comes from the fact that each subnetwork is sparse from the beginning , so that the memory and floating-point operations ( FLOPs ) required by FreeT ickets can be even fewer than training a single dense network . To instantiate the concept of FreeT ickets , we introduce two DST-based ensemble methods , DST Ensemble and EDST Ensemble , as described below .

Summary: The authors propose to improve the generalization of dynamic sparse neural network training methods, such as RiGL, by learning an ensemble of these models. The authors propose two methods of doing so, the Dynamic Sparse Training (DST) Ensemble, and the Efficient Dynamic Sparse Training (EDST) Ensemble. A DST ensemble is a straightforward ensemble of M DST trained models, which like all ensembles incurs high overhead in training and inference cost (training/using M models). An EDST ensemble on the other hand uses approximately the same training compute as a single model. It does so by splitting the training into two phases (of the total amount of steps as a single standard DST training run). The first phase is standard DST training (called the Exploration Phase), where training uses a large learning rate. The second phase (called the Refinement Phase) trains M models initialized from the weights of the model in the first phase, but with much smaller learning rates decayed over training. Hyperparameters determine the amount of exploration v.s. refinement training steps.  The authors evaluate the trained ensembles for Image Classification with Wide ResNet28-10 on CIFAR-10/100 and ResNet-50 on ImageNet,  and compare these to baselines including: an ensemble of sparse trained models (with a static mask), an ensemble of Lottery Ticket Rewound models, an alternative method with similar motivation (MIMO), a single dense model, BatchEnsemble and TreeNet. The DST and EDST Ensembles show similar or marginally higher generalization than baselines, but specifically in the EDST case, with much fewer (theoretical) training FLOPS. The authors also compare the diversity of the ensemble members in a few of these baselines, showing that DST has much higher diversity than EDST (as might be expected), but also interestingly that DST appears to have higher diversity than a dense ensemble. The authors perform an ablation study on their results, attempting to discern the affects of parameter exploration (i.e. dynamic mask updates in DST methods), affect of sparsity and affect of ensemble size on the results. The authors also claim to show better out-of-distribution performance and robustness, although these results are in the appendix and only referred to in the main text.