More or Less: When and How to Build Convolutional Neural Network Ensembles

1 INTRODUCTION . Scaling capacity of deep learning models . Convolutional neural network models are becoming as accurate as humans on perceptual tasks . They are now used in numerous and diverse applications such as drug discovery , data compression , and automating gameplay . These models increasingly grow in size with more parameters and layers , driven by two major trends . First , there is a continuous rise in data complexity and sizes in many applications ( Shazeer et al. , 2017 ) . Second , there is an increasing need for higher accuracy as models are utilized in more critical applications – such as self-driving cars and medical diagnosis ( Grzywaczewski , 2017 ) . This effect is especially pronounced in computer vision and natural language processing : Model sizes are three orders of magnitude larger than they were just three years ago ( Sanh et al. , 2019 ) . With bigger model sizes , the time , computation , and memory needed to train and deploy such models also increase . Thus , it is a consistent challenge to design models that maximize accuracy while remaining practical with respect to the resources they need ( Lee et al. , 2015 ; Huang et al. , 2017b ) . In this paper , we study the following question : Given a number of parameters ( neurons ) , how to design a convolutional neural network to optimize holistically for accuracy , training cost , and inference cost ? The holistic design space is very complex . Designers of convolutional neural network models navigate a complex design landscape to address this question : First , they need to decide on network architecture . Then , they have to consider whether to use a single network or build an ensemble model with multiple networks . Additionally , they have to decide how many neural networks to use and their individual designs , i.e. , the depth , width , and number of networks in their model . Modern applications with diverse requirements further complicate these decisions as what is desirable varies . Facebook , for instance , requires convolutional neural network models that strike specific tradeoffs between accuracy and inference time across 250 different types of smartphones ( Wu et al. , 2019 ) . As a result , not just accuracy but a diversity of metrics – such as inference time and memory usage – inform whether a model gets used ( Sze et al. , 2017b ) . Scattered conventional wisdom . There exist bits and pieces of scattered conventional wisdom to guide a neural network designer . These take the form of various empirical studies that demonstrate how depth and width in a single neural network model relate to certain metrics such as accuracy . First , it is generally known that deeper and wider networks can improve accuracy . In fact , recent convolutional architectures – such as ResNets and DenseNets – are designed precisely to enable this outcome ( He et al. , 2016 ; Huang et al. , 2017b ; a ) . The caveat with beefing up a neural network is that accuracy runs into diminishing returns as we continue to add more layers or widen existing ones ( Coates et al. , 2011 ; Dauphin and Bengio , 2013 ) . On the other hand , increasing the number of networks in the model , i.e. , building ensembles , is considered a relatively robust but expensive approach to improve accuracy as ensemble models train and deploy k networks instead of one ( Russakovsky et al. , 2015 ; Wasay et al. , 2020 ) . The consensus is to use ensembles when the goal is to achieve high accuracy without much regard to training cost , inference time , and memory usage , e.g. , competitions such as COCO and ImageNet ( Lee et al. , 2015 ; Russakovsky et al. , 2015 ; Huang et al. , 2017a ; Ju et al. , 2017 ) . All these studies , however , exist in silos . Any form of cross-comparison is impossible as they use different data sets , network architectures , and hardware . Lack of a robust and holistic assessment . Most past studies operate within the confines of a single convolutional network and do not consider the dimension of ensemble models . Those that compare with ensembles mostly do so unfairly comparing ensembles with k networks against a model that contains only one such network ( Lee et al. , 2015 ; Russakovsky et al. , 2015 ; Huang et al. , 2017a ; Ju et al. , 2017 ) . There are recent studies that make this comparison under a fixed parameter budget ( Chirkova et al. , 2020 ; Kondratyuk et al. , 2020 ) . However , these studies consider only the metric of generalization accuracy and explore a very small part of the design space – two different classes of convolutional architectures with a single depth . A holistic analysis needs to include resource-related metrics such as training time , inference cost , and memory usage . All these metrics are critical for practical applications ( Sze et al. , 2017a ; Wu et al. , 2019 ) . Furthermore , to provide reliable guidance to a model designer , a robust comparison needs to consider a range of architectures and model sizes with various depth and width configurations . This is critical , especially because varying just the width of convolutional networks in isolation , as done by recent studies ( Chirkova et al. , 2020 ; Kondratyuk et al. , 2020 ) , is known to be far less effective to improve accuracy ( Eigen et al. , 2013 ; Ba and Caruana , 2014 ) . Single networks vs. ensembles . In this paper , we bridge the gap in the understanding of the design space by providing answers to the following questions . Given specific requirements in terms of accuracy , training time , and inference time , should we train and deploy a convolutional model with a single network or one that contains an ensemble of networks ? How should we design the networks within an ensemble ? As these constraints and requirements evolve , should we switch between these alternatives , why , and when ? Method . We introduce the following methodology to map the design space accurately . Since there is no robust theoretical framework to consistently analyze the design space and the complex interactions among its many parameters and metrics , we develop a detailed and extensive experimental framework to isolate the impact of the critical design knobs : ( i ) depth , ( ii ) width , and ( iii ) number of networks , on all relevant metrics : ( i ) accuracy , ( ii ) training time , ( iii ) inference time , and ( iv ) memory usage . Crucially the number of parameters is a control knob in our framework , and we only compare alternatives under the same parameter budget . To establish the robustness of our findings , we experiment across various architectures , data complexities , and classification tasks . We present and analyze data amounting to over one year of GPU run time . We also explain trends breaking down metrics into their constituents when necessary . Results : The Ensemble Switchover Threshold ( EST ) . ( i ) Contrary to conventional wisdom , we show that when we make a holistic and robust comparison between single convolutional networks and ensembles of networks , we discover a vast design space where ensembles provide not just better overall accuracy but also train faster compared to a single network . ( ii ) Specifically , we uncover the Ensemble Switchover Threshold ( EST ) . This is the amount of resources ( measured in terms of the number of parameters and training epochs ) beyond which ensembles provide superior generalization accuracy to a single model . ( iii ) We show that EST occurs consistently across numerous data sets and architectures . ( iv ) We demonstrate that the number of networks in an ensemble and their individual designs determine the EST . ( v ) Ensembles can also provide comparable inference times for a considerable part of the design space . ( vi ) We also show that ensembles require significantly less memory to train for the same number of parameters . ( vii ) Finally , we make available a superset of our results for visual exploration and help with model design at : daslab.seas.harvard.edu/more-or-less . , 2 FRAMEWORK : DESIGN SPACE . The design space we explore consists of single convolutional neural network models and two classes of architecturally-homogenous ensembles . These ensemble classes help isolate the effect of the two design knobs – depth and width – on the quality and cost of an ensemble design . We first describe how we ensure a robust comparison of alternative model designs and then explain the degrees of freedom we explore . Establishing grounds for fair ranking . A key element of our framework is that the possible model designs are compared to each other only under equivalent resources . We ensure this by only comparing designs that have the same number of parameters . This comparison allows us to separate the quality of a design from the amount of resources given to it . Another way to think about this is that given a parameter budget , we can investigate how the three design classes rank for all relevant metrics ( training and inference time , accuracy , and memory usage ) . We fix the number of parameters because of its two distinctive properties over other metrics ( that we could have fixed ) , such as training time , inference time , or accuracy : First , the number of parameters of a network is directly proportional to all other resource-related metrics ( Jain et al. , 2020 ; Wasay et al. , 2020 ) . Second , the number of parameters is independent of the hardware or the software platform used and can be computed exactly from a network specification . The single network versus ensemble design space . Our design space considers a convolutional neural network architecture S ( w , d ) from a class of neural network architectures C. S ( w , d ) has width factor w , depth d , and number of parameters |S| . Similarly an ensemble is described as E = { E1 . . . Ek } . Ensembles are architecturally-homogenous i.e. , all ensemble networks E1 . . . Ek have the same architecture and each network has |E|/k parameters . When we compare a single network S ( w , d ) from C with an ensemble E we ensure that E1 . . . Ek ∈ C and |E1|+. . .+|Ek| = |S| . The reason why we restrict the design space to homogenous ensembles is to reduce the otherwise intractably large space1 of all possible ensembles given a single network to a size that we can 1Given a single network with |S| number of parameters , there are { |S| k } ( Stirling number of the second kind ) as many ways of forming ensembles of size k. This number grows at a similar rate to exponential polynomials , ( Boyadzhiev , 2009 ) e.g. , { 100 4 } ≈ 1059. feasibly and thoroughly experiment with and reason about . Furthermore , many neural network ensembles introduced in research and used in practice are similarly homogenous , for instance , SnapShot Ensembles and Fast Geometric Ensembles ( Huang et al. , 2017a ; Garipov et al. , 2018 ) . Additionally , our method provides a deterministic procedure of going between single network models and ensembles given a certain amount of parameters . Major sources of diversity in neural network ensembles are random weight initialization and stochastic training , both of which we incorporate in our framework . Depth-equivalent and width-equivalent ensembles . Convolutional neural network architectures are determined by two design knobs – the depth and the width factor . Corresponding to these two design knobs , we create two classes of ensembles : depth-equivalent ensembles and width-equivalent ensembles . These are depicted in Figure 1 : In depth-equivalent ensembles , the depth of the individual ensemble networks is the same as S ( i.e. , d ) , and the width factor is set to the highest possible value ( i.e. , w′ ) without exceeding the parameter budget of |S| . In width-equivalent ensembles , on the other hand , the width factor is conserved across all ensemble networks ( i.e. , w ) , and the depth is modulated to the highest possible value ( i.e. , d′ ) without exceeding |S| : w′ : k · |E ( w ′ , d ) i | ≤ |S ( w , d ) | ≤ k · |E ( w ′+1 , d ) i | d ′ : k · |E ( w , d ′ ) i | ≤ |S ( w , d ) | ≤ k · |E ( w , d ′+1 ) i | The above definition follows that neural networks in depth-equivalent ensembles have higher depth than those in width-equivalent ensembles . Width-equivalent ensembles contain wider neural networks than their depth-equivalent counterparts . In this way , we isolate and study the effect of depth and width on ensemble accuracy and resource requirement . Overall , our design space spans three classes of convolutional neural network designs : ( i ) single network models , ( ii ) width-equivalent ensembles , and ( iii ) depth-equivalent ensembles . Every class contains several model designs instantiated by the four-tuple { w , d , |S| , C } . We next describe how we designed an exhaustive experimental framework to cover various configurations of these four-tuples .

Summary: This paper establish a robust and holistic framework to compare scaling up an ensemble with scaling up a single networks, where test accuracy, number of paramaters, inference time, memory consumption and training time to converge are considered.  To reduce the intractably large design space of scaling up an ensemble, the author mainly investigate two types of ensembles: depth-equivalent and width-equivalent ensembles.  Through extensive experiments on SVHN, CIFAR-10, CIFAR-100, and Tiny ImageNet with VGGNets, ResNets, DenseNets and WideResNets, the authors discovered an surprising and consistently emerging phenomenon named The Ensemble Switchover Threshold: When the amount of resources (measured by number of parameters, training cost) is beyond this threshold, ensembles methods provide better performance and computation trade-off.