More or Less: When and How to Build Convolutional Neural Network Ensembles
Authors: Abdul Wasay
Stratos Idreos
Venue: ICLR 2021 Poster (2021)
Keywords: ensemble learning
empirical study
machine learning systems
computer vision

Abstract: Convolutional neural networks are utilized to solve increasingly more complex problems and with more data. As a  result, researchers and practitioners seek to scale the representational power of such models by adding more parameters. However, increasing parameters requires additional critical resources in terms of memory and compute,  leading to increased training and inference cost. Thus a consistent challenge is to obtain as high as possible accuracy within a parameter budget. As neural network designers navigate this complex landscape, they are guided by conventional wisdom that is informed from past empirical studies. We identify a critical part of this design space that is not well-understood: How to decide between the alternatives of expanding a single convolutional network model or increasing the number of networks in the form of an ensemble. We study this question in detail across various network architectures and data sets. We build an extensive experimental framework that captures numerous angles of the possible design space in terms of how a new set of parameters can be used in a model.  We consider a holistic set of metrics such as training time, inference time, and memory usage. The framework provides a robust assessment by making sure it controls for the number of parameters. Contrary to conventional wisdom, we show that when we perform a holistic and robust assessment, we uncover a wide design space, where ensembles provide better accuracy, train faster, and deploy at speed comparable to single convolutional networks with the same total number of parameters. 

Published as a conference paper at ICLR 2021
MORE OR LESS: WHEN AND How TO BUILD CONVOLUTIONAL NEURAL NETWORK ENSEMBLES
Abdul Wasay Harvard University awasay seas. heeeadaavvrddedu
Stratos Idreos
Harvard University
stratos @ seas. haraar
ABSTRACT
Convolutional neural networks are utilized to solve increasingly more complex problems and with more data. As result, researchers and practitioners seek to scale the representational power such models by adding more parameters. However, increasing parameters requires additional critical resources in terms of memory and compute, leading to increased training and inference cost. Thus a consistent challenge is obtain as high as possible accuracy within parameter budget. As neural network designers navigate this complex landscape, they are guided by conventional wisdom that is informed from past empirical studies. We identify a critical part of this design space that is not well-understood: How to decide between the alternatives expanding single convolutional network model or increasing the number of networks in the form of an ensemble. We study this question in detail across various network architectures and data sets. We build an extensive experimental framework that captures numerous angles of the possible design space in terms how a new set parameters can used We consider a holistic set of metries such as training time, inference time, and memory usage. The framework provides a robust assessment by making sure it controls for the number of parameters. Contrary to conventional wisdom, we show that when we perform a holistic and robust assessment, we uncover a wide design space, where ensembles provide better accuracy, train faster, and deploy speed comparable to single convolutional networks with the same total number parameters.
model.
INTRODUCTION
Scaling capacity of deep learning models. Convolutional neural network models are becoming as accurate humans on perceptual tasks. They are now used in numerous and diverse applications such as drug discovery, data compression, and automating gameplay. These models increasingly grow in size with more parameters and layers, driven by two major trends. First, there a continuous rise in data complexity and sizes in many applications (Shazeer al. (017). Second, there is an increasing need for higher accuracy as models are utilized in more critical applications - such as self-driving cars and medical diagnosis (Grzywaczewski 2017). This effect is especially pronounced in computer vision and natural language processing: Model sizes are three orders of magnitude larger than they were just three years ago (Sanh al.) 2019).
With bigger model sizes, the time, computation, and memory needed to train and deploy such models also increase. Thus, is consistent challenge to design models that maximize accuracy while remaining practical with respect to the resources they need (Lee et al., 2015) Huang et al.. 20176). In this paper, we study the following question: Given number of parameters (ncurons), how to design a convolutional neural network to optimize holistically for accuracy, training cost, and inference cost?
The holistic design space is very complex. Designers of convolutional neural network models navigate a complex design landscape to address this question: First, they need to decide on network architecture. Then, they have to consider whether use a single network or build ensemble model with multiple networks. Additionally, they have to decide how many neural networks to use and their individual designs, i.e., the depth, width, and number of networks in their model. Modern

---

Published as a conference paper at ICLR 2021
applications with diverse requirements further complicate these decisions as what is desirable varies. Facebook, for instance, requires convolutional neural network models that strike specific tradcoffs between accuracy and inference time across 250 different types of smartphones (Wu al. 2019). As a result, not just accuracy but diversity of metrics such as inference time and memory usage- inform whether a model gets used (Sze et al. (017b).
Scattered conventional wisdom. There exist bits and pieces of scattered conventional wisdom to guide neural network designer. These take the form of various empirical studies that demonstrate how depth and width single neural network model relate to certain metrics such as accuracy. First, it is generally known that deeper and wider networks
improve accuracy. In fact, recent convolutional architectures - such as ResNets and DenseNets - are designed precisely to enable this outcome (He et al.) 2016: Huang al. 201 7b a). The caveat with beefing up neural network is that accuracy runs into diminishing returns as we continue to add more layers or widen existing ones (Coates al. 2011 Dauphin and Bengio, 2013). On the other hand, increasing the number of networks in the model, i.c., building ensembles, is considered a relatively robust but expensive approach to improve accuracy as ensemble models train and deploy k networks instead of one (Russakovsky et al. 2015)] Wasay et al. 2020). The consensus is to use ensembles when the goal is to achieve high accuracy without much regard to training cost, inference time, and memory usage, e.g.. competitions such as COCO and ImageNet (Lee et al. 2015 Russakovsky et al. 2015 Huang et al., 2017a| Ju et al. 2017). All these studies, however, exist in silos. Any form of cross-comparison is impossible they use different data sets, network architectures, and hardware.
Lack of robust and holistic assessment. Most past studies operate within the confines of single convolutional network and do not consider the dimension of ensemble models. Those that compare with ensembles mostly do so unfairly comparing ensembles with k networks against a model that contains only one such network (Lee et al., 2015; Russakovsky et al., 2015 Huang et al. 201 7 Ju et al. 2017). There are recent studies that make this comparison under a fixed parameter budget (Chirkova et al., 2020; Kondratyuk et al.] 2020). However, these studies consider only the metric of generalization accuracy and explore very small part of the design space - two different classes of convolutional architectures with single depth.
holistic analysis needs to include resource-related metrics such as training time, inference cost, and memory usage. All these metrics are critical for practical applications (Sze et al. 2017a: Wu et al. 2019). Furthermore, to provide reliable guidance to a model designer, a robust comparison needs to consider range of architectures and model sizes with various depth and width configurations. This is critical, especially because varying just the width of convolutional networks in isolation, as done
by recent studies (Chirkova al. 2020 Kondratyuk et al. (2020), is known be far less effective to improve accuracy (Eigen et al. 2013 Ba and Caruana 2014).
Single networks vs. ensembles. In this paper, we bridge the gap in the understanding of the design space by providing answers to the following questions. Given specific requirements in terms of accuracy, training time, and inference time, should we train and deploy convolutional model with a single network or one that contains an ensemble of networks? How should we design the networks within an ensemble? As these constraints and requirements evolve, should switch between these alternatives, why, and when?
Method. We introduce the following methodology to map the design space accurately. Since there is no robust theoretical framework to consistently analyze the design space and the complex interactions among its many parameters and metrics, we develop detailed and extensive experimental framework to isolate the impact of the critical design knobs: (i) depth, (ii) width, and (iii) number of networks, on all relevant metrics: (i) accuracy, (ii) training time, (iii) inference time, and (iv) memory usage. Crucially the number of parameters is a control knob in our framework, and we only compare alternatives under the same parameter budget. To establish the robustness of our findings, we experiment across various architectures, data complexities, and classification tasks. We present and analyze data amounting to one year of GPU run time. We also explain trends breaking down metrics into their constituents when necessary.
Results: The Ensemble Switchover Threshold (EST). (i) Contrary to conventional wisdom, we show that when we make a holistic and robust comparison between single convolutional networks
2

---

Published as a conference paper at ICLR 2021
Param.: [S]] Depth: d Width:
Param.:
Depth: d| Width: w'
Param.: [S]] Depth: d'| Width:
Depth
E2
E3
Width
(a) Single network
(b) Depth-oquivalent ensemble
(c) Width-equivalent ensemble
We explore a design space consisting of three design classes: (a) Single convolutional network models, (b) Depth-equivalent ensembles, and (c) Width-equivalent ensembles. The two ensemble design classes are created by distributing either the width factor or the depth corresponding the single network amongst the ensemble networks while keeping the other factor fixed.
Figure I:
and ensembles networks, we discover a vast design space where ensembles provide not just better overall accuracy but also train faster compared to single network. (ii) Specifically, we uncover the Ensemble Switchover Threshold (EST). This the amount of resources (measured in terms of the number of parameters and training epochs) beyond which ensembles provide superior generalization accuracy to a single model. (ii) We show that EST occurs consistently across numerous data sets and architectures. (iv) We demonstrate that the number of networks in ensemble and their individual designs determine the EST. (v) Ensembles can also provide comparable inference times for a considerable part of the design space. (vi) We also show that ensembles require significantly less memory to train for the same number of parameters. (vii) Finally, we make available superset of our results for visual exploration and help with model design at: das lab. seas hww.aard. www.:a/dd..ddnnnoooo-o-eeee
2 FRAMEWORK: DESIGN SPACE
The design space we explore consists of single convolutional neural network models and two classes of architecturall--onoogee design knobs depth and width on the quality and cost of an ensemble design. We first describe how we ensure robust comparison of alternative model designs and then explain the degrees of freedom we explore.
ensembles. These ensemble classes help isolate the effect of the two
Establishing grounds for l'air ranking, A key element of our framework is that the possible model designs are compared each other only under equivalent resources. We ensure this by only comparing designs that have the same number of parameters. This comparison allows us to separate the quality of a design from the amount of resources given to it. Another way to think about this that given parameter budget, we can investigate how the three design classes rank for all relevant metrics (training and inference time, accuracy, and memory usage).
We fix the number of parameters because of its two distinctive properties over other metrics (that could have fixed), such as training time, inference time, or accuracy: First, the number parameters of network is directly proportional to all other resource-relann related metrics (Jain et al.) 2020 Wasay jet al. (2200. Second, the number of parameters is independent of the hardware or the software platform used and can be computed exactly from network specification.
The single network versus ensemble design space. Our design space considers a convolutional neural network architecture ((w.d) from class of neural network architectures C. ((w,d) has width factor w, depth d, and number of parameters |S). Similarly an ensemble is described as E [E ... E Ensembles are architecturally. saaiitecuraally-homgennous i.e., all ensemble networks E₁ ... E& have the same architecture and each network has JE / parameters. When we compare single network (( from C' with an ensemble E we ensure that E Eg E C. and E + = SS].
The reason why we restrict the design space homogenous ensembles is to reduce the otherwise intractably large space"] of all possible ensembles given single network to size that we can
Given a single network with |S] number of parameters, there are {1S (Stirling number of the second kind) many ways of forming ensembles of size k. This number grows at similar rate to exponential polynomials, Boyadzhiev 2009) e.g., (100) 50
3

---

Published as a conference paper at ICLR 2021
feasibly and thoroughly experiment with and reason about. Furthermore, many neural network ensembles introduced in research and used in practice are similarly homogenous, for instance, SnapShot Ensembles and Fast Geometric Ensembles (Huang et al. 2017a; Garipov al. 2018). Additionally, our method provides a deterministic procedure of going between single network models and ensembles given a certain amount of parameters. Major sources of diversity in neural network ensembles are random weight initialization and stochastic training, both of which we incorporate in our framework.
and width--qqivvlent ensembles. Convolutional neural network architectures determined by two design knobs the depth and the width factor. Corresponding to these two design knobs, we create two classes of ensembles: depth cepth-equivalent ensembles and width-equivalent ensembles. These are depicted in Figure T In depth-equivalent ensembles, the depth of the individual ensemble networks is the same as S (i.e., d), and the width factor is set to the highest possible value (i.e., w') without exceeding the parameter budget of [S]. In width-equivalent ensembles, hand, the width factor is conserved across all ensemble networks (i.e., w), and the depth is modulated the highest possible value (i.e., d') without exceeding |S]:
the other
w/ k. | di
< [S(w.d)
k- [E, (w +1
d' k- IE, 1 S [S(wa)) < <k IE |E + +1)
The above definition follows that neural networks in depth-equivalent ensembles have higher depth than those in width-equ equivalent ensembles. Width Width-equivalent ensembles contain wider neural networks than their depth-equivalent counterparts. In this way, we isolate and study the effect of depth and width on ensemble accuracy and resource requirement.
Overall, our design space spans three classes of convolutional neural network designs: (i) single network models, (ii) width-equivalens ensembles, and (iii) depth-equivalent ensembles. Every class contains several model designs instantiated by the four-tuple {w, d. C). We next describe how we designed an exhaustive experimental framework to cover various configurations of these four-tuples.
3 FRAMEWORK: DATA, ARCHITECTURES, AND METRICS
Datasets and architectures. We include widely-used tatte-of-theaas network architectures and data sets in study. These include DenseNets, and (Wide) ResNets architectures as well as SVHN, C10 and C100, and ImageNet datasets. Table in the appendix summarizes these networks and training data sets as well corresponding hyperparameters. We implement our experimental framework in PyTorch and used an Nvidia V GPU to run all experiments.
Evaluation metries. We study all three design classes single network, width- and depth-equivalent ensembles - across five metrics: (i) generalization accuracy, (ii) training time per epoch, (iii) time to accuracy, (iv) inference time, and (v) memory usage. When considered together, these metrics provide a holistic picture of the quality and practicality of models.
4 ENSEMBLES OUTPERFORM SINGLE NETWORK MODELS AFTER A Low TO MODERATE PARAMETER THRESHOLD
We observe that both classes of ensembles - depth- and models after a resource threshold. We call this threshold the Ensemble Switchover Threshold (EST). Beyond the EST, ensemble models achieve to 3 percent lower test error rates (across various architectures and data sets) compared with single networks having the same number of parameters.
-otpuiivallnttoupprroooo single network
The EST appears consistently across wide range of data sets and architectures (Figure 2 a) through Figure ()) as well as ensemble sizes (Figure ( a) through Figure Fe) and appendix Figure B a) through Figure c)). In these figures, we use discrete heat maps to visualize which of the three design classes - single network models (single), ensembles (weq)- dominates in terms of generalization accuracy for a given resource budget. This resource budget takes the form of the number of parameters (on the z-axis) and epochs (on the y-axis). We also mark areas where both classes of ensembles outperform single network models. Figure 3 shows the test error rates achieved on various data sets for DenseNet models. We present this metric for the rest of the architectures in Figure A in the appendix.
ensembles (deq), and width-equivalent
4

---

Published as a conference paper at ICLR 2021
DenseNets CIEAR-10
(b) DenseNess CIEAR-1O0 (km4)
DenseNets ImageNet
both
numbbrr of spppctss
weq unly
40
dog only
10
single
(9) IC
801 1
and 19
widhh factor depch) number parametess
CIFAR-IO
ReoNets CIFAR-LOO
both ens.
number of epocits
only
deg only
90
single
width factor depth number
Figure 2: The Ensemble Switchover Threshold (EST) occurs consistently across various network architectures and data sets. Beyond this resource threshold, ensemble designs outperform single network models.
The occurrence of EST both expands and questions the general consensus the relative effectiveness of ensemble versus single network models. First, even when allocated the same amount of resources, ensemble models still outperform single network models. This observation expands upon past empirical studies that only show how a k-network ensemble is more accurate than any of the single network models that it contains (Lee al. 2015; Huang et al. [017a). Second, the EST occurs in low- to moodraate-resouge settings. For instance, in all of our experiments, we observe the EST at the 1M 1.5M parameter range and after no later than half of the training epochs. This trend challenges the widespread notion that neural network ensembles are useful only when we have tons of resources at our disposal (Lee et al.) [2015] Ju et al. 2017). Overall, our results indicate that ensembles of convolutional models are preferable to single network models for much wider range of use cases than previously understood.
On the superior generalization of ensembles under parameter budget. To interpret why en- sembles outperform single network models under a parameter budget, we use the phenomenon of diminishing returns on increasing model sizes. In the past, this effect has been independently investigated by [Eigen et al. (2013) and Dauphin and Bengio (2013) for a single network model. We hypothesize that as we increase the number of parameters, single network models exhibit more dimin- ishing returns and plateau faster than ensemble networks. When the single network's generalization accuracy starts showing diminishing returns, the corresponding width-equivalent and depth -euivaleuuv ensembles have smaller networks with 1/k as many parameters (assuming the parameters are spread equally along ensemble networks). These individual networks in the ensemble are affected less by the plateau because they have 1 1/k as many parameters as the single network model. Thus, utilizing these networks in an ensemble leads better generalization accuracy overall because they do not hit the threshold of the diminishing returns while still being able to benefit from the known properties that ensembles provide: (i) They enrich the space of hypotheses that are considered by the base model class and (ii) By averaging over various models, ensembles reduce the variance of base models, smoothing out variations due to initialization and the learning process (Lee et 2015).
"Wdde ResNets are an exception: This is because, compared to other convolutional architectures, Wide ResNets have an order of magnitude more parameters even for modest depths and width factors. For instance. networks presented in the Wide ResNet paper have anywhere hetween 8M to 37M parameters compared to the 1M to 10M range for NenseNets (Zagoruyko and Komodakis 2016).
5

---

Published as a conference paper at ICLR 2021
(0) DenseNets CI
(b) DanseNas CIFAR- 100 (K--4)
(c) DenseNets Tay ImageNet (k:4)
single
49.
68-
30
66
45-
44
24 111111117 101117
20101222
S0 33
L7
width factor depph) number parameters (M)
Figure 3: Ensembles arrive at lower test error rates than single network models after the EST has been reached.
DenseNess CIBAR-10
(b) DenseNets CIEAR-10 (k-6)
(C) DesseNets CIFAR-II (kms)
both ons.
number of epocts
100
199
100
only
deg only
single
width factur depth factor num puram (M)
Figure 4: The Ensemble Switchover Threshold moves to the right as we increase the number of networks in the ensemble.
Next, we parse out how the data complexity and the composition of the ensemble networks affect the EST and, in turn, the ranking of the three design classes.
Ensembles are more effective for more complex data sets. We observe that the EST shifts closer to the origin as the training data set's complexity increases. This can be seen in Figure ((a) through (ec) where train DenseNets progressively more complex data sets (CIFAR-10, 100 CIFAR-100, and Tiny ImageNet). This observation indicates that ensemble models are preferable to single models when training on more complex data for even wider range of available resources. This observation again expands the utility of ensembles. There is theoretical and empirical work establishing that ensembles do better for complex data (Bonab and Can 2017 Huang al., 201 7a). We, however, establish this phenomenon in the resource-equive eeuourceeequvvlentt setting as opposed to past studies that do for ensembles and single networks with drastically different numbers of parameters.
Large ensembles are effective under a large parameter budget. As we increase the number of networks l within ensemble without increasing the parameter budget, the overall accuracy of ensemble designs diminishes, pushing the EST to a higher resource limit. phenomenon for DenseNets and Figure B in the appendix shows it for ResNets. For instance, for DenseNet models, the EST moves from the 1.5M range for k = 4 to the 3M range for = and, then, to the 5M range for ls 8. This shift can be explained by looking at individual accuracy of ensemble networks. Figure C in the appendix shows test еггог rates of the ensemble as whole as Figure We in Figure a)) to l = 8 in Figure C())), the individual test error rates (shown as dotted lines) increases. This increase happens because individual networks' size goes down (as we keep fixed parameter budget). This observation implies that larger-size ensembles are desirable over smaller sizes only when we have sufficient parameter budget to assign to every single network in the model. As opposed previous work, experiments decouple the parameter budget from the number of networks in the model. We discover that just increasing the ensemble size without increasing the total number of parameters hurts accuracy.
demonstratss this
well as the average test error rates of individual ensemble networks corresponding
observe that as we increase the number of networks & (from
C[a)

---

Published as a conference paper at ICLR 2021
6x- stegle woq deg
201
singe wag deg
siegle sog deg
wich facte (Agyh) taasher ou pasuadiirr (M)
wath facoe (apph) mamber
widt Sxter dopla cammber pacanetezs (ND)
(c) Wide ResNets CIFAR- 10 (k=4) Figure 5: When ensemble designs can provide better accuracy, they can also do faster than single network models (missing bars indicate that designs cannot reach single network model accuracy).
(a) DenseNets CIFAR-IO (k=4)
(b) ResNets CIFAR-I0 10 (k=4)
200
vag
paining tnn.giimeeooooo sprrcpps
single woo dog
single
del
200
1000
11500
Witt Suzer (dagt) rumber of prrmeeree (M)
wilts Batorr dapsi mummber p (24)
width Exter darti (aamber pacameteza (MI
(a) DenseNets CIFAR-10 (k=4)
(b) ResNets CIFAR-10 10 (k=4)
(c) Wide ResNets CIFAR-10 (k=4)
Figure 6: eouth-equivalent ensembles take longer to train per epoch as compared to single network
models. Width-equivalent ensembles take comparable time.
Depth-equivalent ensembles outperform width-equivalent ensembles. We observe that depth- equivalent ensembles are overall more accurate than width-equivaler ensembles (as shown in Figure 3 and appendix Figure Ap. They also consistently demonstrate EST a lower resource range. This can be explained by the fact that modern convolutional neural network architectures provide better accuracy with increasing depth (Eigen et al., 2013) Urban et al.] (016). Here, depth-equivalent ensembles have deeper ensemble networks with better individual accuracy. Thus, when used together in an ensemble, they also provide better ensemble accuracy. This way, when designing ensemble models for high accuracy, deeper networks are preferable to wider networks.
EST vs. Memory Split Advantage. For a limited part of the design space, recent work has observed the existence of parameter limit beyond which depth-equivalent ensembles outperform single networks. This is termed as the Memory Split Advantage or the MSA (Kondratyuk al. 2020). The EST, however, is not defined just with respect to the number of parameters but also the number of training epochs, inference cost, and memory usage as by only looking at these metrics in conjunction. we can get complete picture of the relative effectiveness of ensembles vs. single networks. In this way, the EST subsumes the MSA, and we also verify the MSA across a significantly larger design space (e.g.. consider 3x more data sets and twice as many architecture types) than has been done before.
5 ENSEMBLES TRAIN FASTER AND PROVIDE COMPARABLE INFERENCE TIME
First, we analyze the training time. Despite taking longer per epoch, both ensemble classes achieve the accuracy of single network models significantly faster for a considerable part of the design space (c.g.. 1.2x 5x faster across our experiments). This happens after the EST has been reached, i.c.. when ensemble designs can provide better accuracy, they can also do so faster than single network models. This can be seen in Figure 5 Here, we plot the total training time needed for any of the three design classes to achieve the maximum accuracy of single network models under the same parameter budget. F shows the corresponding training time per epoch.
The combined depth determines per epoch training time. We observe that both classes of ensem- bles, on average, take longer train per epoch as compared to single network models as they train
7

---

Published as a conference paper at ICLR 2021
Infornoetie teregg (tre)
Infowwein line geilngg
sngle weg dag
siogle dag
tnorge
-Sngle wol dog
e2-
150000
wilh Back (icpth mamber parmmciion (M)
dapii mazzhrr parazsters
wishh facioe
depti
Iuanher paramaders (M)
(a) DenseNets C10 (k=4)
(b) ResNets CIFAR-I0 11FAR-10C0 (k=4)
(c) Wide ResNets CIFAR- (k=4)
Figure 7: Width-equivalent ensembles take comparable time to single network models for inference. Depth-equivalent ensembles take significantly longer.
networks instead of one. How much more time ensembles take per epoch depends heavily on the ensemble networks' design: This additional time is negligible for width-equivalent ensembles whereas, for depth-equivalent ensembles, it results in 2x more expensive per epoch training. This trend can be explained by how the training time per epoch scales with respect to the width and depth of convolutional neural network models. This ultimately connects to how GPUs process data, which favors networks with few wider layers over those that have multiple thinner layers.
We break down the training time per epoch of all designs into two constituents: time spent per layer and number of layers. Figure in the appendix shows this breakdown for various architectures. We observe that the total number of layers in a model (for ensembles, this is the sum of all networks' depth) majorly determines the training time per epoch. For the same parameter budget, depth- equivalent ensembles have proportionally more layers, whereas width-equivalent ensembles have proportionally more width. The average time per layer depends on the width and does not increase significantly as we move from depth-equivalent ensembles to width-equivalent ensembles. On the other hand, the total number of layers scales linearly with depth. For the same parameter budget, the total number of layers is significantly higher for eepth-equivalent ensembles than the other two designs, resulting in higher per epoch training.
From a GPU perspective, wider and shallower networks are more efficient to execute than narrower and deeper networks for the same parameter budget. This can be attributed to the massive amount of data parallelism in modern GPUs. Increasing the network's width just increases the number of kernels within layers. This increase more efficiently utilizes GPU's massive capacity to perform the same operation on multiple data items. On the other hand, deepening network introduces new layers (and operations) that require additional synchronization steps slowing down the overall execution.
Networks in ensemble models converge faster than single network models for the same param- eter budget. The fact that ensemble designs can reach the same accuracy faster than single network models can be attributed to the fact that, for the same parameter budget, all networks in the ensemble model are smaller than the single network model. Smaller networks are known to converge faster albeit to lower accuracy than larger networks.
However, we observe that the distinct advantage ensemble designs provide over the single model is that when we use smaller networks in ensemble, we get the best of both worlds. We converge faster at an individual network level, and ensembling makes up for the generalization accuracy.
Overall, these observations again question the conventional wisdom of ensembles being significantly slower to train than single network models. When we analyze the design space under a fixed parameter budget, uncover that for vast range of the design space: (i) width-equivalent ensembles introduce negligible overhead to per epoch training time as compared to single network models and (ii) both ensemble designs achieve and surpass accuracy of single network models in considerably less training time.
Width eeuiileequivaentt ensembles provide competitive inference time. We provide the inference time per image in Figure and observe similar trend to training time per epoch. While depth-equivalent ensembles significantly slower, width-equivvlent ensembles provide comparable inference speed to single network models. Again, this questions conventional wisdom that expects ensembles to be substantially slower in inference.
8

---

Published as a conference paper at ICLR 2021
manery (889
cogle
azggle dog
1E20000
(ipth permato (M)
parsmomr (M4)
(a) DenseNets CIFAR-IO (k=4)
(b) ResNets CIFAR-IO (k=4)
(c) Wide ResNets CIFAR-I0 (k=4)
Figure 8: Both classes of ensemble models are significantly more memory efficient.
ENSEMBLES ARE MEMORY-EFFICIEN
Regarding memory usage we observe that the trend favors both classes of ensemble designs over single network models. Figure ensembles, width-eque equivalent ensembles, and single network models. This is the minimum amount of memory that a GPU needs to train any of these designs for the batch sizes provided in Table A This memory is majorly used to store model parameters and intermediate results (Jain et al. 2020).
8 provides the amount of memory used as we train depth-equivalent
The superior memory efficiency of ensemble models is because when we train kenewworks ensemble, at any point during the training process, we only need as much memory to train one of the k networks (having& & as many parameters compared to the single network). This observation has two important implications: First, we can use larger batch sizes for the same GPU while training an ensemble of networks. This, for instance, is useful when training complex data sets such as ImageNet (Smith can feasibly train the same number of parameters in ensemble using
et al. 2018). Additionally, lower-end GPUs with less memory.
Additional results. We show that the same resource-related trends hold for the rest of architectures and data sets from Table A| Figure D Figure F. Figure Gi. and Figure H demonstrate these trends for metrics of time to accuracy, time per epoch, inference time, and memory usage respectively. We also provide results on a downsampled version of ImageNet-IK data set, CIFAR- 100 with data augmentation, and SVHN data sets in Figure I
Figure D
and Figure in in the appendix respectively.
Table B through Ta provide the conversion between the width factor and depth of single network models and the two ensemble design classes.
7 THE COMPLETE PICTURE
The results presented question conventional wisdom concerning the design decision on whether to use an ensemble of convolutional networks or not. By creating a detailed framework that (a) allows to fix resources and (b) spans a large design space, we show that for a considerable part of the design space, given an amount of resources, ensembles (i) achieve better accuracy than single network models, (ii) train faster, (iii) provide comparable inference, and (iv) need much less memory. Future work includes the addition of fast ensemble training approaches (such as SnapShot, TreeNets, and MotherNets (Lee et al. 2015 Huang et al.) 2017a| Wasay et al. 2020)) and parallel training, both of which can move EST further favor of ensembles.
Limitations and future directions. The analysis framework as presented in this paper covers extensively homogeneous ensembles. There are several opportunities for parts of the design space not covered by this work where it would be extremely useful to verify the results further. In particular, this includes (1) heterogeneous ensembles (e.g., various ratios of width-equivalent and depth-equivalent networks), (2) arbitrary network architectures (including using network architecture search approaches), and (3) application domains other than image classification such as object detection, machine translation, and deep generative models.
Acknowledgements. Harvard DASlab member Pablo R. Ruiz, Haochen Yang, Longshen Ou, and Ziyi Guo helped with parts of the implementation and the demo. This work is partly funded by the USA Department of Energy Project DE-SCC020000.

---

Published as a conference paper at ICLR 2021
REFERENCES
J. Ba and R. Caruana. Do deep nets really need be deep? In Advances in neural information 2014.
processing systems, pages
H. R. Bonab and F. Can. Less is more: A comprchensive framework for the number of components of ensemble classifiers. arXiv preprint pttp:/7999..22 2017.
K. N. Boyadzhiev. Exponential polynomials, stirling numbers, and evaluation of some gamma integrals. In Abstract and Applied Analysis. Hindawi, 2009.
N. Chirkova, E. Lobacheva, and D. Vetrov. Deep ensembles on fixed memory budget: One wide network or several thinner ones?, 2020.
Chrabaszez, Loshchilov, and F. Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint 1 2017.
A. Coates, A. Ng. and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 213-223, 2011.
E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition Workshops, pages 702-703, 2020.
Y. N. Dauphin and Y. Bengio. Big neural networks waste capacity. arXiv preprint arXiv: 2013.
D. Eigen, J. Rolfe, R. Fergus, and Y. LeCun. Understanding deep architectures using a recursive convolutional network. arXiv preprint 13Xiv:1312.1847 2013.
T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems, pages 8789-8798, 2018.
A. Grzywaczewski. lenge of scale. Training ai for self-driving vehicles: the chal- seaining-self-daee htw.aaa--aa..iiaaaaaiiiiooooaaaa
https://deveoppe:: nvidia wwm/ blog/ 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger. Snapshot ensembles: Train get for free. arXiv preprint orXiv::7044 00900, 2017a
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017b.
Jain, A. Jain, A. Nrusimha, A. Gholami, P. Abbeel, J. Gonzalez, K. Keutzer, and Stoica. Checkmate: Breaking the memory wall with optimal tensor rematerialization. Proceedings of Machine Learning and Systems, 2:497-511, 2020.
C. Ju, A. Bibaut, and M. van der Laan. The relative performance of ensemble methods with deep convolutional neural networks for image classification. arXiv preprint TrXYv:100.001666 2017.
D. Kondratyuk, M. Tan, M. Brown, and B. Gong. When ensembling smaller models is more efficient than single large models, 2020.
Lee, S. Purushwalkam, M. Cogswell, D. Crandall, and D. Batra. Why m heads are better than one: Training diverse ensemble of deep networks. arXiv preprint arXiv: /5/ 1.551100314 2015.
O. Russakovsky. J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1 11(3):211-255 2015.
10

---

Published as a conference paper at ICLR 2021
V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01100, 2019.
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint 1rXiv:1701.666: 2017.
S. L. Smith, P.-J Kindermans, C. Ying, and Q. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
V. Sze, Y. Chen, J. S. Einer, A. Suleiman, and Zhang. Hardware for machine learning: Challenges and opportunities. In 2017 IEEE Custom Integrated Circuits Conference, CICC 2017, Austin, 7X, USA, April 30 May 3, 2017, pages 1-8, 2017a. doi: 10. N0.10999CCC.22277799666. URL https: doi org. 10. 1109/ 2017 7993626
V. Sze, Y.-H. Chen, T.-J. Yang, and S. Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):225--2329, 2017b.
G. Urban, J. Geras, S. E. Kahou, O. Aslan, S. Wang, R. Caruana, A. Mohamed, M. Philipose, and M. Richardson. Do deep convolutional nets really need be deep and convolutional? arXiv preprint arXiv:1603.05669 2016.
A. Wasay, B. Hentschel, Y. Liao, S. Chen, and S. Idreos. Mothernets: Rapid deep ensemble learning. In Proceedings of the Conference on Machine Learning and Systems (MIsyss), 2020.
C.J. Wu, D. Brooks, K. Chen, D. Chen, S. Choudhury, M. Dukhan, K. Hazelwood, E. Isaac, Y. Jia, B. Jia, et al. Machine learning at facebook: Understanding inference the edge. In 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 331-344. IEFE, 2019.
Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint 1605.00000.7 2016.

---

Published as a conference paper at ICLR 2021
APPENDIX
Table A: For all networks, we use training hyperparameters listed in their respective papers (Ir: leamiing rate, wd: weight decay).
Architecture
Data sets
Epochs
Lr schedule
batch size
Wd
DenseNet
SVHN
100
0.1, 0.01(30), 0.001(60)
128
0.001
DenseNet
C10 and C100
120
0.1, (.0), 0.001(90)
128
0.001
DenseNet
Tiny ImageNet
0.1, (.01,)), 0.001
64
0.001
DenseNet
ImageNet32-IK
90
0.1, 0.01(30))) 0.001(60)
64
0.001
ResNet
C10
250
0.1, 0.01 0,001
128
0.0001
ResNet
C100
500
0.1, 0.01 0.001(375)
128
0,0001
Wide ResNet
C10 and C100
200
0.1, 0.01 0.001(200)
128
0.0001
(n) RocNess CIPAR-18 (k-4)
(b) ResNets CIFAR-1O0 (Kx44)
(6) VOGNN CIFAR-10 (k-4)
Lost sroor ralo (90)
24-
16
1S9 100 in
250
MIS 4981
width factor depth jaumber of parameters (M)
Figure A: Ensembles arrive at lower test error rates than single network models after the EST has been reached.
A ON THE INSTABILITY OF WIDTH- -QUTH-EQUIVALENT ENSEMBLES
The performance width-equivalent ensembles (weq ensembles) exhibits unstable behavior. In particular, it has local spikes when it comes the test error rates. This is particularly pronounced for the ResNet ensembles (Figure A[a) and Figure AJb).
Our interpretation of this phenomenon is that these local spikes have do with the relative depth of networks in the width-equivalent ensemble designs. When comparing designs that are close together in the parameter range, we observe that the designs with more depth (of ensemble networks) generally outperform those with less depth even if the latter have more parameters. The depth of networks in the weq ensemble plays more dominant role than the total number of parameters they have. As an example of this, consider Figure A(a) and A(b) in the revised version of the paper (ResNet CIFAR-10 10 (k=4) and ResNet CIFAR- 100 (k=4). Weq exhibit three spikes 2.24M, 3.28M, and 5.03M parameters. All three of these points are flanked on both sides by designs that have similar number parameters but more depth.
This observation is consistent with past observations that depth is more influential in determining the accuracy of networks (Eigen et al. 2013).
B ADDITIONAL EXPERIMENTS IN UNDERFITTING SCENARIOS
We now provide experiments two additional data sets from underfitting scenarios: (i) ImageNet- IK-32 a downsampled version of ImageNet- k with 1.2 million images and 1000 labels (every image downsampled to be of size 32 32), and (ii) CIFAR-100 with aggressive rand-augment (Cubuk et al.! 2020).
The result of these experiment can be seen in Figure and Figure J respectively. Overall, our observations hold for these data sets as well. After a certain parameter budget (defined in terms of number of parameters and training epochs), ensembles of neural networks consistently outperform the single network and also reach that accuracy faster when compared to the single model.
12

---

Published as a conference paper at ICLR 2021
RosNNe CIEAR-IO (k-4)
(b) ResNer CIEAR-TO (km6)
(c) RedNer CIEAR-1O (k (k-8)
both
numbee of epochss
268
weq unly
150
168
190
199
dog only
30
20
singlc
DE
widhh factor depth number of parameters (M)
Figure B: The Ensemble Switchover Threshold moves to the right increase the number of networks in the ensemble. Here, we demonstrate this phenomenon for ResNet models.
(1-6)
(k-5)
single
14
dog ora, act
15-
10-
panm
Figure C: As we increase the size of ensembles, accuracy individual networks in the ensemble decreases. This results in an overall reduction in ensemble accuracy shifting the EST to the high- resource space.
thre wagghaatiin (m91)
600-
stngle wog deg
single
deg
1988-
153
653
(App))
(M4
(a) DenseNets CIFAR-100 (k=4) (b) DenseNets Tiny ImageNet (k=4)
(c) ResNets CIFAR-100 (k=4)
Figure D: When ensemble designs can provide better accuracy, they can also do so faster than single network models (missing bars indicate that designs cannot reach single network model accuracy).
13

---

Published as a conference paper at ICLR 2021
single
tasuppiin layer (nin]
Sinle
iinepe lever(mins
sangle deg
per
0
tecal foues
una leyer
40600
(dafik (oamber (Cgarannttss
with (opth) inabbe permmnc O6
With Sxtor dayli (xubeer pacanetes (M)
(a) DenseNets CIFAR-10 10
(b) ResNets on CIFAR-10
(c) Wide Res CIFAR-10
Figure E: We break down per epoch training time into: (i) time spent per layer and (ii) total number of layers. We observe that the total number of layers in the model more significantly determines the per epoch training time as compared to the width.
siegle
-20
1030-
ysining
100+
(M
Prye
parmmnor (M)
(a) DenseNets on CIFAR- 100
(b) DenseNets on Tiny ImageNet
(c) ResNets on CIFAR-100
Figure F: Depth-equivalent (deq) ensembles take longer to train per epoch as compared to single
network models. Width-equivalent ensembles (weq) take comparable time.
cingle wag dag
sizgle dog
Infersotinn per aag
single
deg
car
Width Buttrr deph memberet (amomatze((
porsmiior (M)
(a) DenseNets on CIFAR-100
(b) DenseNets on Tiny ImageNet
(c) ResNets on CIFAR-100
Figure G: Ensemble models take longer to perform inference.
(GB))
engle
singl: dec
TAX momoryy many
1462
LE
permiciion
(a) DenseNets CIFAR-10
(b) DenseNets on Tiny ImageNet
(c) ResNets
CIFAR-100
Figure H: Ensemble models are significantly more memory efficient.
14

---

Published as a conference paper at ICLR 2021
1000 eidd w0w.mmsy (1603
both ens.
dog
single wog dog
onlyy
dog only
single
wddh fader dapla saethe d/penaate:000 (NI
widh Seke (dgth) mazber panaeca (M)
vilk Sacke Harh matder of pananeka (M)
(a) Accuracy heat map
(b) Final test error rate
(c) Time to single model accuracy
Figure I: We observe the same accuracy and resource related trends for DenseNets trained on
ImageNet32, a downsampled version the ImageNet- 1k data set (Chrabaszez et al. 2017).
both ens. only
single wag
Tho anveeeeeraan cae C10
single N00
430
deg
12
single
21
with Destr Akph sumber (M))
with Setoe dqph Fumber permerrs(
WMD Seary (iph) rumber oepemettss (M)
(c) Time to single model accuracy Figure J: We observe the same accuracy and resource related trends for ResNets trained CIFAR 100 with agressive data augmentation to mimic underfitting scenario. In particular, we use RandAugment with N = 2 and M 14 (Cubuk al. 2020
(a) Accuracy heat map
(b) Final test error rate
Enee shee souraa (18))
Nunherffpoals
both ens.
both ens.
218-
weq only
wao only
only single
Stante
dog only
single
S73 1-1-9-12
12
with (ader] (kpta sanbes
paRanCcc (4)
panascik
(a) Total budget: 30 epochs
(b) Total budget: 90 epochs
(c) Time to single acc. (90 epochs)
Figure K: We repeat our experiments with different total training budgets and set up the learning rate schedule proportionally for every budget. Here we show results for training DenseNets on the CIFAR-10 data set. We observe the same trend as the total budget increase, ensembles provide better accuracy and can train faster.
15

---

Published as a conference paper at ICLR 2021
1990 50 stock scarms (116)))
100 ooo-nn or
weq only
Narche
dog only single
(a) Accuracy heat map
(b) Final test errоr rate
(c) Time single model accuracy
(a))der Soman 1EL
single
Infoss cnseppttieet ieniiinee
100
F g
(doth parancters
widts Sator depti number (amm
factor number parmmnderr (M)
(d) Training time per epoch
(e) Inference time per image
(f) Memory usage
Figure L: We observe similar accuracy and resource related trends for DenseNets trained on the SVHN data
Table B: DenseNet
single depth
single width factor
Weq depth
Deq width factor
40
12
19
5
40
20
19
9
40
28
19
13
40
36
19
17
40
48
19
23
64
12
28
6
64
20
31
10
64
28
31
14
64
36
31
18
64
48
31
24
12
40
6
88
20
40
10
88
28
43
14
88
36
43
18
88
48
43
24
112
12
52
6
112
20
52
10
112
28
52
14
112
36
52
18
16

---

Published as a conference paper at ICLR 2021
Table C: ResNet
single depth
single width factor
Weq depth
Deg width factor
26
16
8
24
12
26
32
8
16
26
48
24
38
16
8
8
38
24
12
38
32
8
16
38
48
8
24
50
16
14
8
50
24
14
12
50
32
14
16
50
48
14
24
16
8
62
24
14
12
62
32
16
62
48
14
24
Table D: WideResNet
single depth
single width factor
Weq depth
Deq width factor
28
10
28
8
10
3
28
12
10
5
52
4
16
1
52
8
16
3
52
12
16
5
58
16
1
58
8
16
3
58
12
16
5
17


Reviews:
[{"review_id": "uSRMED6NgdO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides a comprehensive evaluation of building ensemble networks compared to a single network while keeping the number of parameters equal. The experiments are conducted on four datasets. The comparison is conducted for test error, training time per epoch, time to optimization, inference time, and memory usage. \n\nThe paper is well written and it is a pleasure to read it. The experiments are thorough and comparisons are thoughtful. One minor comment on readability is that a lot of results are pointing to the appendix. However, I do understand that this is due to the limited space. The authors may try to move similar results to the appendix and bring the other important results back in the paper. For example, it might be okay to limit figure 2 to three diverse plots instead of all 6 plots. \n\nMinor comments/questions:\n\n- Width equivalent ensemble shows quite an unstable result (Figure A). Do authors have any intuition about it?\n- I would like to see more discussion on the generalization capability of ensemble versus individual networks. Authors have mentioned it as one of their evaluation metrics but very little light is shed on this.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qnjnttSzD1Z", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper establish a robust and holistic framework to compare scaling up an ensemble with scaling up a single networks, where test accuracy, number of paramaters, inference time, memory consumption and training time to converge are considered.  To reduce the intractably large design space of scaling up an ensemble, the author mainly investigate two types of ensembles: depth-equivalent and width-equivalent ensembles.  Through extensive experiments on SVHN, CIFAR-10, CIFAR-100, and Tiny ImageNet with VGGNets, ResNets, DenseNets and WideResNets, the authors discovered an surprising and consistently emerging phenomenon named The Ensemble Switchover Threshold: When the amount of resources (measured by number of parameters, training cost) is beyond this threshold, ensembles methods provide better performance and computation trade-off. \n\nOverall, I recommend this paper to be accepted because: \n\n1. This is the first paper to  conduct an extensive and robust comparison between scaling up a single model and scaling up an ensemble. \n2. The EST phenomenon uncovered by this work is supervising and country to common knowledge. \n\nMy major concerns:\n1. Though the author conducted extensive results, but most of the experiments are conducted in the scenario where overfitting is the major problem affecting the test accuracy rather than under-fitting due to the authors' limited computing resources(One V100 as mentioned).  I am interested if the EST still holds in under-fitting scenarios.  On possible way to investigate this phenomenon in the under-fitting without huge training cost is to mimic the under-fitting scenarios by carrying out aggressive data augmentations during training, like RandAug[1].  \n2. On the details of measure the training time needed for different models to reach a fixed performance in your figure 6.  As you are using the step-wise constant learning rate schedule (Tabel A in appendix), which easily leads to a flattened loss curve during training. To measure the performance of one method with a set of training budgets,  you should adjust the learning rate schedule accordingly and measure the performance separately in different runs.  It maybe be wrong if you are measuring the performance of different checkpoints generated in one run.  \n\n[1] Cubuk, Ekin D., et al. \"Randaugment: Practical automated data augmentation with a reduced search space.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.\n\nOverall, it is an interesting paper. \n\n==================Post-discussion Update===========================\n\nThanks to the authors for addressing my concerns. \n\nHowever, after viewing the other responses (especially the comments from Ekaterina Lobacheva) as well as the author's explanation, I think this submission missed out some quite important references. In addition, its contribution over previous works appears to be marginal after viewing these references. Therefore, I would like to lower my rating from 6 to 5. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "_i-DD9z6aJP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--------------\nThe paper evaluates an under-explored space of neural models: ensemble of smaller networks (shallower or narrower). Extensive experiments show that when growing the total capacity (number of parameters) beyond a threshold, these ensembles get better performance than a single network, and train faster.\n\nPros\n-------\n- Sound methodology to quantify the \"conventional wisdom\" around ensembles of networks, and explore a larger design space\n- Good experimental design and exploration, especially with limited resources (1 GPU-year)\n- Really interesting main result, how ensembles typically get better at exploiting additional capacity than single networks, once the capacity is large enough\n\nCons\n--------\n- Code is not released\n- Only small to medium datasets are used (nothing like the scale of ImageNet-21k for instance), so some of the observations may not hold for larger datasets or models (although the main conclusions are likely to)\n\nRecommendation\n---------------------------\nI recommend **acceptance** of this paper, as it provides new insight on when to use ensembles of smaller models, and justifies it by exhaustive experiments. It provides a sound basis for exploring a wider space of neural-based systems, and understanding better the strong points of ensembles.\n\nArguments\n----------------\n- A better evaluation of ensembles or neural nets, as well as their trade-offs, is quite *significant* for the machine learning community, as it could change how we think of structuring systems.\n- Thinking of exploiting ensembles of smaller models, and not only for scaling up beyond how one model can reasonably be, is novel and *original* to my knowledge, and goes against the usual principle of \"jointly training everything\" that is pervasive in the deep learning community. The results are somewhat unexpected, and clearly show the advantage of this approach.\n- The *quality* of the methodology is really high, as it enabled exploration of a wide space in a reasonable and straightforward way (with architecturally-homogenous ensembles), when an exhaustive enumeration would be impossible. The metrics evaluated strongly support the conclusion.\n- The design, experiments, and results are very clearly explained, are exposed in a straightforward way and easy to follow.\n\nQuestions\n---------------\nThe main question I had reading the article was \"how useful is that additional part of the design space?\". The comparisons were mostly between single and ensembles for a given number of parameters, but the absolute performance of neither was addressed.\nPlots like Figure 2 do not show the absolute accuracy reached, or which part of that plane correspond to \"reasonable\" performance, which makes it harder to understand if a victory of ensembles in part of that space is important or not. For instance, in the lower-left corner of plots of Figure 3, one could think that the better performance of ensemble methods does not matter much since it is before the model has converged, and that the performance of all models would be low.\nAs another example, figure 4 shows that for larger models, ensembles tend to reach the accuracy of a single *comparable* model, and do so faster, but that would not mean much if the performance of that single model was bad for some reason (overfitting, for instance).\nIt is only in the Appendix (Figure A) that we see that the performance of ensemble actually gets better than the *best* single model, rather than only the comparable one. I think that figure deserves to be in the main paper, and would help make the argument.\n\nAdditional feedback\n----------------------------\n- In Figure 1 (b) and (c), colors seem to be mixed up between E2 and E3.\n- Duplicate reference to Lee et. al (2015): a and b are the same article.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PKMDtYNfLoT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper addresses when to use a single network model vs an ensemble of convolutional neural network models based on resource budgets. The authors challenge the notion that ensemble methods should only be used when resources are a non-issue. The authors compare single networks to width-equivalent and depth-equivalent ensemble methods for SVHN, cifar10, cifar100 and tiny imagenet across multiple network architectures and describe the 'Ensemble Switchover Threshold (EST)', the amount of resources beyond which ensembles provide better generalization accuracy than single models. \n\nStrengths:\nThe authors robustly test this threshold where an ensemble method outperforms single network models. \n\nThe authors consider more than just accuracy but also at inference cost and memory usage which are important parameters for deployable code. \n\nThe provided demo is an impressive and straight-forward visualization tool for understanding when to use which type of model. \n\nThe authors explore how performance changes across number of models in the appendix -- a question I thought of while reading the paper and did not expect to get answered. \n\nWeaknesses:\nThere are some inevitable limitations to this type of study that make the EST hard to interpret. The authors do not include heterogenous ensemble methods or other hyper-parameterization that might vary between the three setups: single model, width and depth-equivalent. \n\nThe title and majority of the abstract brag a great scope than the paper considers. It should be made clear in the title that only CNNs are considered for this study. The work done within this scope is thorough and impressive, and stands alone in its value. \n\nQuestions:\nThe authors assumption that the number of parameters is directly proportional to the resources used seems reasonable but I did wonder if there was a citation or related work to back this point up? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper looks into performance of a single network vs ensemble CNN networks of similar no. of parameters, through lens of accuracy, training time, memory used, inference time. 
the authors show that after some threshold, the ensemble model starts to outperform a single model and make better use of its capacity.
although this is not the first paper to look into this question and there are two other earlier results from this year, the current paper looks into more measures and not just accuracy.
Although initially the paper only looked at over-parameterized regime, the authors added experiments on under-parametrized case as well. moreover, the authors address the issue of only looking into small and medium sized datasets by adding two more ImageNet experiments.

I thank the authors for engaging with the reviewers, addressing their comments and updating the paper accordingly.

It's of interest for follow up work to consider large data regime and transformer style models as well.
Decision: Accept (Poster)