Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary
--------------
The paper evaluates an under-explored space of neural models: ensemble of smaller networks (shallower or narrower). Extensive experiments show that when growing the total capacity (number of parameters) beyond a threshold, these ensembles get better performance than a single network, and train faster.

Pros
-------
- Sound methodology to quantify the "conventional wisdom" around ensembles of networks, and explore a larger design space
- Good experimental design and exploration, especially with limited resources (1 GPU-year)
- Really interesting main result, how ensembles typically get better at exploiting additional capacity than single networks, once the capacity is large enough

Cons
--------
- Code is not released
- Only small to medium datasets are used (nothing like the scale of ImageNet-21k for instance), so some of the observations may not hold for larger datasets or models (although the main conclusions are likely to)

Recommendation
---------------------------
I recommend **acceptance** of this paper, as it provides new insight on when to use ensembles of smaller models, and justifies it by exhaustive experiments. It provides a sound basis for exploring a wider space of neural-based systems, and understanding better the strong points of ensembles.

Arguments
----------------
- A better evaluation of ensembles or neural nets, as well as their trade-offs, is quite *significant* for the machine learning community, as it could change how we think of structuring systems.
- Thinking of exploiting ensembles of smaller models, and not only for scaling up beyond how one model can reasonably be, is novel and *original* to my knowledge, and goes against the usual principle of "jointly training everything" that is pervasive in the deep learning community. The results are somewhat unexpected, and clearly show the advantage of this approach.
- The *quality* of the methodology is really high, as it enabled exploration of a wide space in a reasonable and straightforward way (with architecturally-homogenous ensembles), when an exhaustive enumeration would be impossible. The metrics evaluated strongly support the conclusion.
- The design, experiments, and results are very clearly explained, are exposed in a straightforward way and easy to follow.

Questions
---------------
The main question I had reading the article was "how useful is that additional part of the design space?". The comparisons were mostly between single and ensembles for a given number of parameters, but the absolute performance of neither was addressed.
Plots like Figure 2 do not show the absolute accuracy reached, or which part of that plane correspond to "reasonable" performance, which makes it harder to understand if a victory of ensembles in part of that space is important or not. For instance, in the lower-left corner of plots of Figure 3, one could think that the better performance of ensemble methods does not matter much since it is before the model has converged, and that the performance of all models would be low.
As another example, figure 4 shows that for larger models, ensembles tend to reach the accuracy of a single *comparable* model, and do so faster, but that would not mean much if the performance of that single model was bad for some reason (overfitting, for instance).
It is only in the Appendix (Figure A) that we see that the performance of ensemble actually gets better than the *best* single model, rather than only the comparable one. I think that figure deserves to be in the main paper, and would help make the argument.

Additional feedback
----------------------------
- In Figure 1 (b) and (c), colors seem to be mixed up between E2 and E3.
- Duplicate reference to Lee et. al (2015): a and b are the same article.
review_title: Well-designed exploration and evaluation of ensembles of smaller neural nets, showing they usually outperform single comparable networks.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: This paper addresses when to use a single network model vs an ensemble of convolutional neural network models based on resource budgets. The authors challenge the notion that ensemble methods should only be used when resources are a non-issue. The authors compare single networks to width-equivalent and depth-equivalent ensemble methods for SVHN, cifar10, cifar100 and tiny imagenet across multiple network architectures and describe the 'Ensemble Switchover Threshold (EST)', the amount of resources beyond which ensembles provide better generalization accuracy than single models. 

Strengths:
The authors robustly test this threshold where an ensemble method outperforms single network models. 

The authors consider more than just accuracy but also at inference cost and memory usage which are important parameters for deployable code. 

The provided demo is an impressive and straight-forward visualization tool for understanding when to use which type of model. 

The authors explore how performance changes across number of models in the appendix -- a question I thought of while reading the paper and did not expect to get answered. 

Weaknesses:
There are some inevitable limitations to this type of study that make the EST hard to interpret. The authors do not include heterogenous ensemble methods or other hyper-parameterization that might vary between the three setups: single model, width and depth-equivalent. 

The title and majority of the abstract brag a great scope than the paper considers. It should be made clear in the title that only CNNs are considered for this study. The work done within this scope is thorough and impressive, and stands alone in its value. 

Questions:
The authors assumption that the number of parameters is directly proportional to the resources used seems reasonable but I did wonder if there was a citation or related work to back this point up? 
review_title: Demonstration of when to use ensemble methods of CNNs

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper establish a robust and holistic framework to compare scaling up an ensemble with scaling up a single networks, where test accuracy, number of paramaters, inference time, memory consumption and training time to converge are considered.  To reduce the intractably large design space of scaling up an ensemble, the author mainly investigate two types of ensembles: depth-equivalent and width-equivalent ensembles.  Through extensive experiments on SVHN, CIFAR-10, CIFAR-100, and Tiny ImageNet with VGGNets, ResNets, DenseNets and WideResNets, the authors discovered an surprising and consistently emerging phenomenon named The Ensemble Switchover Threshold: When the amount of resources (measured by number of parameters, training cost) is beyond this threshold, ensembles methods provide better performance and computation trade-off. 

Overall, I recommend this paper to be accepted because: 

1. This is the first paper to  conduct an extensive and robust comparison between scaling up a single model and scaling up an ensemble. 
2. The EST phenomenon uncovered by this work is supervising and country to common knowledge. 

My major concerns:
1. Though the author conducted extensive results, but most of the experiments are conducted in the scenario where overfitting is the major problem affecting the test accuracy rather than under-fitting due to the authors' limited computing resources(One V100 as mentioned).  I am interested if the EST still holds in under-fitting scenarios.  On possible way to investigate this phenomenon in the under-fitting without huge training cost is to mimic the under-fitting scenarios by carrying out aggressive data augmentations during training, like RandAug[1].  
2. On the details of measure the training time needed for different models to reach a fixed performance in your figure 6.  As you are using the step-wise constant learning rate schedule (Tabel A in appendix), which easily leads to a flattened loss curve during training. To measure the performance of one method with a set of training budgets,  you should adjust the learning rate schedule accordingly and measure the performance separately in different runs.  It maybe be wrong if you are measuring the performance of different checkpoints generated in one run.  

[1] Cubuk, Ekin D., et al. "Randaugment: Practical automated data augmentation with a reduced search space." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.

Overall, it is an interesting paper. 

==================Post-discussion Update===========================

Thanks to the authors for addressing my concerns. 

However, after viewing the other responses (especially the comments from Ekaterina Lobacheva) as well as the author's explanation, I think this submission missed out some quite important references. In addition, its contribution over previous works appears to be marginal after viewing these references. Therefore, I would like to lower my rating from 6 to 5. 
review_title: Final review from R4

final_score: rating: 7: Good paper, accept
review_content: The paper provides a comprehensive evaluation of building ensemble networks compared to a single network while keeping the number of parameters equal. The experiments are conducted on four datasets. The comparison is conducted for test error, training time per epoch, time to optimization, inference time, and memory usage. 

The paper is well written and it is a pleasure to read it. The experiments are thorough and comparisons are thoughtful. One minor comment on readability is that a lot of results are pointing to the appendix. However, I do understand that this is due to the limited space. The authors may try to move similar results to the appendix and bring the other important results back in the paper. For example, it might be okay to limit figure 2 to three diverse plots instead of all 6 plots. 

Minor comments/questions:

- Width equivalent ensemble shows quite an unstable result (Figure A). Do authors have any intuition about it?
- I would like to see more discussion on the generalization capability of ensemble versus individual networks. Authors have mentioned it as one of their evaluation metrics but very little light is shed on this.
review_title: The paper provides a comprehensive evaluation of building ensemble networks compared to single network while keep the number of paramters equal. The experiments are conducted on four datasets. The comparison is conducted for test error, training time per epoch, time to optimization, inference time and memory usage.

Meta-review: The paper looks into performance of a single network vs ensemble CNN networks of similar no. of parameters, through lens of accuracy, training time, memory used, inference time. 
the authors show that after some threshold, the ensemble model starts to outperform a single model and make better use of its capacity.
although this is not the first paper to look into this question and there are two other earlier results from this year, the current paper looks into more measures and not just accuracy.
Although initially the paper only looked at over-parameterized regime, the authors added experiments on under-parametrized case as well. moreover, the authors address the issue of only looking into small and medium sized datasets by adding two more ImageNet experiments.

I thank the authors for engaging with the reviewers, addressing their comments and updating the paper accordingly.

It's of interest for follow up work to consider large data regime and transformer style models as well.
Decision: Accept (Poster)