Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tackled a studying the effect of ensembling neural networks to particularly improve accuracy in the low data regime.  The paper is well laid out and the experiments are somewhat well motivated as, much experimental study in DL has been around *larger and larger* datasets. The paper seems well written and the experiments are well constructed with means and standard deviations reported for every experiment. These experiments do match prior subsampling experiments literature (at least for Cifar-10). 
One minor call out I'd make is that Shankar Et al  (https://arxiv.org/pdf/2003.02237.pdf) shows that a non ensembled CNN has similar performance to a CNTK on subsampled Cifar-10.

My primary problem with this work is a the somewhat expected nature of the findings and the limited scope of the experiments. First of all while Cifar-10 and Cifar-100 are great datasets that researchers should benchmark on, but for this line of work I would have liked to see more. How do these trends look like on ImageNet? What about for non vision tasks, does something similar happen on SQUAD.

Furthermore I'd encourage the authors to explore the tradeoff between number of ensembles vs depth of network more. I'd like to see a 2/3d plot with depth/size of network on one axis, and number of ensmbles on the other, how does the tradeoff frontier look like? It is simply not convincing to just look at a resnet20 vs resnet-8.

I also find the loss comparison direction less interesting than the tradeoff between depth/width/number of ensembles.

Anyway the reason for my score is that this seems like a promising direction but looks like early work. With a couple more datasets and a more thorough experimentation section I'd accept this paper.
review_title: Through experiments that are consistent with prior literature, but unclear if findings are novel.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an ensemble learning method for deep networks in the low data regime. Specifically, the authors empirically compare several ensemble configurations by varying the complexity of base members given a total fixed computational budget. Experiments are conducted on CIFAR-10 and CIFAR-100 datasets. It shows that good results are obtained by keeping low the complexity of single models and increasing the ensemble dimension.

Paper strengths:
+ The paper is well written and organized. It is easy to follow.
+ The topic, i.e., ensemble learning for deep networks, is interesting and deserves further studies.

Paper weaknesses:
- The novelty of this paper is low. No significant technique contribution is performed in this paper. The proposed method is a simple weighted average ensemble. Additionally, the loss functions employed were developed in previous work, which are also not the contributions by the authors. Thus, does it seem an empirical study paper?
- If it is an empirical paper, the experiments are also weak. Only two small-scale (w.r.t. deep learning) datasets, i.e., CIFAR-10 and CIFAR-100 are conducted for evaluation comparisons. Meanwhile, only the ResNet family is utilized as the backbones. It is encouraged to perform comprehensive experiments on large-scale, diverse (e.g., object-centric data and scene-centric data) vision datasets, as well as different network architectures (e.g., ResNets, VGGs, MobileNets, etc). Current experimental results are not sufficient enough to support the conclusions of this paper.
- More analyses are required, such as attempting to reveal why the observations in this paper can happen? In addition, is there a possibility that few training data cannot support the training of big networks (e.g., ResNet-110), rather than the effectiveness of the proposed ensemble process? Thus, small data trained with small networks (e.g., ResNet-8) can achieve better results since small networks can be trained until parameter convergence.

Minor issues:
- The references are not formal. For example, several references have inconsistent formats, e.g., "In Proceedings of the 28th International Conference on machine learning (ICML-11), " of [Deisenroth and Rasmussen, 2011] vs. "In international conference on machine learning," of [Gal and Ghahramani, 2016].
- There are also several typos in this paper. The authors should carefully proofread the paper.
review_title: Official Blind Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
In this paper, the authors provide a series of experimens where they show that when dealing with a very small dataset, a single very deep network is outperformed by an ensemble of multiple more shallow networks. More specifically, the authors artificially create training sets from CIFAR10 and CIFAR100 datasets where the number of images per category is limited to 10-250 samples. Then, they compare the test performance of ResNet101, an ensemble of 5 ResNet 20 and an ensemble of 20 ResNet8, trained for classification with different loss functions, i.e. cross-entropy and cosine distance. The bottom line is that the ensembles work better and have a comparable computational complexity in FLOPs.

Strengths:
- The topic of the paper fits well in the paradigm of representation learning.
- The work demonstrates that the community does not have a good understanding of what kind of models must be used when little data is available for training and brings attention to classical techniques for variance reduction.

Weaknesses:
- The paper is basically a compilation of experiments with no explanations of the observed phenomena. The authors perform a set of experiments with already known methods and merey propose the reader to look at the results. I would like to know not only that we need to do ensembles of small networks but also why these ensembles are more efficient than a single deep network in the low data scenario. Why do we observe the difference between using cross-entropy and cosine losses, depending on the network, dataset and its size?
- The novelty of the paper is limited. It is already known from [1] that using ensemble methods in few-shot problems helps the performance a lot. Even if the authors propose a different evaluation strategy, referencing existing work in this field is still required.
- Abblation studies are missing. To be more convinced by the experiments I would like to see how the performance differes if you vary the ensemble size and the single network's capacity. That may improve our understanding of the phenomena too. Experimenting with more datasets may help to answer the question of why the behavior of different loss functions is so different between the two used datasets.
- A question of wheather a single ResNet101 with vanilla training is a fair baseline. It’s been known [2] that to achieve better results on a small-data task it is beneficial to train deeper networks with proper regularization rather than shallow networks. Using an ensemble of N networks is identical to using a single network where each layer is N times wider; each convolutional layer will have N times more filters (that could be obtained by concatenating the weights of the original network), however the convolution operation now changes from a standard to a grouped one (has N groups). The resulting output of the fused network must be averaged across the groups to match the ensemble definition exactly. The group-separated convolutions restrict representational power of the network and introduce stronger regularization, which is most likely the reason for the ensemble to perform better. If we speak about regularizing ResNet101 what kind of regularization did you introduce to adapt it to the small size dataset? It is possible that vanilla training with higher weight decay and more data augmentation is not actually efficient in the case of ResNet101 on the small datasets. Instead, it may require introducing more aggressive data aufmentation [3,4] or some structural changes must be introduced, as for example in [5].


Even though the direction of reserch is interesting and deffinitely useful for the community the work still needs development to be recommended for acceptance.


[1] - Dvornik et.al "Diversity with Cooperation: Ensemble Methods for Few-Shot Classification"
[2] - Geiger at.al "The jamming transition as a paradigm to understand the loss landscape of deep neural networks"
[3] - DeVries et.al "Improved Regularization of Convolutional Neural Networks with Cutout"
[4] - Zhang et.al "Mixup: beyond empirical risk minimization"
[5] - Gastaldi "Shake-Shake regularization"


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Update after the author's comment:

I appreciate the effort of the authors to add more experiments that all suggest that the ensembles tend to perform better in the small data regime. This makes the case stronger and the story more compelling, hence I raise my score. However, the paper is still missing the core explanations or a hint of why this may be happening, hence I still can not recommend the paper for acceptance.
review_title: The paper is missing explanations of the observed phenomena

final_score: rating: 3: Clear rejection
review_content: The paper studies the performance of ensembles of deep networks on small-data tasks taken from subsets of Cifar10 and Cifar100, with either the cross-entropy loss or the cosine loss. The authors conduct extensive experiments on these datasets with various choices of sample size, and are careful of evaluating models with comparable computational budget, by considering ResNet architectures of varying depths and with different numbers of models in the ensemble. They find that ensembles of small models tend to outperform single large models.

The approach seems promising, and the extensive experiments provide a comprehensive picture of the performance of various choices of model and ensemble sizes on the Cifar datasets.
Nevertheless, the proposed method is not compared to any existing models and regularization approaches which are applicable to small datasets, including the cited references or other approaches (e.g. [1-4]). This makes the statement of "improving the state-of-the-art" questionable. The evaluation also does not seem to perform adequate cross-validation (e.g. the authors use the "best test performance" across any epoch).

I thus encourage the authors to perform a more comprehensive evaluation of the proposed approach, and further compare to other methods. Some confidence estimates for comparing methods would also be useful, as such small datasets may lead to large variance across different choices of samples. Also, it would be interesting to see how the approach performs beyond just the Cifar dataset, perhaps in other domains where data is more scarce. Other ways to control for computation in each model of the ensemble would be interesting, e.g. how would controlling width instead of depth affect performance?

[1] Arora et al "Harnessing the power of infinitely wide deep nets on small-data tasks."
[2] Bietti et al. "A Kernel Perspective for Regularizing Deep Neural Networks"
[3] Drucker and Lecun "Improving generalization performance using double back-propagation"
[4] Miyato et al. "Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning"
review_title: needs more extensive evaluation and comparison with related methods

Meta-review: The paper received negative and borderline reviews. The reviewers have raised several concerns about the novelty of the approach and the lack of convincing experiments. The rebuttal only partially addresses these concerns. Overall, the area chair agrees with the reviewer's assessment and follows their recommendation.
Decision: Reject