Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes neural network ensembles with varying architectures, arguing these will be produce better uncertainty estimates, and considers various straightforward techniques to aggregate (ie. ensemble) the predictions from a pool of independently trained heterogeneous neural network architectures. The aggregation algorithm in this work is simply Ensemble Selection from Caruana et al., and random search / evolution methods are considered for automatically constructing the neural architectures in the base learner pool, rather than requiring these are predefined.

main_review: Two key questions are considered in this paper:

Q1) How to generate a pool of neural networks to ensemble?
Q2) How to aggregate the predictions from this NN pool into a single prediction?

There is little theory or conceptual justification for the authors' choices to address each question, so the empirical comparison must be sufficiently convicing to justfy the contribution of this paper. However, I do not find the comparisons sufficently thorough in the current paper.

Furthermore, the novelty of the proposed methods is limited (which would not be a problem if the proposed straightforward methods showed clear superiority over existing state of the art, but this is not clearly demonstrated in my opinion). A standard approach in eg AutoML systems is to do hyperparameter optimization (HPO) followed by ensembling of the models corresponding to best hyperparameter candidates. This work essentially considers the same framework, except the hyperparameter search space now includes neural architectures and rather than Bayesian optimization HPO algorithms, simple NAS algorithms are considered for searching through this space.

Regarding Q2: it is unclear what is specific to the base learners being NN models, and it seems like any arbitrary ensembling algorithm could apply. The authors here advocate for Ensemble Selection (ES) from Caruana et al., which is certainly a reasonable choice, but it would be good to see more comparisons against alternative standard ensembling algorithms. The alternative ensembling algorithms considered by the authors as baselines vs. ES are too limited and non-adaptive unlike many ensembling algorithms which exist today (and where are the empirical results that back up the claim: " these three performed comparatively or worse
204 than our choice ForwardSelect"?). Here are some straightforward alternatives off the top of my head:

A) Train a linear model or neural network to combine the predictions. Ie. stacking, unclear why this left for future work as it is simple to try out, see for example:

AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data.
Erickson et al. 2020.
https://arxiv.org/abs/2003.06505

where this has been shown to be highly performant method for ensembling heterogeneous base learners. Also blending is another standard alternative to stacking. AutoGluon also appears to use Bayesian optimization to tune architectural hyperparameters of NN base learners (contrary to what the authors write in the "AutoML and ensembles of varying architectures" section).

B) Various voting ensemble schemes that have been previously proposed, eg:
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html

Furthermore, it seems promising to exploit the fact that the base learners are all NN models to consider more interesting differentiable aggregators such that more interesting overall strategies can be considered like those of Webb et al. [49]. For example, rather than purely ensembling based on validation score, perhaps the aggregator should also consider some properties of the architectures being ensembled which might lead to better OOD uncertainty estimation, something that is not easily achieved solely thru validation-score ensemble construction.

The authors should also justify why they aggregate predicted probabilities instead of say logits, and why a simple average of these is considered rather than say a weighted combination. For example: in the baseline "Select the top M networks by validation performance", will performance improve if the resulting predictions of the networks weighted by their validation performance in the aggregator (as in Bayesian model averaging)?


Regarding Q1:

- As a baseline, why not use standard NAS algorithms (eg. ENAS, ProxylessNAS) to generate the pool? This could still be followed by ensemble-selection just as applied to pools considered in this paper. I suppose one issue is the weight-sharing scheme in standard algorithms like ENAS but it would still be insightful to see how ensembles of these weight-shared candidates perform.

- Does the pool improve if bootstrapping is used to train each NN base learner on a different subsampled dataset? This would presumably increase diversity.

- For efficiently generating more pool candidates, you could also consider ensemble selection applied to all NN checkpoints (eg. every few epochs) rather than only fully trained NN base learners. Such checkpoint-ensembles are known to be superior to individual NN models.


- Other comparisons the paper should consider:

While briefly cited as a related work, Webb et al. 2019 [49] needs to be discussed in more detail, and conceptually constrasted against the proposed ideas in this paper. You only cite [49] as evidence that joint training of ensemble is worse than indpeendent training of base learners, but [49] does a much deeper analysis and finds some amount of joint training (ie. end-to-end learning) combined with independent training can lead to the best results.

Appendix C.4 should also compare against another version of HyperEns, where similar architectural design search spaces as considered in NES are considered as hyperparameters to be ensembled in HyperEns. This is a trivial modification of HyperEns and is a highly relevant baseline to compare against (probably should even appear in the main paper).

- How were hyperparameters chosen in this work?

- When it comes to epistemic uncertainty (ie. model uncertainty), the typical assumption is that model family (ie. hypothesis class) is correctly specified and we just don't have enough data to properly identify the right model from the class (this is sometimes called the "M-closed" assumption in Statistics). A major advantage of ensembling heterogeneous models as done here would be the ability to provide better uncertainty estimates under model misspecification, which is always present in practice (especially with neural network models). I think the authors could mention this connection and cite some related literature in this area. I'm not sure Proposition 3.1 is the best way to motivate having more diverse ensembles, as there are other standard arguments in favor of diversity presented in the ensembling literature (eg. statistical independence of base learner errors).


- A remaining concern I have is the following: This paper seems to train networks on D_train and the construct ensemble based on D_val, and finally score the ensemble on D_test. However an alternative standard procedure is to refit all networks on D_train+D_val before evaluating on D_test. I suspect DeepEnsembles would fare better under this alternative than Ensemble-Selection whose outperformance is more contingent on the base learners behaving similarly when fit to just D_train vs D_train+D_val. It would be good to address this point.


- There is also much related work in which ensembles of heterogeneous NN architectures have been found useful that could be referenced. Here's just a couple papers I found thru google:

Ensembles of Networks Produced from Neural Architecture Search. 
Herron et al. 2020.
https://link.springer.com/chapter/10.1007/978-3-030-59851-8_14

Ensembling neural networks: Many could be better than all.
Zhou et al. 2002.
https://www.sciencedirect.com/science/article/pii/S000437020200190X?via%3Dihub

Label Denoising with Large Ensembles of Heterogeneous Neural Networks.
Ostyakov et al. 2019.
https://arxiv.org/abs/1809.04403

Antibody Complementarity Determining Region Design Using High-Capacity Machine Learning.
Liu et al. 2020.
https://academic.oup.com/bioinformatics/article/36/7/2126/5645171

Solo or Ensemble? Choosing a CNN Architecture for Melanoma Classification. 
Perez et al. 2019.
https://openaccess.thecvf.com/content_CVPRW_2019/html/ISIC/Perez_Solo_or_Ensemble_Choosing_a_CNN_Architecture_for_Melanoma_Classification_CVPRW_2019_paper.html

An ensemble of neural networks for weather forecasting. 
Maqsood et al. 2004.
https://link.springer.com/article/10.1007/s00521-004-0413-4

NADS: Neural Architecture Distribution Search for Uncertainty Awareness. 
Ardywibowo et al. 2020.
https://arxiv.org/abs/2006.06646


### Update after reading author response to my original review ###

I am glad to see the authors' additional experiments that addressed my major concerns and have increased my score. I encourage the authors to include all of the points in their response in the revised paper (can squeeze these into appendix as necessary), as I think they will be useful to future readers. And I encourage the authors to tone down any claims of being the first work to ensemble diverse architectures and instead highlight the novelty of this work in terms of discovering the utility of architecture-ensembles for better uncertainty estimation and their new ensemble-aware architecture search algorithm.

Regarding: "perhaps the aggregator should also consider some properties of the architectures being ensembled which might lead to better OOD uncertainty estimation, something that is not easily achieved solely thru validation-score ensemble construction"

One idea could be look at the correlation/variance in the individual ensemble-members predictions as an auxiliary diversity objective (or take a weighted combination of validation-score objective and diversity objective). Some examples are considered in these works:

Brown, G. 2004. Diversity in neural network ensembles. Ph.D. Dissertation, University of Birmingham.

Jain et al. 2020. Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles. https://arxiv.org/abs/1906.07380 
limitations_and_societal_impact: I believe the authors have adequately addressed societal impact of their work.
However a key limitation of the proposed ensembling is that inference latency will be slower for deploying a neural ensemble vs a single network, and may not be suitable for real-time applications. It would be insightful if the authors could compare their ensemble against a single large network where both models are restricted to a total number of NN parameters (summed across all ensemble members in the former case) or restricted to the same inference latency.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes neural ensemble search (NES) for adding architecture variations in forming ensembles. The main observation is that the architecture variations can be more helpful in terms of accuracy and uncertainty estimation, due to their potentially more diverse predictions, compared with constructing ensembles with a fixed architecture and based on random initializations.
main_review: Improving uncertainty estimation of neural networks which the paper targets is an important problem. The proposed method builds a pool of architectures, either by random sampling or an evolutionary method, and selects an ensemble of smaller size with a greedy approach. Both the evolutionary method and greedy ensemble selection are based on previous work, without any major modification.

Using diverse models for predictive tasks is common but the work seems to be the first to systematically focus on and evaluate the effect of varying architectures in ensembles within a NAS space on improving accuracy and uncertainty estimates.

The paper is written clearly and describes the methods and choices made in the design of the method well. It performs several ablation studies to show the effect of the different parts of the method and details about the experimental setups are provided.

There are some improvements that can be made in the experiments.
For the DARTS dataset shift experiments, to isolate the effect of ensembles based on one architecture vs. multiple, DeepEns can be tested with the best architecture according to error on shifted validation data among architectures trained in the proposed NES methods. 

In Figure 7, showing only the number of networks trained may not be the best metric as some methods are embarrassingly parallel. Also, why the DeepEns (RS) versions are expected to perform better than other DeepEns versions when there is no dataset shift? The ablation study is important in empirically validating the paper’s claim, so it would be good to report similar results for more than one setup and dataset and also to test DeepEns with the best architecture among the ones which NES methods select from.

The ECE results (Figure 5), and predictive disagreement (Table 2) do not conclusively demonstrate that the proposed methods are performing better in terms of uncertainty and diversity. Experiments on common OOD setups can help the claims about better uncertainty estimation.

Overall, although the methodological contributions of the paper are not very significant, the execution and the results are valuable. 


--After author response--

Thanks to the authors for their replies. Including the additional results and discussion will further improve the paper and support its claims. I remain positive about the paper. 


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces ‘Neural Ensemble Search’, a method for (automatically) constructing ensembles of NNs which (in contrast with most iterations of deep ensembles) generally do not have the same architecture. The authors show that their method consistently outperforms baseline ensemble methods in accuracy and uncertainty calibration. They hypothesize that the success of this method is mainly due to the varied architectures leading to increased diversity of the predictive distributions of ensemble members, and they provide evidence supporting this claim.
main_review: From a practical standpoint, the method proposed offers modest gains over baselines, particularly considering the high training cost associated with producing the pool of candidate models (though this is less important than test-time cost, which is comparable to other methods, and much of the training can be parallelized). 

However, the contributions of this paper largely outweigh this limitation. The relationship between diversity and performance has emerged as an important topic in understanding ensembles. This paper takes a significant step in addressing the effect of varied architectures on ensembles, which has not been previously studied in detail to my knowledge. The authors are thorough in their experiments, both to show their method outperforms baselines and to attempt to isolate the effects of architecture.

Some ideas that could potentially strengthen this submission:

* Experimental results for ensemble methods in which architecture varies, but ‘ForwardSelect’ is not used. Some examples are described in lines 198-204. Such results would complement the ablation experiments in which you use ForwardSelect over initializations for Deep Ensembles, and (one hopes) further support the claim that the variation in architecture, rather than the selection algorithm, is the main reason for the success of this method

* More analysis focusing on the architecture in detail. For example, is there a clear relationship between how different two models’ architectures are and how diverse their predictions are? I would be very interested in a t-SNE or UMAP plot, similar to that in Figure 2a, where each dot in a color represents a single mutation from the initial architecture instead of a different initialization of it. As another example, are there characteristics that tend to be shared among high performing baseline learners, and do similar architectures (differing by only a few mutations) have similar performances? Such insights could conceivably be leveraged into even better strategies for producing a pool of models and for model selection.

** Update after author response period ** Thanks to the authors for their replies. I remain positive about the work and keep my original recommendation.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel approach for searching ensembles of neural networks to improve the quality of uncertainty estimation and  performance improvement under the dataset shift. The main idea of the paper is to construct a pool of architectures, from which a new ensemble member is selected using a forward greedy algorithm. The paper is empirical, and all claims are strongly supported by thorough experiments. 
main_review: First, this is a great paper, and it has been my pleasure to read it. I also highly appreciate the authors releasing the code. 


I have only two points that would be great to discuss (or add additional results if possible) in the camera-ready version:

1. What is the effect of data augmentation? I have understood from the paper that no augmentations have been used. Why is that so? Is there an ablation study in the supplementary that I am missing?

2. It would be important to see the results on selecting an ensemble with NAS when $D_{val}^{shift}$ is not used for model selection. Since the proposed method used the $D_{val}^{shift}$, and Deep Ensembles did not, it does not seem to be a fair comparison. One can easily add a term to DEs model selection where $D_{val}^{shift}$ is used. Please discuss this in the paper.

limitations_and_societal_impact: The main limitation of this work is the computational cost of the introduced method. This, however, has been discussed by the authors. Other limitations include the concerns raised in the main review, and may only require textual changes in the manuscript.
review_title: null

Meta-review: This paper extends prior work on deep ensembles by introducing methods for constructing ensembles of varying architectures instead of relying on multiple random nationalizations of the same architecture. The methods introduced in the paper represent a novel combination of existing approaches applied specifically to developing architecturally diverse ensembles to improve uncertainty estimation. The writing is clear and the methods are technically correct. The authors present extensive experiments showing that the proposed method outperforms prior approaches. The reviewers had many questions and suggestions for the authors in their initial reviews. Following the discussion, the reviewers were in agreement that their primary questions had been adequately addressed and that the paper should be accepted. The authors need to be sure to include all of the discussed updates in the final version of the paper.

Decision: Accept (Poster)